2017-09-14 15:10:02,028 | DEBUG | node_runner.py ( 18) | run_node | You can find logs in /home/sovrin/.sovrin/Node2.log 2017-09-14 15:10:02,029 | DEBUG | node_runner.py ( 21) | run_node | Sovrin related env vars: [] 2017-09-14 15:10:03,893 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: git 2017-09-14 15:10:03,937 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: hg 2017-09-14 15:10:04,106 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: svn 2017-09-14 15:10:04,107 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: bzr 2017-09-14 15:10:04,916 | DEBUG | selector_events.py ( 53) | __init__ | Using selector: EpollSelector 2017-09-14 15:10:04,917 | DEBUG | looper.py ( 125) | __init__ | Setting handler for SIGINT 2017-09-14 15:10:05,007 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-09-14 15:10:05,070 | DEBUG | ledger.py ( 67) | recoverTree | Recovering tree from transaction log 2017-09-14 15:10:05,173 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.10313456500080065 seconds 2017-09-14 15:10:05,330 | DEBUG | idr_cache.py ( 25) | __init__ | Initializing identity cache Node2 2017-09-14 15:10:05,362 | INFO | node.py (2420) | initStateFromLedger | Node2 found state to be empty, recreating from ledger 2017-09-14 15:10:05,531 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-09-14 15:10:05,580 | DEBUG | ledger.py ( 67) | recoverTree | Recovering tree from transaction log 2017-09-14 15:10:05,667 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.08667582700036291 seconds 2017-09-14 15:10:05,676 | INFO | node.py (2420) | initStateFromLedger | Node2 found state to be empty, recreating from ledger 2017-09-14 15:10:05,790 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-09-14 15:10:05,790 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-09-14 15:10:05,790 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: typing.Dict<~KT, ~VT>] because it does not have a 'pluginType' attribute 2017-09-14 15:10:05,790 | INFO | plugin_loader.py ( 117) | _load | plugin FirebaseStatsConsumer successfully loaded from module plugin_firebase_stats_consumer 2017-09-14 15:10:05,790 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-09-14 15:10:05,790 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-09-14 15:10:05,791 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkPerformance will be repeating every 60 seconds 2017-09-14 15:10:05,791 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 1 to run in 60 seconds 2017-09-14 15:10:05,792 | DEBUG | replica.py ( 313) | h | Node2:0 set watermarks as 0 300 2017-09-14 15:10:05,792 | DISPLAY | replicas.py ( 36) | grow | Node2 added replica Node2:0 to instance 0 (master) 2017-09-14 15:10:05,792 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkPerformance will be repeating every 10 seconds 2017-09-14 15:10:05,792 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 1 to run in 10 seconds 2017-09-14 15:10:05,792 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkNodeRequestSpike will be repeating every 60 seconds 2017-09-14 15:10:05,793 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 2 to run in 60 seconds 2017-09-14 15:10:05,793 | DEBUG | plugin_helper.py ( 24) | loadPlugins | Plugin loading started to load plugins from basedir: /home/sovrin/.sovrin 2017-09-14 15:10:05,793 | DEBUG | plugin_helper.py ( 33) | loadPlugins | Plugin directory created at: /home/sovrin/.sovrin/plugins 2017-09-14 15:10:05,793 | DEBUG | plugin_helper.py ( 68) | loadPlugins | Total plugins loaded from basedir /home/sovrin/.sovrin are : 0 2017-09-14 15:10:05,793 | DEBUG | node.py ( 340) | __init__ | total plugins loaded in node: 0 2017-09-14 15:10:05,962 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-09-14 15:10:06,005 | DEBUG | ledger.py ( 67) | recoverTree | Recovering tree from transaction log 2017-09-14 15:10:06,092 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.08644740599993384 seconds 2017-09-14 15:10:06,115 | DEBUG | upgrader.py ( 118) | check_upgrade_succeeded | Node Node2 has no upgrade events 2017-09-14 15:10:06,115 | INFO | node.py (2420) | initStateFromLedger | Node2 found state to be empty, recreating from ledger 2017-09-14 15:10:06,116 | DEBUG | motor.py ( 34) | set_status | Node2 changing status from stopped to starting 2017-09-14 15:10:06,116 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-09-14 15:10:06,116 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-09-14 15:10:06,116 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-09-14 15:10:06,116 | DEBUG | zstack.py ( 319) | start | Node2 starting with restricted as True and reSetupAuth as True 2017-09-14 15:10:06,117 | DEBUG | authenticator.py ( 31) | start | Starting ZAP at inproc://zeromq.zap.1 2017-09-14 15:10:06,117 | DEBUG | base.py ( 72) | allow | Allowing 0.0.0.0 2017-09-14 15:10:06,117 | DEBUG | base.py ( 112) | configure_curve | Configure curve: *[/home/sovrin/.sovrin/Node2/public_keys] 2017-09-14 15:10:06,118 | DEBUG | zstack.py ( 347) | open | Node2 will bind its listener at 9703 2017-09-14 15:10:06,118 | INFO | stacks.py ( 84) | start | CONNECTION: Node2 listening for other nodes at 0.0.0.0:9703 2017-09-14 15:10:06,118 | DEBUG | zstack.py ( 319) | start | Node2C starting with restricted as False and reSetupAuth as True 2017-09-14 15:10:06,119 | DEBUG | authenticator.py ( 31) | start | Starting ZAP at inproc://zeromq.zap.2 2017-09-14 15:10:06,119 | DEBUG | base.py ( 72) | allow | Allowing 0.0.0.0 2017-09-14 15:10:06,119 | DEBUG | base.py ( 112) | configure_curve | Configure curve: *[*] 2017-09-14 15:10:06,123 | DEBUG | zstack.py ( 347) | open | Node2C will bind its listener at 9704 2017-09-14 15:10:06,123 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action propose_view_change with id 3 to run in 60 seconds 2017-09-14 15:10:06,123 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action dump_json_file with id 4 to run in 3 seconds 2017-09-14 15:10:06,123 | DEBUG | has_action_queue.py ( 79) | startRepeating | dump_json_file will be repeating every 60 seconds 2017-09-14 15:10:06,123 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action dump_json_file with id 5 to run in 60 seconds 2017-09-14 15:10:06,124 | INFO | node.py ( 621) | start | Node2 first time running... 2017-09-14 15:10:06,126 | DEBUG | kit_zstack.py ( 97) | connectToMissing | CONNECTION: Node2 found the following missing connections: Node1, Node3 2017-09-14 15:10:06,126 | TRACE | remote.py ( 86) | connect | connecting socket 78 41031728 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-09-14 15:10:06,133 | INFO | zstack.py ( 590) | connect | CONNECTION: Node2 looking for Node1 at 10.0.0.2:9701 2017-09-14 15:10:06,134 | DEBUG | zstack.py ( 645) | sendPingPong | Node2 pinged Node1 2017-09-14 15:10:06,142 | TRACE | remote.py ( 86) | connect | connecting socket 81 41053920 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-09-14 15:10:06,142 | INFO | zstack.py ( 590) | connect | CONNECTION: Node2 looking for Node3 at 10.0.0.4:9705 2017-09-14 15:10:06,142 | DEBUG | zstack.py ( 645) | sendPingPong | Node2 pinged Node3 2017-09-14 15:10:06,142 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:10:06,154 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'pi' to Node1 2017-09-14 15:10:06,154 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'pi' to Node1 2017-09-14 15:10:06,154 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'pi' to Node3 2017-09-14 15:10:06,154 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'pi' to Node3 2017-09-14 15:10:06,217 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:10:06,218 | DEBUG | zstack.py ( 663) | handlePingPong | Node2 got ping from Node3 2017-09-14 15:10:06,218 | DEBUG | zstack.py ( 645) | sendPingPong | Node2 ponged Node3 2017-09-14 15:10:06,218 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'po' to Node3 2017-09-14 15:10:06,218 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'po' to Node3 2017-09-14 15:10:06,232 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-09-14 15:10:06,232 | DEBUG | zstack.py ( 669) | handlePingPong | Node2 got pong from Node3 2017-09-14 15:10:06,232 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"primary":[],"op":"CURRENT_STATE","viewNo":0}', '{"op":"LEDGER_STATUS","txnSeqNo":3,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","viewNo":null,"ppSeqNo":null,"ledgerId":0}']} 2017-09-14 15:10:06,233 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"primary":[],"op":"CURRENT_STATE","viewNo":0}', '{"op":"LEDGER_STATUS","txnSeqNo":3,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","viewNo":null,"ppSeqNo":null,"ledgerId":0}']}, 'Node3') 2017-09-14 15:10:06,233 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"primary":[],"op":"CURRENT_STATE","viewNo":0}', '{"op":"LEDGER_STATUS","txnSeqNo":3,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","viewNo":null,"ppSeqNo":null,"ledgerId":0}']} 2017-09-14 15:10:06,233 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: CURRENT_STATE{'primary': [], 'viewNo': 0} 2017-09-14 15:10:06,233 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'CURRENT_STATE', 'primary': [], 'viewNo': 0}, 'Node3') 2017-09-14 15:10:06,233 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox CURRENT_STATE{'primary': [], 'viewNo': 0} 2017-09-14 15:10:06,234 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} 2017-09-14 15:10:06,234 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'ledgerId': 0, 'op': 'LEDGER_STATUS', 'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'viewNo': None}, 'Node3') 2017-09-14 15:10:06,234 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} 2017-09-14 15:10:06,234 | DEBUG | node.py ( 907) | process_current_state_message | Node2 processing current state CURRENT_STATE{'primary': [], 'viewNo': 0} from Node3 2017-09-14 15:10:06,234 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} from Node3 2017-09-14 15:10:06,235 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 3 with 3 2017-09-14 15:10:06,235 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 3 with 3 2017-09-14 15:10:06,235 | DEBUG | keep_in_touch.py ( 68) | conns | Node2's connections changed from set() to {'Node3'} 2017-09-14 15:10:06,235 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node2 now connected to Node3 2017-09-14 15:10:06,236 | DEBUG | motor.py ( 34) | set_status | Node2 changing status from starting to started_hungry 2017-09-14 15:10:06,236 | DEBUG | node.py ( 941) | checkInstances | Node2 choosing to start election on the basis of count 2 and nodes {'Node3'} 2017-09-14 15:10:06,236 | DEBUG | primary_selector.py ( 77) | get_msgs_for_lagged_nodes | Node2 has no ViewChangeDone message to send for view 0 2017-09-14 15:10:06,236 | DEBUG | node.py ( 902) | send_current_state_to_lagging_node | Node2 sending current state CURRENT_STATE{'primary': [], 'viewNo': 0} to lagged node Node3 2017-09-14 15:10:06,236 | DEBUG | node.py (2608) | send | Node2 sending message CURRENT_STATE{'primary': [], 'viewNo': 0} to 1 recipients: ['Node3'] 2017-09-14 15:10:06,245 | DEBUG | node.py (2608) | send | Node2 sending message LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} to 1 recipients: ['Node3'] 2017-09-14 15:10:06,246 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-09-14 15:10:06,247 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"CURRENT_STATE","primary":[],"viewNo":0}', b'{"ledgerId":0,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","txnSeqNo":3,"viewNo":null}']) 2017-09-14 15:10:06,247 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[],\\"viewNo\\":0}","{\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":null,\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"txnSeqNo\\":3,\\"viewNo\\":null}"],"signature":null}' 2017-09-14 15:10:06,247 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[],\\"viewNo\\":0}","{\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":null,\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"txnSeqNo\\":3,\\"viewNo\\":null}"],"signature":null}' to Node3 2017-09-14 15:10:06,258 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-09-14 15:10:06,259 | DEBUG | zstack.py ( 663) | handlePingPong | Node2 got ping from Node1 2017-09-14 15:10:06,259 | DEBUG | zstack.py ( 645) | sendPingPong | Node2 ponged Node1 2017-09-14 15:10:06,259 | DEBUG | zstack.py ( 669) | handlePingPong | Node2 got pong from Node1 2017-09-14 15:10:06,259 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'po' to Node1 2017-09-14 15:10:06,260 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'po' to Node1 2017-09-14 15:10:06,260 | DEBUG | keep_in_touch.py ( 68) | conns | Node2's connections changed from {'Node3'} to {'Node1', 'Node3'} 2017-09-14 15:10:06,260 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node2 now connected to Node1 2017-09-14 15:10:06,260 | DEBUG | motor.py ( 34) | set_status | Node2 changing status from started_hungry to started 2017-09-14 15:10:06,260 | DEBUG | node.py ( 941) | checkInstances | Node2 choosing to start election on the basis of count 3 and nodes {'Node1', 'Node3'} 2017-09-14 15:10:06,260 | DEBUG | primary_selector.py ( 77) | get_msgs_for_lagged_nodes | Node2 has no ViewChangeDone message to send for view 0 2017-09-14 15:10:06,261 | DEBUG | node.py ( 902) | send_current_state_to_lagging_node | Node2 sending current state CURRENT_STATE{'primary': [], 'viewNo': 0} to lagged node Node1 2017-09-14 15:10:06,261 | DEBUG | node.py (2608) | send | Node2 sending message CURRENT_STATE{'primary': [], 'viewNo': 0} to 1 recipients: ['Node1'] 2017-09-14 15:10:06,261 | DEBUG | node.py (2608) | send | Node2 sending message LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} to 1 recipients: ['Node1'] 2017-09-14 15:10:06,262 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-09-14 15:10:06,262 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"CURRENT_STATE","primary":[],"viewNo":0}', b'{"ledgerId":0,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","txnSeqNo":3,"viewNo":null}']) 2017-09-14 15:10:06,262 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[],\\"viewNo\\":0}","{\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":null,\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"txnSeqNo\\":3,\\"viewNo\\":null}"],"signature":null}' 2017-09-14 15:10:06,262 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[],\\"viewNo\\":0}","{\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":null,\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"txnSeqNo\\":3,\\"viewNo\\":null}"],"signature":null}' to Node1 2017-09-14 15:10:06,285 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-09-14 15:10:06,285 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":2},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}']} 2017-09-14 15:10:06,286 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"params":{"ledgerId":2},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":2},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}']}, 'Node3') 2017-09-14 15:10:06,286 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":2},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}']} 2017-09-14 15:10:06,286 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:10:06,286 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node3') 2017-09-14 15:10:06,286 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:10:06,286 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:10:06,286 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node3') 2017-09-14 15:10:06,287 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:10:06,287 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}']} 2017-09-14 15:10:06,287 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}'], 'signature': None}, 'Node1') 2017-09-14 15:10:06,287 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}']} 2017-09-14 15:10:06,287 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:10:06,287 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node1') 2017-09-14 15:10:06,287 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:10:06,288 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:10:06,288 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node1') 2017-09-14 15:10:06,288 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:10:06,288 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}} to 1 recipients: ['Node3'] 2017-09-14 15:10:06,288 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}} to 1 recipients: ['Node3'] 2017-09-14 15:10:06,289 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}} to 1 recipients: ['Node1'] 2017-09-14 15:10:06,289 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}} to 1 recipients: ['Node1'] 2017-09-14 15:10:06,289 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-09-14 15:10:06,290 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']) 2017-09-14 15:10:06,290 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}"],"signature":null}' 2017-09-14 15:10:06,290 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}"],"signature":null}' to Node1 2017-09-14 15:10:06,291 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-09-14 15:10:06,291 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']) 2017-09-14 15:10:06,292 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}"],"signature":null}' 2017-09-14 15:10:06,292 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}"],"signature":null}' to Node3 2017-09-14 15:10:06,304 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:10:06,304 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: BATCH{'signature': None, 'messages': ['{"viewNo":0,"op":"CURRENT_STATE","primary":[]}', '{"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","op":"LEDGER_STATUS","viewNo":null,"ledgerId":0,"txnSeqNo":3,"ppSeqNo":null}', '{"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","op":"LEDGER_STATUS","viewNo":null,"ledgerId":1,"txnSeqNo":5,"ppSeqNo":null}']} 2017-09-14 15:10:06,305 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'messages': ['{"viewNo":0,"op":"CURRENT_STATE","primary":[]}', '{"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","op":"LEDGER_STATUS","viewNo":null,"ledgerId":0,"txnSeqNo":3,"ppSeqNo":null}', '{"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","op":"LEDGER_STATUS","viewNo":null,"ledgerId":1,"txnSeqNo":5,"ppSeqNo":null}'], 'signature': None}, 'Node1') 2017-09-14 15:10:06,305 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"viewNo":0,"op":"CURRENT_STATE","primary":[]}', '{"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","op":"LEDGER_STATUS","viewNo":null,"ledgerId":0,"txnSeqNo":3,"ppSeqNo":null}', '{"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","op":"LEDGER_STATUS","viewNo":null,"ledgerId":1,"txnSeqNo":5,"ppSeqNo":null}']} 2017-09-14 15:10:06,305 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: CURRENT_STATE{'primary': [], 'viewNo': 0} 2017-09-14 15:10:06,305 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'CURRENT_STATE', 'primary': [], 'viewNo': 0}, 'Node1') 2017-09-14 15:10:06,305 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox CURRENT_STATE{'primary': [], 'viewNo': 0} 2017-09-14 15:10:06,305 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} 2017-09-14 15:10:06,305 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'ledgerId': 0, 'op': 'LEDGER_STATUS', 'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'viewNo': None}, 'Node1') 2017-09-14 15:10:06,306 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} 2017-09-14 15:10:06,306 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'txnSeqNo': 5, 'ledgerId': 1, 'viewNo': None} 2017-09-14 15:10:06,306 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'ledgerId': 1, 'op': 'LEDGER_STATUS', 'ppSeqNo': None, 'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'txnSeqNo': 5, 'viewNo': None}, 'Node1') 2017-09-14 15:10:06,306 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'txnSeqNo': 5, 'ledgerId': 1, 'viewNo': None} 2017-09-14 15:10:06,306 | DEBUG | node.py ( 907) | process_current_state_message | Node2 processing current state CURRENT_STATE{'primary': [], 'viewNo': 0} from Node1 2017-09-14 15:10:06,306 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} from Node1 2017-09-14 15:10:06,307 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 3 with 3 2017-09-14 15:10:06,307 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 3 with 3 2017-09-14 15:10:06,307 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node1', 'Node3'} that its ledger of type 0 is latest 2017-09-14 15:10:06,307 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node2 found from ledger status LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} that it does not need catchup 2017-09-14 15:10:06,307 | DEBUG | node.py (1479) | preLedgerCatchUp | Node2 going to process any ordered requests before starting catchup. 2017-09-14 15:10:06,307 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node2:0 going to remove 0 Ordered messages from outbox 2017-09-14 15:10:06,307 | DEBUG | node.py (1888) | force_process_ordered | Node2 processed 0 Ordered batches for instance 0 before starting catch up 2017-09-14 15:10:06,307 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-09-14 15:10:06,307 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-09-14 15:10:06,308 | INFO | node.py (1489) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 0 2017-09-14 15:10:06,308 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 3 with 3 2017-09-14 15:10:06,308 | DEBUG | node.py ( 941) | checkInstances | Node2 choosing to start election on the basis of count 3 and nodes {'Node1', 'Node3'} 2017-09-14 15:10:06,308 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node1'] 2017-09-14 15:10:06,308 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node1 for ledger status of ledger 2 2017-09-14 15:10:06,308 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to all recipients: ['Node1', 'Node3'] 2017-09-14 15:10:06,308 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node2 for ledger status of ledger 2 2017-09-14 15:10:06,309 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-09-14 15:10:06,309 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node3 for ledger status of ledger 2 2017-09-14 15:10:06,309 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node2 going to process 0 stashed ledger statuses for ledger 2 2017-09-14 15:10:06,313 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 0, caught up 0 in total 2017-09-14 15:10:06,313 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'txnSeqNo': 5, 'ledgerId': 1, 'viewNo': None} from Node1 2017-09-14 15:10:06,313 | DEBUG | ledger_manager.py (1016) | stashLedgerStatus | Node2 stashing ledger status LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'txnSeqNo': 5, 'ledgerId': 1, 'viewNo': None} from Node1 2017-09-14 15:10:06,314 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-09-14 15:10:06,314 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-09-14 15:10:06,314 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' 2017-09-14 15:10:06,314 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' to Node1 2017-09-14 15:10:06,314 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-09-14 15:10:06,314 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-09-14 15:10:06,315 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' 2017-09-14 15:10:06,315 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' to Node3 2017-09-14 15:10:06,349 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-09-14 15:10:06,350 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":1},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":1},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']} 2017-09-14 15:10:06,350 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"params":{"ledgerId":1},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":1},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']}, 'Node3') 2017-09-14 15:10:06,350 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":1},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":1},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']} 2017-09-14 15:10:06,350 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:10:06,350 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node3') 2017-09-14 15:10:06,350 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:10:06,350 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:10:06,351 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node3') 2017-09-14 15:10:06,351 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:10:06,351 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}} 2017-09-14 15:10:06,351 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}}, 'Node3') 2017-09-14 15:10:06,351 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}} 2017-09-14 15:10:06,352 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}} 2017-09-14 15:10:06,352 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}}, 'Node3') 2017-09-14 15:10:06,352 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}} 2017-09-14 15:10:06,352 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}']} 2017-09-14 15:10:06,352 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}'], 'signature': None}, 'Node1') 2017-09-14 15:10:06,352 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}']} 2017-09-14 15:10:06,353 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}} 2017-09-14 15:10:06,353 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}}, 'Node1') 2017-09-14 15:10:06,353 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}} 2017-09-14 15:10:06,353 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}} 2017-09-14 15:10:06,353 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}}, 'Node1') 2017-09-14 15:10:06,353 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}} 2017-09-14 15:10:06,353 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:10:06,354 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node1') 2017-09-14 15:10:06,354 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:10:06,354 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:10:06,354 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node1') 2017-09-14 15:10:06,354 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:10:06,355 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'txnSeqNo': 5, 'ledgerId': 1, 'viewNo': None}} to 1 recipients: ['Node3'] 2017-09-14 15:10:06,355 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'txnSeqNo': 5, 'ledgerId': 1, 'viewNo': None}} to 1 recipients: ['Node3'] 2017-09-14 15:10:06,355 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None} from Node3 2017-09-14 15:10:06,359 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-09-14 15:10:06,359 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-09-14 15:10:06,360 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None} from Node3 2017-09-14 15:10:06,360 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-09-14 15:10:06,360 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-09-14 15:10:06,360 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None} from Node1 2017-09-14 15:10:06,361 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-09-14 15:10:06,361 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-09-14 15:10:06,361 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node1', 'Node3'} that its ledger of type 2 is latest 2017-09-14 15:10:06,361 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node2 found from ledger status LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None} that it does not need catchup 2017-09-14 15:10:06,361 | DEBUG | node.py (1479) | preLedgerCatchUp | Node2 going to process any ordered requests before starting catchup. 2017-09-14 15:10:06,361 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node2:0 going to remove 0 Ordered messages from outbox 2017-09-14 15:10:06,361 | DEBUG | node.py (1888) | force_process_ordered | Node2 processed 0 Ordered batches for instance 0 before starting catch up 2017-09-14 15:10:06,361 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-09-14 15:10:06,362 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-09-14 15:10:06,362 | INFO | node.py (1489) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 2 2017-09-14 15:10:06,362 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-09-14 15:10:06,362 | DEBUG | pool_config.py ( 35) | processLedger | processing config ledger for any POOL_CONFIGs 2017-09-14 15:10:06,362 | DEBUG | upgrader.py ( 208) | processLedger | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb processing config ledger for any upgrades 2017-09-14 15:10:06,362 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node1'] 2017-09-14 15:10:06,363 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node1 for ledger status of ledger 1 2017-09-14 15:10:06,363 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to all recipients: ['Node1', 'Node3'] 2017-09-14 15:10:06,363 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node2 for ledger status of ledger 1 2017-09-14 15:10:06,363 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-09-14 15:10:06,363 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node3 for ledger status of ledger 1 2017-09-14 15:10:06,363 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node2 going to process 1 stashed ledger statuses for ledger 1 2017-09-14 15:10:06,364 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'txnSeqNo': 5, 'ledgerId': 1, 'viewNo': None} from Node1 2017-09-14 15:10:06,364 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 5 with 5 2017-09-14 15:10:06,364 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 5 with 5 2017-09-14 15:10:06,364 | INFO | upgrader.py ( 150) | should_notify_about_upgrade_result | Node's 'Node2' last upgrade txn is None 2017-09-14 15:10:06,364 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 2, caught up 0 in total 2017-09-14 15:10:06,364 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None} from Node1 2017-09-14 15:10:06,365 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-09-14 15:10:06,365 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-09-14 15:10:06,365 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'txnSeqNo': 5, 'ledgerId': 1, 'viewNo': None}} to 1 recipients: ['Node1'] 2017-09-14 15:10:06,365 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'txnSeqNo': 5, 'ledgerId': 1, 'viewNo': None}} to 1 recipients: ['Node1'] 2017-09-14 15:10:06,366 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 4 msgs to Node1 into one transmission 2017-09-14 15:10:06,373 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null}}']) 2017-09-14 15:10:06,373 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":5,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":5,\\"viewNo\\":null}}"],"signature":null}' 2017-09-14 15:10:06,374 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":5,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":5,\\"viewNo\\":null}}"],"signature":null}' to Node1 2017-09-14 15:10:06,374 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 4 msgs to Node3 into one transmission 2017-09-14 15:10:06,374 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-09-14 15:10:06,375 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":5,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":5,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' 2017-09-14 15:10:06,375 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":5,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":5,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' to Node3 2017-09-14 15:10:06,409 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-09-14 15:10:06,410 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,5,"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","op":"VIEW_CHANGE_DONE","viewNo":0}']} 2017-09-14 15:10:06,410 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,5,"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","op":"VIEW_CHANGE_DONE","viewNo":0}'], 'signature': None}, 'Node1') 2017-09-14 15:10:06,410 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,5,"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","op":"VIEW_CHANGE_DONE","viewNo":0}']} 2017-09-14 15:10:06,410 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'txnSeqNo': 5, 'ledgerId': 1, 'viewNo': None}} 2017-09-14 15:10:06,410 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'txnSeqNo': 5, 'ledgerId': 1, 'viewNo': None}}, 'Node1') 2017-09-14 15:10:06,410 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'txnSeqNo': 5, 'ledgerId': 1, 'viewNo': None}} 2017-09-14 15:10:06,411 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'txnSeqNo': 5, 'ledgerId': 1, 'viewNo': None}} 2017-09-14 15:10:06,411 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'txnSeqNo': 5, 'ledgerId': 1, 'viewNo': None}}, 'Node1') 2017-09-14 15:10:06,411 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'txnSeqNo': 5, 'ledgerId': 1, 'viewNo': None}} 2017-09-14 15:10:06,411 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-09-14 15:10:06,411 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'VIEW_CHANGE_DONE', 'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0}, 'Node1') 2017-09-14 15:10:06,411 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-09-14 15:10:06,412 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null}}', '{"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null}}', '{"ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,5,"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","viewNo":0,"name":"Node1"}']} 2017-09-14 15:10:06,412 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null}}', '{"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null}}', '{"ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,5,"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","viewNo":0,"name":"Node1"}']}, 'Node3') 2017-09-14 15:10:06,412 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null}}', '{"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null}}', '{"ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,5,"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","viewNo":0,"name":"Node1"}']} 2017-09-14 15:10:06,412 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'txnSeqNo': 5, 'ledgerId': 1, 'viewNo': None}} 2017-09-14 15:10:06,412 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'txnSeqNo': 5, 'ledgerId': 1, 'viewNo': None}}, 'Node3') 2017-09-14 15:10:06,412 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'txnSeqNo': 5, 'ledgerId': 1, 'viewNo': None}} 2017-09-14 15:10:06,412 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'txnSeqNo': 5, 'ledgerId': 1, 'viewNo': None}} 2017-09-14 15:10:06,412 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'txnSeqNo': 5, 'ledgerId': 1, 'viewNo': None}}, 'Node3') 2017-09-14 15:10:06,413 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'txnSeqNo': 5, 'ledgerId': 1, 'viewNo': None}} 2017-09-14 15:10:06,413 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-09-14 15:10:06,413 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'VIEW_CHANGE_DONE', 'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0}, 'Node3') 2017-09-14 15:10:06,413 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-09-14 15:10:06,413 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'txnSeqNo': 5, 'ledgerId': 1, 'viewNo': None} from Node1 2017-09-14 15:10:06,414 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 5 with 5 2017-09-14 15:10:06,414 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 5 with 5 2017-09-14 15:10:06,414 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'txnSeqNo': 5, 'ledgerId': 1, 'viewNo': None} from Node1 2017-09-14 15:10:06,414 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 5 with 5 2017-09-14 15:10:06,414 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 5 with 5 2017-09-14 15:10:06,415 | DEBUG | node.py (1205) | sendToElector | Node2 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0}, 'Node1') 2017-09-14 15:10:06,415 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'txnSeqNo': 5, 'ledgerId': 1, 'viewNo': None} from Node3 2017-09-14 15:10:06,415 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 5 with 5 2017-09-14 15:10:06,415 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 5 with 5 2017-09-14 15:10:06,415 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node1', 'Node3'} that its ledger of type 1 is latest 2017-09-14 15:10:06,415 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node2 found from ledger status LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'txnSeqNo': 5, 'ledgerId': 1, 'viewNo': None} that it does not need catchup 2017-09-14 15:10:06,415 | DEBUG | node.py (1479) | preLedgerCatchUp | Node2 going to process any ordered requests before starting catchup. 2017-09-14 15:10:06,416 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node2:0 going to remove 0 Ordered messages from outbox 2017-09-14 15:10:06,416 | DEBUG | node.py (1888) | force_process_ordered | Node2 processed 0 Ordered batches for instance 0 before starting catch up 2017-09-14 15:10:06,416 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-09-14 15:10:06,416 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-09-14 15:10:06,416 | INFO | node.py (1489) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 1 2017-09-14 15:10:06,416 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 5 with 5 2017-09-14 15:10:06,420 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 1, caught up 0 in total 2017-09-14 15:10:06,421 | DEBUG | node.py (1609) | num_txns_caught_up_in_last_catchup | Node2 caught up to 0 txns in the last catchup 2017-09-14 15:10:06,421 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-09-14 15:10:06,421 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-09-14 15:10:06,421 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node2 needs 1 ViewChangeDone messages 2017-09-14 15:10:06,421 | DEBUG | node.py (1568) | caught_up_for_current_view | Node2 does not have view change quorum for view 0 2017-09-14 15:10:06,421 | DEBUG | node.py (1552) | is_catchup_needed | Node2 is not caught up for the current view 0 2017-09-14 15:10:06,421 | DEBUG | node.py (1609) | num_txns_caught_up_in_last_catchup | Node2 caught up to 0 txns in the last catchup 2017-09-14 15:10:06,421 | DEBUG | node.py (1557) | is_catchup_needed | Node2 ordered till last prepared certificate 2017-09-14 15:10:06,422 | INFO | node.py (1537) | allLedgersCaughtUp | CATCH-UP: Node2 does not need any more catchups 2017-09-14 15:10:06,422 | DEBUG | primary_decider.py ( 134) | send | Node2's elector sending VIEW_CHANGE_DONE{'ledgerInfo': [(0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'name': 'Node1', 'viewNo': 0} 2017-09-14 15:10:06,422 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node2 got view change quorum (1 >= 1) 2017-09-14 15:10:06,422 | DEBUG | primary_selector.py ( 209) | has_view_change_from_primary | Node2 has not received ViewChangeDone from the next primary Node1 (viewNo: 0, totalNodes: 3) 2017-09-14 15:10:06,422 | DEBUG | primary_selector.py ( 266) | _startSelection | Node2 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-09-14 15:10:06,423 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'txnSeqNo': 5, 'ledgerId': 1, 'viewNo': None} from Node3 2017-09-14 15:10:06,423 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 5 with 5 2017-09-14 15:10:06,423 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 5 with 5 2017-09-14 15:10:06,423 | DEBUG | node.py (1205) | sendToElector | Node2 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0}, 'Node3') 2017-09-14 15:10:06,424 | DEBUG | node.py (2608) | send | Node2 sending message VIEW_CHANGE_DONE{'ledgerInfo': [(0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'name': 'Node1', 'viewNo': 0} to all recipients: ['Node1', 'Node3'] 2017-09-14 15:10:06,424 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node2's primary selector started processing of ViewChangeDone msg from Node1 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-09-14 15:10:06,424 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node2 got view change quorum (2 >= 1) 2017-09-14 15:10:06,424 | DEBUG | primary_selector.py ( 215) | has_view_change_from_primary | Node2 received ViewChangeDone from primary Node1 2017-09-14 15:10:06,425 | DEBUG | primary_selector.py ( 238) | has_sufficient_same_view_change_done_messages | Node2 found acceptable primary Node1 and ledger info ((0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')) 2017-09-14 15:10:06,425 | DEBUG | primary_selector.py ( 281) | _startSelection | Node2 starting selection 2017-09-14 15:10:06,425 | DISPLAY | primary_selector.py ( 291) | _startSelection | PRIMARY SELECTION: Node2:0 selected primary Node1:0 for instance 0 (view 0) 2017-09-14 15:10:06,425 | INFO | node.py ( 510) | start_participating | Node2 started participating 2017-09-14 15:10:06,425 | DEBUG | replica.py ( 408) | primaryName | Node2:0 setting primaryName for view no 0 to: Node1:0 2017-09-14 15:10:06,425 | DEBUG | replica.py (1694) | _gc | Node2:0 cleaning up till (0, 0) 2017-09-14 15:10:06,425 | DEBUG | replica.py (1709) | _gc | Node2:0 found 0 3-phase keys to clean 2017-09-14 15:10:06,426 | DEBUG | replica.py (1711) | _gc | Node2:0 found 0 request keys to clean 2017-09-14 15:10:06,426 | DEBUG | replica.py ( 313) | h | Node2:0 set watermarks as 0 300 2017-09-14 15:10:06,426 | DISPLAY | primary_selector.py ( 315) | _startSelection | VIEW CHANGE: Node2:0 declares view change 0 as completed for instance 0, new primary is Node1:0, ledger info is [(0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-09-14 15:10:06,426 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node2's primary selector started processing of ViewChangeDone msg from Node3 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-09-14 15:10:06,426 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} because it already decided primary which is Node1:0 2017-09-14 15:10:06,426 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,5,"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0}' to Node1 2017-09-14 15:10:06,433 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,5,"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0}' to Node1 2017-09-14 15:10:06,433 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,5,"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0}' to Node3 2017-09-14 15:10:06,433 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,5,"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0}' to Node3 2017-09-14 15:10:09,124 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action dump_json_file with id 4 2017-09-14 15:10:15,802 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 1 2017-09-14 15:10:15,803 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:10:15,803 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:10:15,803 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:10:15,803 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:10:15,803 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:10:15,803 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:10:15,803 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 6 to run in 10 seconds 2017-09-14 15:10:21,136 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:10:21,136 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:10:21,137 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:10:25,808 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 6 2017-09-14 15:10:25,809 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:10:25,809 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:10:25,809 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:10:25,809 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:10:25,809 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:10:25,809 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:10:25,809 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 7 to run in 10 seconds 2017-09-14 15:10:35,819 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 7 2017-09-14 15:10:35,819 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:10:35,819 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:10:35,820 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:10:35,820 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:10:35,820 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:10:35,820 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:10:35,820 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 8 to run in 10 seconds 2017-09-14 15:10:36,143 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:10:36,143 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:10:36,144 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:10:45,826 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 8 2017-09-14 15:10:45,826 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:10:45,827 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:10:45,827 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:10:45,827 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:10:45,827 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:10:45,827 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:10:45,827 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 9 to run in 10 seconds 2017-09-14 15:10:51,157 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:10:51,157 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:10:51,157 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:10:55,838 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 9 2017-09-14 15:10:55,839 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:10:55,839 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:10:55,839 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:10:55,839 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:10:55,839 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:10:55,839 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:10:55,839 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 10 to run in 10 seconds 2017-09-14 15:11:05,793 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 2 2017-09-14 15:11:05,793 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node2 checking its request amount 2017-09-14 15:11:05,794 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:11:05,794 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 11 to run in 60 seconds 2017-09-14 15:11:05,794 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 1 2017-09-14 15:11:05,794 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-09-14 15:11:05,794 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 2 to run in 60 seconds 2017-09-14 15:11:05,843 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 10 2017-09-14 15:11:05,844 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:11:05,844 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:11:05,844 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:11:05,844 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:11:05,844 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:11:05,844 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:11:05,844 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 12 to run in 10 seconds 2017-09-14 15:11:06,042 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:11:06,043 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} 2017-09-14 15:11:06,043 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'INSTANCE_CHANGE', 'viewNo': 1, 'reason': 26}, 'Node1') 2017-09-14 15:11:06,043 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} 2017-09-14 15:11:06,043 | DEBUG | node.py (1915) | processInstanceChange | Node2 received instance change request: INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} from Node1 2017-09-14 15:11:06,043 | DEBUG | node.py (1958) | do_view_change_if_possible | Node2 has no quorum for view 1 2017-09-14 15:11:06,043 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:11:06,043 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:11:06,044 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:11:06,044 | DEBUG | node.py (1946) | processInstanceChange | Node2 received instance change message INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} but did not find the master to be slow or has already sent an instance change message 2017-09-14 15:11:06,132 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action dump_json_file with id 5 2017-09-14 15:11:06,133 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action dump_json_file with id 13 to run in 60 seconds 2017-09-14 15:11:06,133 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action propose_view_change with id 3 2017-09-14 15:11:06,133 | TRACE | node.py (2097) | propose_view_change | The primary is already connected so view change will not be proposed 2017-09-14 15:11:06,157 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:11:06,157 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:11:06,157 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:11:15,849 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 12 2017-09-14 15:11:15,849 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:11:15,849 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:11:15,850 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:11:15,850 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:11:15,850 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:11:15,850 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:11:15,850 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 14 to run in 10 seconds 2017-09-14 15:11:21,163 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:11:21,163 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:11:21,164 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:11:25,850 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 14 2017-09-14 15:11:25,851 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:11:25,851 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:11:25,851 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:11:25,851 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:11:25,851 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:11:25,851 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:11:25,851 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 15 to run in 10 seconds 2017-09-14 15:11:35,859 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 15 2017-09-14 15:11:35,859 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:11:35,860 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:11:35,860 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:11:35,860 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:11:35,860 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:11:35,860 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:11:35,860 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 16 to run in 10 seconds 2017-09-14 15:11:36,171 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:11:36,172 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:11:36,172 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:11:45,863 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 16 2017-09-14 15:11:45,863 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:11:45,864 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:11:45,864 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:11:45,864 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:11:45,864 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:11:45,864 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:11:45,864 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 17 to run in 10 seconds 2017-09-14 15:11:51,179 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:11:51,179 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:11:51,180 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:11:55,874 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 17 2017-09-14 15:11:55,874 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:11:55,874 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:11:55,874 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:11:55,875 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:11:55,875 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:11:55,875 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:11:55,875 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 18 to run in 10 seconds 2017-09-14 15:12:05,796 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 11 2017-09-14 15:12:05,796 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node2 checking its request amount 2017-09-14 15:12:05,796 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:12:05,797 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 19 to run in 60 seconds 2017-09-14 15:12:05,798 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 2 2017-09-14 15:12:05,798 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-09-14 15:12:05,798 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 3 to run in 60 seconds 2017-09-14 15:12:05,880 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 18 2017-09-14 15:12:05,880 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:12:05,881 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:12:05,881 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:12:05,881 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:12:05,881 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:12:05,881 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:12:05,881 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 20 to run in 10 seconds 2017-09-14 15:12:06,135 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action dump_json_file with id 13 2017-09-14 15:12:06,136 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action dump_json_file with id 21 to run in 60 seconds 2017-09-14 15:12:06,185 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:12:06,185 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:12:06,185 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:12:15,893 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 20 2017-09-14 15:12:15,893 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:12:15,893 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:12:15,893 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:12:15,893 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:12:15,893 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:12:15,893 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:12:15,893 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 22 to run in 10 seconds 2017-09-14 15:12:21,186 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:12:21,186 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:12:21,186 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:12:25,902 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 22 2017-09-14 15:12:25,902 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:12:25,902 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:12:25,902 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:12:25,902 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:12:25,902 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:12:25,903 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:12:25,903 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 23 to run in 10 seconds 2017-09-14 15:12:35,909 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 23 2017-09-14 15:12:35,909 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:12:35,909 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:12:35,909 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:12:35,909 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:12:35,909 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:12:35,909 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:12:35,909 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 24 to run in 10 seconds 2017-09-14 15:12:36,191 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:12:36,191 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:12:36,192 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:12:41,835 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2C got 1 messages through listener 2017-09-14 15:12:41,838 | DEBUG | zstack.py ( 663) | handlePingPong | Node2C got ping from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:12:41,838 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'po' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:12:41,839 | DEBUG | zstack.py ( 645) | sendPingPong | Node2C ponged b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:12:41,857 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2C got 1 messages through listener 2017-09-14 15:12:41,857 | TRACE | node.py (1388) | validateClientMsg | Node2C received CLIENT message: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'CWoRZ62yMB12JyQEUAYU73CNBqJXWaKUafN9yiR9Dre2', 'txnSeqNo': 4, 'ledgerId': 0, 'viewNo': None} 2017-09-14 15:12:41,858 | DEBUG | node.py (1434) | processClientInBox | Node2C processing b'v#ZljK34WWooA6%RgESn.Ox' request LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'CWoRZ62yMB12JyQEUAYU73CNBqJXWaKUafN9yiR9Dre2', 'txnSeqNo': 4, 'ledgerId': 0, 'viewNo': None} 2017-09-14 15:12:41,858 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'CWoRZ62yMB12JyQEUAYU73CNBqJXWaKUafN9yiR9Dre2', 'txnSeqNo': 4, 'ledgerId': 0, 'viewNo': None} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:12:41,858 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 3 with 4 2017-09-14 15:12:41,858 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 3 with 4 2017-09-14 15:12:41,858 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 3 with 4 2017-09-14 15:12:41,859 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"ledgerId":0,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","txnSeqNo":3,"viewNo":null}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:12:45,917 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 24 2017-09-14 15:12:45,917 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:12:45,917 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:12:45,917 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:12:45,917 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:12:45,917 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:12:45,917 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:12:45,917 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 25 to run in 10 seconds 2017-09-14 15:12:51,193 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:12:51,194 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:12:51,194 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:12:55,925 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 25 2017-09-14 15:12:55,925 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:12:55,925 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:12:55,925 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:12:55,926 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:12:55,926 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:12:55,926 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:12:55,926 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 26 to run in 10 seconds 2017-09-14 15:13:05,799 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 19 2017-09-14 15:13:05,799 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node2 checking its request amount 2017-09-14 15:13:05,800 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:13:05,800 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 27 to run in 60 seconds 2017-09-14 15:13:05,800 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 3 2017-09-14 15:13:05,800 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-09-14 15:13:05,800 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 4 to run in 60 seconds 2017-09-14 15:13:05,929 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 26 2017-09-14 15:13:05,929 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:13:05,929 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:13:05,930 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:13:05,930 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:13:05,930 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:13:05,930 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:13:05,930 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 28 to run in 10 seconds 2017-09-14 15:13:06,145 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action dump_json_file with id 21 2017-09-14 15:13:06,146 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action dump_json_file with id 29 to run in 60 seconds 2017-09-14 15:13:06,201 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:13:06,201 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:13:06,201 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:13:15,934 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 28 2017-09-14 15:13:15,935 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:13:15,935 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:13:15,935 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:13:15,935 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:13:15,935 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:13:15,935 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:13:15,935 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 30 to run in 10 seconds 2017-09-14 15:13:21,203 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:13:21,203 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:13:21,204 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:13:25,947 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 30 2017-09-14 15:13:25,947 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:13:25,947 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:13:25,947 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:13:25,947 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:13:25,947 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:13:25,947 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:13:25,947 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 31 to run in 10 seconds 2017-09-14 15:13:35,952 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 31 2017-09-14 15:13:35,953 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:13:35,953 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:13:35,953 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:13:35,953 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:13:35,954 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:13:35,954 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:13:35,954 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 32 to run in 10 seconds 2017-09-14 15:13:36,214 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:13:36,214 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:13:36,214 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:13:36,733 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2C got 1 messages through listener 2017-09-14 15:13:36,736 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1505402016669500 2017-09-14 15:13:36,741 | TRACE | node.py (1388) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402016669500, 'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi'} 2017-09-14 15:13:36,741 | DEBUG | node.py (1434) | processClientInBox | Node2C processing b'v#ZljK34WWooA6%RgESn.Ox' request SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402016669500, 'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi'} 2017-09-14 15:13:36,741 | DEBUG | node.py (1708) | processRequest | Node2 received client request: SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402016669500, 'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi'} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:13:36,742 | TRACE | propagator.py ( 165) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402016669500, 'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi'} 2017-09-14 15:13:36,742 | INFO | propagator.py ( 148) | propagate | Node2 propagating request ('V4SGRU86Z58d6TV7PBUe6f', 1505402016669500) from client b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:13:36,742 | DEBUG | node.py (2608) | send | Node2 sending message PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402016669500, 'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} to all recipients: ['Node1', 'Node3'] 2017-09-14 15:13:36,742 | DEBUG | propagator.py ( 214) | forward | Node2 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1505402016669500) to 0 replicas 2017-09-14 15:13:36,742 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402016669500}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:13:36,743 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"request":{"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe61"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402016669500,"signature":"3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node1 2017-09-14 15:13:36,743 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"request":{"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe61"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402016669500,"signature":"3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node1 2017-09-14 15:13:36,749 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"request":{"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe61"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402016669500,"signature":"3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node3 2017-09-14 15:13:36,749 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"request":{"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe61"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402016669500,"signature":"3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node3 2017-09-14 15:13:36,750 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-09-14 15:13:36,751 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1505402016669500 2017-09-14 15:13:36,751 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PROPAGATE{'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402016669500, 'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:13:36,752 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402016669500, 'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi'}, 'op': 'PROPAGATE', 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'}, 'Node1') 2017-09-14 15:13:36,752 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402016669500, 'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:13:36,752 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1505402016669500 2017-09-14 15:13:36,753 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PROPAGATE{'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402016669500, 'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:13:36,753 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402016669500, 'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox', 'op': 'PROPAGATE'}, 'Node3') 2017-09-14 15:13:36,753 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402016669500, 'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:13:36,753 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402016669500, 'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:13:36,753 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402016669500, 'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi'} 2017-09-14 15:13:36,753 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402016669500, 'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi'} to its replicas since already forwarded 2017-09-14 15:13:36,753 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402016669500, 'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:13:36,754 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402016669500, 'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi'} 2017-09-14 15:13:36,754 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402016669500, 'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi'} to its replicas since already forwarded 2017-09-14 15:13:36,754 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:13:36,758 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PREPREPARE{'instId': 0, 'discarded': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1505402016669500]], 'digest': '62b4f2d2503fa0bc8be5767f4b54fed1f273375fa9d8cedbb60ff060ef786196', 'txnRootHash': 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE', 'stateRootHash': '6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC', 'ppSeqNo': 1, 'ppTime': 1505402016, 'ledgerId': 1, 'viewNo': 0} 2017-09-14 15:13:36,758 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'discarded': 1, 'instId': 0, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1505402016669500]], 'stateRootHash': '6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC', 'digest': '62b4f2d2503fa0bc8be5767f4b54fed1f273375fa9d8cedbb60ff060ef786196', 'txnRootHash': 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE', 'op': 'PREPREPARE', 'ppTime': 1505402016, 'ppSeqNo': 1, 'ledgerId': 1, 'viewNo': 0}, 'Node1') 2017-09-14 15:13:36,758 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPREPARE{'instId': 0, 'discarded': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1505402016669500]], 'digest': '62b4f2d2503fa0bc8be5767f4b54fed1f273375fa9d8cedbb60ff060ef786196', 'txnRootHash': 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE', 'stateRootHash': '6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC', 'ppSeqNo': 1, 'ppTime': 1505402016, 'ledgerId': 1, 'viewNo': 0} 2017-09-14 15:13:36,759 | DEBUG | replica.py ( 823) | processPrePrepare | Node2:0 received PRE-PREPARE(0, 1) from Node1:0 at 13393.610633025 2017-09-14 15:13:36,759 | DEBUG | replica.py (1034) | validate_pre_prepare | Node2:0 state root before processing PREPREPARE{'instId': 0, 'discarded': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402016669500)], 'digest': '62b4f2d2503fa0bc8be5767f4b54fed1f273375fa9d8cedbb60ff060ef786196', 'txnRootHash': 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE', 'stateRootHash': '6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC', 'ppSeqNo': 1, 'ppTime': 1505402016, 'ledgerId': 1, 'viewNo': 0} is b'4\xa0"\x9b\xd7\xca\x02\x04/\xd6I@G\x9a\xb1\x81\x0c=m\xed\xcbA"$\x05\x06\xd5.\x8at_\x81', ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj 2017-09-14 15:13:36,760 | DEBUG | replica.py ( 934) | doPrepare | Node2:0 Sending PREPARE(0, 1) at 13393.612312568 2017-09-14 15:13:36,761 | DEBUG | replica.py (2193) | send | Node2:0 sending Prepare 2017-09-14 15:13:36,761 | TRACE | replica.py (2194) | send | Node2:0 sending PREPARE{'instId': 0, 'stateRootHash': '6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC', 'digest': '62b4f2d2503fa0bc8be5767f4b54fed1f273375fa9d8cedbb60ff060ef786196', 'txnRootHash': 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE', 'ppSeqNo': 1, 'ppTime': 1505402016, 'viewNo': 0} 2017-09-14 15:13:36,761 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 0, 'stateRootHash': '6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC', 'digest': '62b4f2d2503fa0bc8be5767f4b54fed1f273375fa9d8cedbb60ff060ef786196', 'txnRootHash': 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE', 'ppSeqNo': 1, 'ppTime': 1505402016, 'viewNo': 0} 2017-09-14 15:13:36,761 | DEBUG | replica.py ( 614) | trackBatches | Node2:0 tracking batch for PREPREPARE{'instId': 0, 'discarded': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402016669500)], 'digest': '62b4f2d2503fa0bc8be5767f4b54fed1f273375fa9d8cedbb60ff060ef786196', 'txnRootHash': 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE', 'stateRootHash': '6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC', 'ppSeqNo': 1, 'ppTime': 1505402016, 'ledgerId': 1, 'viewNo': 0} with state root b'4\xa0"\x9b\xd7\xca\x02\x04/\xd6I@G\x9a\xb1\x81\x0c=m\xed\xcbA"$\x05\x06\xd5.\x8at_\x81' 2017-09-14 15:13:36,761 | DEBUG | replica.py ( 842) | processPrePrepare | Node2:0 processed incoming PRE-PREPARE(0, 1) 2017-09-14 15:13:36,762 | DEBUG | node.py (2608) | send | Node2 sending message PREPARE{'instId': 0, 'stateRootHash': '6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC', 'digest': '62b4f2d2503fa0bc8be5767f4b54fed1f273375fa9d8cedbb60ff060ef786196', 'txnRootHash': 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE', 'ppSeqNo': 1, 'ppTime': 1505402016, 'viewNo': 0} to all recipients: ['Node1', 'Node3'] 2017-09-14 15:13:36,763 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"ppTime":1505402016,"instId":0,"stateRootHash":"6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC","digest":"62b4f2d2503fa0bc8be5767f4b54fed1f273375fa9d8cedbb60ff060ef786196","txnRootHash":"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE","op":"PREPARE","ppSeqNo":1,"viewNo":0}' to Node1 2017-09-14 15:13:36,763 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"ppTime":1505402016,"instId":0,"stateRootHash":"6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC","digest":"62b4f2d2503fa0bc8be5767f4b54fed1f273375fa9d8cedbb60ff060ef786196","txnRootHash":"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE","op":"PREPARE","ppSeqNo":1,"viewNo":0}' to Node1 2017-09-14 15:13:36,767 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"ppTime":1505402016,"instId":0,"stateRootHash":"6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC","digest":"62b4f2d2503fa0bc8be5767f4b54fed1f273375fa9d8cedbb60ff060ef786196","txnRootHash":"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE","op":"PREPARE","ppSeqNo":1,"viewNo":0}' to Node3 2017-09-14 15:13:36,767 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"ppTime":1505402016,"instId":0,"stateRootHash":"6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC","digest":"62b4f2d2503fa0bc8be5767f4b54fed1f273375fa9d8cedbb60ff060ef786196","txnRootHash":"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE","op":"PREPARE","ppSeqNo":1,"viewNo":0}' to Node3 2017-09-14 15:13:36,768 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:13:36,768 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'instId': 0, 'stateRootHash': '6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC', 'digest': '62b4f2d2503fa0bc8be5767f4b54fed1f273375fa9d8cedbb60ff060ef786196', 'txnRootHash': 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE', 'ppSeqNo': 1, 'ppTime': 1505402016, 'viewNo': 0} 2017-09-14 15:13:36,768 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'stateRootHash': '6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC', 'digest': '62b4f2d2503fa0bc8be5767f4b54fed1f273375fa9d8cedbb60ff060ef786196', 'txnRootHash': 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE', 'op': 'PREPARE', 'ppSeqNo': 1, 'ppTime': 1505402016, 'viewNo': 0}, 'Node3') 2017-09-14 15:13:36,768 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'instId': 0, 'stateRootHash': '6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC', 'digest': '62b4f2d2503fa0bc8be5767f4b54fed1f273375fa9d8cedbb60ff060ef786196', 'txnRootHash': 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE', 'ppSeqNo': 1, 'ppTime': 1505402016, 'viewNo': 0} 2017-09-14 15:13:36,769 | DEBUG | replica.py ( 953) | doCommit | Node2:0 Sending COMMIT(0, 1) at 13393.621092412 2017-09-14 15:13:36,769 | DEBUG | replica.py (2193) | send | Node2:0 sending Commit 2017-09-14 15:13:36,769 | TRACE | replica.py (2194) | send | Node2:0 sending COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 0} 2017-09-14 15:13:36,770 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 0} commits where f is 0 2017-09-14 15:13:36,770 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (0, 1) 2017-09-14 15:13:36,770 | DEBUG | node.py (2608) | send | Node2 sending message COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 0} to all recipients: ['Node1', 'Node3'] 2017-09-14 15:13:36,771 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:13:36,771 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 0} 2017-09-14 15:13:36,771 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'ppSeqNo': 1, 'viewNo': 0}, 'Node3') 2017-09-14 15:13:36,772 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 0} 2017-09-14 15:13:36,772 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","ppSeqNo":1,"viewNo":0}' to Node1 2017-09-14 15:13:36,772 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","ppSeqNo":1,"viewNo":0}' to Node1 2017-09-14 15:13:36,773 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","ppSeqNo":1,"viewNo":0}' to Node3 2017-09-14 15:13:36,773 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","ppSeqNo":1,"viewNo":0}' to Node3 2017-09-14 15:13:36,773 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(0, 1) from Node3:0 2017-09-14 15:13:36,774 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 0} commits where f is 0 2017-09-14 15:13:36,774 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(0, 1) 2017-09-14 15:13:36,798 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:13:36,798 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 0} 2017-09-14 15:13:36,798 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'ppSeqNo': 1, 'viewNo': 0}, 'Node1') 2017-09-14 15:13:36,798 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 0} 2017-09-14 15:13:36,799 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(0, 1) from Node1:0 2017-09-14 15:13:36,799 | TRACE | replica.py ( 925) | tryOrder | Node2:0 returning request to node 2017-09-14 15:13:36,799 | DEBUG | replica.py (1444) | doOrder | Node2:0 ordering COMMIT (0, 1) 2017-09-14 15:13:36,799 | DEBUG | replica.py ( 323) | last_ordered_3pc | Node2:0 set last ordered as (0, 1) 2017-09-14 15:13:36,800 | DEBUG | replica.py (2193) | send | Node2:0 sending Ordered 2017-09-14 15:13:36,800 | TRACE | replica.py (2194) | send | Node2:0 sending ORDERED{'instId': 0, 'stateRootHash': '6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC', 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402016669500)], 'txnRootHash': 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE', 'ppSeqNo': 1, 'ppTime': 1505402016, 'ledgerId': 1, 'viewNo': 0} 2017-09-14 15:13:36,800 | INFO | replica.py (1483) | order_3pc_key | Node2:0 ordered batch request, view no 0, ppSeqNo 1, ledger 1, state root 6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC, txn root Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1505402016669500)], discarded [] 2017-09-14 15:13:36,800 | DEBUG | replica.py (1556) | _newCheckpointState | Node2:0 adding new checkpoint state for (1, 100) 2017-09-14 15:13:36,800 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(0, 1) 2017-09-14 15:13:36,800 | DEBUG | node.py (1847) | processOrdered | Node2 executing Ordered batch 0 1 of 1 requests 2017-09-14 15:13:36,801 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-09-14 15:13:36,801 | DEBUG | node.py (2386) | sendReplyToClient | Node2 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1505402016669500) to client 2017-09-14 15:13:36,801 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"op":"REPLY","result":{"txnTime":1505402016,"reqId":1505402016669500,"seqNo":6,"rootHash":"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE","type":"1","identifier":"V4SGRU86Z58d6TV7PBUe6f","dest":"V4SGRU86Z58d6TV7PBUe61","signature":"3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi","auditPath":["BSt4P6vp8UXJgSxC9MPouLUPB4VfTSyweJRx1pusrHzc","FA73SaKHCBxBXhf9gis5FxVgz3KxD5GJ1n12yyVYHhDj"]}}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:13:36,802 | INFO | node.py (2290) | executeBatch | Node2 committed batch request, view no 0, ppSeqNo 1, ledger 1, state root 6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC, txn root Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE, requests: [('V4SGRU86Z58d6TV7PBUe6f', 1505402016669500)] 2017-09-14 15:13:36,802 | DEBUG | node.py (2305) | executeBatch | Node2 storing 3PC key (0, 1) for ledger 1 range (6, 6) 2017-09-14 15:13:36,802 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:13:45,964 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 32 2017-09-14 15:13:45,964 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:13:45,964 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:13:45,964 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:13:45,965 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:13:45,965 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:13:45,965 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:13:45,965 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 33 to run in 10 seconds 2017-09-14 15:13:51,216 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:13:51,216 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:13:51,216 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:13:55,976 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 33 2017-09-14 15:13:55,977 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:13:55,977 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:13:55,977 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:13:55,977 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:13:55,977 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:13:55,977 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:13:55,977 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 34 to run in 10 seconds 2017-09-14 15:14:05,803 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 27 2017-09-14 15:14:05,803 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node2 checking its request amount 2017-09-14 15:14:05,803 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:14:05,803 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 35 to run in 60 seconds 2017-09-14 15:14:05,804 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 4 2017-09-14 15:14:05,804 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-09-14 15:14:05,804 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 5 to run in 60 seconds 2017-09-14 15:14:05,986 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 34 2017-09-14 15:14:05,987 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:14:05,987 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:14:05,987 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:14:05,987 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:14:05,988 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:14:05,988 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:14:05,988 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 36 to run in 10 seconds 2017-09-14 15:14:06,154 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action dump_json_file with id 29 2017-09-14 15:14:06,155 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action dump_json_file with id 37 to run in 60 seconds 2017-09-14 15:14:06,218 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:14:06,218 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:14:06,218 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:14:06,256 | TRACE | remote.py ( 119) | hasLostConnection | Remote Node3:HA(host='10.0.0.4', port=9705) has monitor events: [512, 4] 2017-09-14 15:14:06,256 | DEBUG | remote.py ( 123) | hasLostConnection | Node3:HA(host='10.0.0.4', port=9705) found disconnected event on monitor 2017-09-14 15:14:06,256 | DEBUG | keep_in_touch.py ( 68) | conns | Node2's connections changed from {'Node1', 'Node3'} to {'Node1'} 2017-09-14 15:14:06,257 | INFO | keep_in_touch.py ( 93) | _connsChanged | CONNECTION: Node2 disconnected from Node3 2017-09-14 15:14:06,257 | DEBUG | motor.py ( 34) | set_status | Node2 changing status from started to started_hungry 2017-09-14 15:14:06,257 | DEBUG | node.py ( 941) | checkInstances | Node2 choosing to start election on the basis of count 2 and nodes {'Node1'} 2017-09-14 15:14:12,974 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:14:12,975 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1505402052949993 2017-09-14 15:14:12,975 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PROPAGATE{'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe62', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402052949993, 'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:14:12,975 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe62', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402052949993, 'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk'}, 'op': 'PROPAGATE', 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'}, 'Node1') 2017-09-14 15:14:12,975 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe62', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402052949993, 'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:14:12,975 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe62', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402052949993, 'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:14:12,976 | TRACE | propagator.py ( 165) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe62', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402052949993, 'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk'} 2017-09-14 15:14:12,976 | INFO | propagator.py ( 148) | propagate | Node2 propagating request ('V4SGRU86Z58d6TV7PBUe6f', 1505402052949993) from client v#ZljK34WWooA6%RgESn.Ox 2017-09-14 15:14:12,976 | DEBUG | node.py (2608) | send | Node2 sending message PROPAGATE{'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe62', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402052949993, 'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} to all recipients: ['Node1', 'Node3'] 2017-09-14 15:14:12,976 | DEBUG | propagator.py ( 214) | forward | Node2 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1505402052949993) to 0 replicas 2017-09-14 15:14:12,976 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2C got 1 messages through listener 2017-09-14 15:14:12,977 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1505402052949993 2017-09-14 15:14:12,977 | TRACE | node.py (1388) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402052949993, 'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk'} 2017-09-14 15:14:12,977 | DEBUG | node.py (1434) | processClientInBox | Node2C processing b'v#ZljK34WWooA6%RgESn.Ox' request SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402052949993, 'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk'} 2017-09-14 15:14:12,978 | DEBUG | node.py (1708) | processRequest | Node2 received client request: SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402052949993, 'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk'} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:14:12,978 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402052949993, 'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk'} 2017-09-14 15:14:12,978 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402052949993, 'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk'} to its replicas since already forwarded 2017-09-14 15:14:12,978 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402052949993}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:14:12,978 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"request":{"operation":{"dest":"V4SGRU86Z58d6TV7PBUe62","type":"1"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402052949993,"signature":"48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node1 2017-09-14 15:14:12,979 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"request":{"operation":{"dest":"V4SGRU86Z58d6TV7PBUe62","type":"1"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402052949993,"signature":"48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node1 2017-09-14 15:14:12,979 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"request":{"operation":{"dest":"V4SGRU86Z58d6TV7PBUe62","type":"1"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402052949993,"signature":"48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node3 2017-09-14 15:14:12,979 | DEBUG | zstack.py ( 728) | transmit | Remote Node3 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-09-14 15:14:12,979 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"request":{"operation":{"dest":"V4SGRU86Z58d6TV7PBUe62","type":"1"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402052949993,"signature":"48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node3 2017-09-14 15:14:12,982 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:14:12,983 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PREPREPARE{'instId': 0, 'discarded': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1505402052949993]], 'digest': '13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755', 'txnRootHash': 'C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby', 'stateRootHash': 'BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH', 'ppSeqNo': 2, 'ppTime': 1505402052, 'ledgerId': 1, 'viewNo': 0} 2017-09-14 15:14:12,984 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'discarded': 1, 'instId': 0, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1505402052949993]], 'stateRootHash': 'BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH', 'digest': '13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755', 'txnRootHash': 'C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby', 'op': 'PREPREPARE', 'ppTime': 1505402052, 'ppSeqNo': 2, 'ledgerId': 1, 'viewNo': 0}, 'Node1') 2017-09-14 15:14:12,984 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPREPARE{'instId': 0, 'discarded': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1505402052949993]], 'digest': '13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755', 'txnRootHash': 'C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby', 'stateRootHash': 'BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH', 'ppSeqNo': 2, 'ppTime': 1505402052, 'ledgerId': 1, 'viewNo': 0} 2017-09-14 15:14:12,985 | DEBUG | replica.py ( 823) | processPrePrepare | Node2:0 received PRE-PREPARE(0, 2) from Node1:0 at 13429.836648705 2017-09-14 15:14:12,986 | DEBUG | replica.py (1034) | validate_pre_prepare | Node2:0 state root before processing PREPREPARE{'instId': 0, 'discarded': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402052949993)], 'digest': '13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755', 'txnRootHash': 'C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby', 'stateRootHash': 'BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH', 'ppSeqNo': 2, 'ppTime': 1505402052, 'ledgerId': 1, 'viewNo': 0} is b'M\xe0%[\xda\xe2\xc9\x99\x12N\xc5\xa9\x16\xe1G\xd6\x13\x88&\xb7\xb8\x92\xd7N\xf33\xc3\x98\x12+\x9d\xa7', Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE 2017-09-14 15:14:12,987 | DEBUG | replica.py ( 934) | doPrepare | Node2:0 Sending PREPARE(0, 2) at 13429.839011783 2017-09-14 15:14:12,987 | DEBUG | replica.py (2193) | send | Node2:0 sending Prepare 2017-09-14 15:14:12,987 | TRACE | replica.py (2194) | send | Node2:0 sending PREPARE{'instId': 0, 'stateRootHash': 'BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH', 'digest': '13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755', 'txnRootHash': 'C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby', 'ppSeqNo': 2, 'ppTime': 1505402052, 'viewNo': 0} 2017-09-14 15:14:12,988 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 0, 'stateRootHash': 'BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH', 'digest': '13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755', 'txnRootHash': 'C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby', 'ppSeqNo': 2, 'ppTime': 1505402052, 'viewNo': 0} 2017-09-14 15:14:12,988 | DEBUG | replica.py ( 614) | trackBatches | Node2:0 tracking batch for PREPREPARE{'instId': 0, 'discarded': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402052949993)], 'digest': '13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755', 'txnRootHash': 'C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby', 'stateRootHash': 'BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH', 'ppSeqNo': 2, 'ppTime': 1505402052, 'ledgerId': 1, 'viewNo': 0} with state root b'M\xe0%[\xda\xe2\xc9\x99\x12N\xc5\xa9\x16\xe1G\xd6\x13\x88&\xb7\xb8\x92\xd7N\xf33\xc3\x98\x12+\x9d\xa7' 2017-09-14 15:14:12,988 | DEBUG | replica.py ( 842) | processPrePrepare | Node2:0 processed incoming PRE-PREPARE(0, 2) 2017-09-14 15:14:12,988 | DEBUG | node.py (2608) | send | Node2 sending message PREPARE{'instId': 0, 'stateRootHash': 'BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH', 'digest': '13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755', 'txnRootHash': 'C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby', 'ppSeqNo': 2, 'ppTime': 1505402052, 'viewNo': 0} to all recipients: ['Node1', 'Node3'] 2017-09-14 15:14:12,989 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"ppTime":1505402052,"instId":0,"stateRootHash":"BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH","digest":"13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755","txnRootHash":"C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby","op":"PREPARE","ppSeqNo":2,"viewNo":0}' to Node1 2017-09-14 15:14:12,989 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"ppTime":1505402052,"instId":0,"stateRootHash":"BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH","digest":"13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755","txnRootHash":"C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby","op":"PREPARE","ppSeqNo":2,"viewNo":0}' to Node1 2017-09-14 15:14:12,989 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"ppTime":1505402052,"instId":0,"stateRootHash":"BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH","digest":"13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755","txnRootHash":"C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby","op":"PREPARE","ppSeqNo":2,"viewNo":0}' to Node3 2017-09-14 15:14:12,989 | DEBUG | zstack.py ( 728) | transmit | Remote Node3 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-09-14 15:14:12,989 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"ppTime":1505402052,"instId":0,"stateRootHash":"BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH","digest":"13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755","txnRootHash":"C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby","op":"PREPARE","ppSeqNo":2,"viewNo":0}' to Node3 2017-09-14 15:14:15,997 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 36 2017-09-14 15:14:15,997 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:14:15,997 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:14:15,997 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:14:15,997 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:14:15,997 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:14:15,998 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:14:15,998 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 38 to run in 10 seconds 2017-09-14 15:14:21,219 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:14:21,219 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:14:21,220 | DEBUG | zstack.py ( 645) | sendPingPong | Node2 pinged Node3 2017-09-14 15:14:21,220 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:14:21,231 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'pi' to Node3 2017-09-14 15:14:21,232 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'pi' to Node3 2017-09-14 15:14:26,001 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 38 2017-09-14 15:14:26,001 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:14:26,001 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:14:26,001 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:14:26,001 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:14:26,001 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:14:26,001 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:14:26,002 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 39 to run in 10 seconds 2017-09-14 15:14:27,979 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2C got 1 messages through listener 2017-09-14 15:14:27,980 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1505402052949993 2017-09-14 15:14:27,980 | TRACE | node.py (1388) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe62', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402052949993, 'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk'} 2017-09-14 15:14:27,984 | DEBUG | node.py (1434) | processClientInBox | Node2C processing b'v#ZljK34WWooA6%RgESn.Ox' request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe62', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402052949993, 'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk'} 2017-09-14 15:14:27,984 | DEBUG | node.py (1708) | processRequest | Node2 received client request: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe62', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402052949993, 'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk'} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:14:27,984 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe62', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402052949993, 'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk'} 2017-09-14 15:14:27,984 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe62', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402052949993, 'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk'} to its replicas since already forwarded 2017-09-14 15:14:27,984 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402052949993}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:14:31,468 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:14:31,469 | DEBUG | zstack.py ( 663) | handlePingPong | Node2 got ping from Node3 2017-09-14 15:14:31,469 | DEBUG | zstack.py ( 645) | sendPingPong | Node2 ponged Node3 2017-09-14 15:14:31,469 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'po' to Node3 2017-09-14 15:14:31,469 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'po' to Node3 2017-09-14 15:14:31,575 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:14:31,576 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1505402052949993 2017-09-14 15:14:31,576 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402052949993, 'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:14:31,577 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402052949993, 'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox', 'op': 'PROPAGATE'}, 'Node3') 2017-09-14 15:14:31,577 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402052949993, 'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:14:31,577 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402052949993, 'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:14:31,577 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402052949993, 'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk'} 2017-09-14 15:14:31,577 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402052949993, 'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk'} to its replicas since already forwarded 2017-09-14 15:14:31,630 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-09-14 15:14:31,630 | DEBUG | zstack.py ( 669) | handlePingPong | Node2 got pong from Node3 2017-09-14 15:14:31,630 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"CURRENT_STATE","viewNo":0,"primary":[{"ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,5,"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0}]}', '{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","txnSeqNo":3,"viewNo":null,"op":"LEDGER_STATUS","ppSeqNo":null}']} 2017-09-14 15:14:31,631 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"op":"CURRENT_STATE","viewNo":0,"primary":[{"ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,5,"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0}]}', '{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","txnSeqNo":3,"viewNo":null,"op":"LEDGER_STATUS","ppSeqNo":null}']}, 'Node3') 2017-09-14 15:14:31,631 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"op":"CURRENT_STATE","viewNo":0,"primary":[{"ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,5,"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0}]}', '{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","txnSeqNo":3,"viewNo":null,"op":"LEDGER_STATUS","ppSeqNo":null}']} 2017-09-14 15:14:31,631 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: CURRENT_STATE{'primary': [{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0}], 'viewNo': 0} 2017-09-14 15:14:31,631 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'CURRENT_STATE', 'primary': [{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0}], 'viewNo': 0}, 'Node3') 2017-09-14 15:14:31,631 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox CURRENT_STATE{'primary': [{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0}], 'viewNo': 0} 2017-09-14 15:14:31,632 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} 2017-09-14 15:14:31,632 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'ledgerId': 0, 'op': 'LEDGER_STATUS', 'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'viewNo': None}, 'Node3') 2017-09-14 15:14:31,632 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} 2017-09-14 15:14:31,632 | DEBUG | node.py ( 907) | process_current_state_message | Node2 processing current state CURRENT_STATE{'primary': [{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0}], 'viewNo': 0} from Node3 2017-09-14 15:14:31,632 | DEBUG | node.py (1205) | sendToElector | Node2 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0}, 'Node3') 2017-09-14 15:14:31,632 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} from Node3 2017-09-14 15:14:31,633 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 3 with 3 2017-09-14 15:14:31,633 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 3 with 3 2017-09-14 15:14:31,633 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node2's primary selector started processing of ViewChangeDone msg from Node3 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-09-14 15:14:31,633 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} because it already decided primary which is Node1:0 2017-09-14 15:14:31,636 | TRACE | remote.py ( 119) | hasLostConnection | Remote Node3:HA(host='10.0.0.4', port=9705) has monitor events: [2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 1] 2017-09-14 15:14:31,637 | DEBUG | remote.py ( 123) | hasLostConnection | Node3:HA(host='10.0.0.4', port=9705) found disconnected event on monitor 2017-09-14 15:14:31,637 | DEBUG | keep_in_touch.py ( 68) | conns | Node2's connections changed from {'Node1'} to {'Node1', 'Node3'} 2017-09-14 15:14:31,637 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node2 now connected to Node3 2017-09-14 15:14:31,637 | DEBUG | motor.py ( 34) | set_status | Node2 changing status from started_hungry to started 2017-09-14 15:14:31,637 | DEBUG | node.py ( 941) | checkInstances | Node2 choosing to start election on the basis of count 3 and nodes {'Node1', 'Node3'} 2017-09-14 15:14:31,637 | DEBUG | node.py ( 902) | send_current_state_to_lagging_node | Node2 sending current state CURRENT_STATE{'primary': [VIEW_CHANGE_DONE{'ledgerInfo': ((0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')), 'name': 'Node1', 'viewNo': 0}], 'viewNo': 0} to lagged node Node3 2017-09-14 15:14:31,638 | DEBUG | node.py (2608) | send | Node2 sending message CURRENT_STATE{'primary': [VIEW_CHANGE_DONE{'ledgerInfo': ((0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')), 'name': 'Node1', 'viewNo': 0}], 'viewNo': 0} to 1 recipients: ['Node3'] 2017-09-14 15:14:31,638 | DEBUG | node.py (2608) | send | Node2 sending message LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} to 1 recipients: ['Node3'] 2017-09-14 15:14:31,638 | DEBUG | node.py (2608) | send | Node2 sending message LEDGER_STATUS{'ppSeqNo': 1, 'merkleRoot': 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE', 'txnSeqNo': 6, 'ledgerId': 1, 'viewNo': 0} to 1 recipients: ['Node3'] 2017-09-14 15:14:31,638 | DEBUG | node.py (2608) | send | Node2 sending message LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None} to 1 recipients: ['Node3'] 2017-09-14 15:14:31,640 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 4 msgs to Node3 into one transmission 2017-09-14 15:14:31,640 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"CURRENT_STATE","primary":[{"ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,5,"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0}],"viewNo":0}', b'{"ledgerId":0,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","txnSeqNo":3,"viewNo":null}', b'{"ledgerId":1,"op":"LEDGER_STATUS","ppSeqNo":1,"merkleRoot":"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE","txnSeqNo":6,"viewNo":0}', b'{"ledgerId":2,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","txnSeqNo":0,"viewNo":null}']) 2017-09-14 15:14:31,640 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[{\\"ledgerInfo\\":[[0,3,\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\"],[1,5,\\"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"name\\":\\"Node1\\",\\"viewNo\\":0}],\\"viewNo\\":0}","{\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":null,\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"txnSeqNo\\":3,\\"viewNo\\":null}","{\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":1,\\"merkleRoot\\":\\"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE\\",\\"txnSeqNo\\":6,\\"viewNo\\":0}","{\\"ledgerId\\":2,\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":null,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"txnSeqNo\\":0,\\"viewNo\\":null}"],"signature":null}' 2017-09-14 15:14:31,640 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[{\\"ledgerInfo\\":[[0,3,\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\"],[1,5,\\"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"name\\":\\"Node1\\",\\"viewNo\\":0}],\\"viewNo\\":0}","{\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":null,\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"txnSeqNo\\":3,\\"viewNo\\":null}","{\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":1,\\"merkleRoot\\":\\"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE\\",\\"txnSeqNo\\":6,\\"viewNo\\":0}","{\\"ledgerId\\":2,\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":null,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"txnSeqNo\\":0,\\"viewNo\\":null}"],"signature":null}' to Node3 2017-09-14 15:14:31,680 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-09-14 15:14:31,681 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"name":"Node1","op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,6,"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":0}']} 2017-09-14 15:14:31,681 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"name":"Node1","op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,6,"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":0}']}, 'Node3') 2017-09-14 15:14:31,681 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"name":"Node1","op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,6,"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":0}']} 2017-09-14 15:14:31,681 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:14:31,682 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node3') 2017-09-14 15:14:31,682 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:14:31,682 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:14:31,682 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node3') 2017-09-14 15:14:31,682 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:14:31,682 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:14:31,682 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node3') 2017-09-14 15:14:31,682 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:14:31,683 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:14:31,683 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node3') 2017-09-14 15:14:31,683 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:14:31,687 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 6, 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-09-14 15:14:31,687 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'VIEW_CHANGE_DONE', 'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 6, 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0}, 'Node3') 2017-09-14 15:14:31,687 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 6, 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-09-14 15:14:31,687 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"digest":"13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755","ppTime":1505402052,"stateRootHash":"BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH","op":"PREPARE","ppSeqNo":2,"txnRootHash":"C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby","viewNo":0,"instId":0}', '{"op":"COMMIT","instId":0,"viewNo":0,"ppSeqNo":2}']} 2017-09-14 15:14:31,687 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"digest":"13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755","ppTime":1505402052,"stateRootHash":"BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH","op":"PREPARE","ppSeqNo":2,"txnRootHash":"C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby","viewNo":0,"instId":0}', '{"op":"COMMIT","instId":0,"viewNo":0,"ppSeqNo":2}']}, 'Node3') 2017-09-14 15:14:31,687 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"digest":"13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755","ppTime":1505402052,"stateRootHash":"BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH","op":"PREPARE","ppSeqNo":2,"txnRootHash":"C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby","viewNo":0,"instId":0}', '{"op":"COMMIT","instId":0,"viewNo":0,"ppSeqNo":2}']} 2017-09-14 15:14:31,688 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'instId': 0, 'stateRootHash': 'BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH', 'digest': '13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755', 'txnRootHash': 'C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby', 'ppSeqNo': 2, 'ppTime': 1505402052, 'viewNo': 0} 2017-09-14 15:14:31,688 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'stateRootHash': 'BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH', 'digest': '13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755', 'txnRootHash': 'C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby', 'op': 'PREPARE', 'ppTime': 1505402052, 'ppSeqNo': 2, 'viewNo': 0}, 'Node3') 2017-09-14 15:14:31,688 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'instId': 0, 'stateRootHash': 'BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH', 'digest': '13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755', 'txnRootHash': 'C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby', 'ppSeqNo': 2, 'ppTime': 1505402052, 'viewNo': 0} 2017-09-14 15:14:31,688 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 0} 2017-09-14 15:14:31,688 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'ppSeqNo': 2, 'viewNo': 0}, 'Node3') 2017-09-14 15:14:31,688 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 0} 2017-09-14 15:14:31,692 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}} to 1 recipients: ['Node3'] 2017-09-14 15:14:31,693 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}} to 1 recipients: ['Node3'] 2017-09-14 15:14:31,693 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': LEDGER_STATUS{'ppSeqNo': 1, 'merkleRoot': 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE', 'txnSeqNo': 6, 'ledgerId': 1, 'viewNo': 0}} to 1 recipients: ['Node3'] 2017-09-14 15:14:31,693 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': LEDGER_STATUS{'ppSeqNo': 1, 'merkleRoot': 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE', 'txnSeqNo': 6, 'ledgerId': 1, 'viewNo': 0}} to 1 recipients: ['Node3'] 2017-09-14 15:14:31,694 | DEBUG | node.py (1205) | sendToElector | Node2 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 6, 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0}, 'Node3') 2017-09-14 15:14:31,694 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node2's primary selector started processing of ViewChangeDone msg from Node3 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 6, 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-09-14 15:14:31,694 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 6, 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} because it already decided primary which is Node1:0 2017-09-14 15:14:31,694 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 4 msgs to Node3 into one transmission 2017-09-14 15:14:31,695 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE","ppSeqNo":1,"txnSeqNo":6,"viewNo":0}}', b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE","ppSeqNo":1,"txnSeqNo":6,"viewNo":0}}']) 2017-09-14 15:14:31,695 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE\\",\\"ppSeqNo\\":1,\\"txnSeqNo\\":6,\\"viewNo\\":0}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE\\",\\"ppSeqNo\\":1,\\"txnSeqNo\\":6,\\"viewNo\\":0}}"],"signature":null}' 2017-09-14 15:14:31,695 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE\\",\\"ppSeqNo\\":1,\\"txnSeqNo\\":6,\\"viewNo\\":0}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE\\",\\"ppSeqNo\\":1,\\"txnSeqNo\\":6,\\"viewNo\\":0}}"],"signature":null}' to Node3 2017-09-14 15:14:31,695 | DEBUG | replica.py ( 953) | doCommit | Node2:0 Sending COMMIT(0, 2) at 13448.547325132 2017-09-14 15:14:31,696 | DEBUG | replica.py (2193) | send | Node2:0 sending Commit 2017-09-14 15:14:31,696 | TRACE | replica.py (2194) | send | Node2:0 sending COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 0} 2017-09-14 15:14:31,696 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 0} commits where f is 0 2017-09-14 15:14:31,696 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (0, 2) 2017-09-14 15:14:31,696 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(0, 2) from Node3:0 2017-09-14 15:14:31,696 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 0} commits where f is 0 2017-09-14 15:14:31,696 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(0, 2) 2017-09-14 15:14:31,696 | DEBUG | node.py (2608) | send | Node2 sending message COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 0} to all recipients: ['Node1', 'Node3'] 2017-09-14 15:14:31,697 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:14:31,697 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 0} 2017-09-14 15:14:31,697 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'ppSeqNo': 2, 'viewNo': 0}, 'Node1') 2017-09-14 15:14:31,697 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 0} 2017-09-14 15:14:31,698 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","ppSeqNo":2,"viewNo":0}' to Node1 2017-09-14 15:14:31,698 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","ppSeqNo":2,"viewNo":0}' to Node1 2017-09-14 15:14:31,698 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","ppSeqNo":2,"viewNo":0}' to Node3 2017-09-14 15:14:31,698 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","ppSeqNo":2,"viewNo":0}' to Node3 2017-09-14 15:14:31,699 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(0, 2) from Node1:0 2017-09-14 15:14:31,699 | TRACE | replica.py ( 925) | tryOrder | Node2:0 returning request to node 2017-09-14 15:14:31,699 | DEBUG | replica.py (1444) | doOrder | Node2:0 ordering COMMIT (0, 2) 2017-09-14 15:14:31,699 | DEBUG | replica.py ( 323) | last_ordered_3pc | Node2:0 set last ordered as (0, 2) 2017-09-14 15:14:31,699 | DEBUG | replica.py (2193) | send | Node2:0 sending Ordered 2017-09-14 15:14:31,701 | TRACE | replica.py (2194) | send | Node2:0 sending ORDERED{'instId': 0, 'stateRootHash': 'BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH', 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402052949993)], 'txnRootHash': 'C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby', 'ppSeqNo': 2, 'ppTime': 1505402052, 'ledgerId': 1, 'viewNo': 0} 2017-09-14 15:14:31,701 | INFO | replica.py (1483) | order_3pc_key | Node2:0 ordered batch request, view no 0, ppSeqNo 2, ledger 1, state root BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH, txn root C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1505402052949993)], discarded [] 2017-09-14 15:14:31,701 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(0, 2) 2017-09-14 15:14:31,701 | DEBUG | node.py (1847) | processOrdered | Node2 executing Ordered batch 0 2 of 1 requests 2017-09-14 15:14:31,702 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-09-14 15:14:31,702 | DEBUG | node.py (2386) | sendReplyToClient | Node2 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1505402052949993) to client 2017-09-14 15:14:31,702 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"op":"REPLY","result":{"txnTime":1505402052,"dest":"V4SGRU86Z58d6TV7PBUe62","reqId":1505402052949993,"seqNo":7,"rootHash":"C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby","identifier":"V4SGRU86Z58d6TV7PBUe6f","type":"1","signature":"48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk","auditPath":["Dvt18TQs3z1emxZcnmW3jpcjdsycqSQwPFm63UUJzJbs","FA73SaKHCBxBXhf9gis5FxVgz3KxD5GJ1n12yyVYHhDj"]}}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:14:31,702 | INFO | node.py (2290) | executeBatch | Node2 committed batch request, view no 0, ppSeqNo 2, ledger 1, state root BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH, txn root C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby, requests: [('V4SGRU86Z58d6TV7PBUe6f', 1505402052949993)] 2017-09-14 15:14:31,703 | DEBUG | node.py (2305) | executeBatch | Node2 storing 3PC key (0, 2) for ledger 1 range (7, 7) 2017-09-14 15:14:31,703 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:14:36,002 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 39 2017-09-14 15:14:36,002 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:14:36,002 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:14:36,002 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:14:36,003 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:14:36,003 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:14:36,003 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:14:36,003 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 40 to run in 10 seconds 2017-09-14 15:14:36,231 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:14:36,231 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:14:36,231 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:14:37,043 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2C got 1 messages through listener 2017-09-14 15:14:37,044 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1505402077029452 2017-09-14 15:14:37,044 | TRACE | node.py (1388) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe63'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402077029452, 'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2'} 2017-09-14 15:14:37,044 | DEBUG | node.py (1434) | processClientInBox | Node2C processing b'v#ZljK34WWooA6%RgESn.Ox' request SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe63'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402077029452, 'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2'} 2017-09-14 15:14:37,044 | DEBUG | node.py (1708) | processRequest | Node2 received client request: SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe63'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402077029452, 'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2'} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:14:37,044 | TRACE | propagator.py ( 165) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe63'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402077029452, 'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2'} 2017-09-14 15:14:37,045 | INFO | propagator.py ( 148) | propagate | Node2 propagating request ('V4SGRU86Z58d6TV7PBUe6f', 1505402077029452) from client b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:14:37,045 | DEBUG | node.py (2608) | send | Node2 sending message PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe63'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402077029452, 'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} to all recipients: ['Node1', 'Node3'] 2017-09-14 15:14:37,045 | DEBUG | propagator.py ( 214) | forward | Node2 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1505402077029452) to 0 replicas 2017-09-14 15:14:37,045 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402077029452}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:14:37,045 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"request":{"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe63"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402077029452,"signature":"vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node1 2017-09-14 15:14:37,046 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"request":{"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe63"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402077029452,"signature":"vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node1 2017-09-14 15:14:37,046 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"request":{"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe63"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402077029452,"signature":"vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node3 2017-09-14 15:14:37,046 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"request":{"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe63"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402077029452,"signature":"vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node3 2017-09-14 15:14:37,061 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-09-14 15:14:37,062 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1505402077029452 2017-09-14 15:14:37,062 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe63'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402077029452, 'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:14:37,062 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe63'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402077029452, 'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox', 'op': 'PROPAGATE'}, 'Node3') 2017-09-14 15:14:37,062 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe63'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402077029452, 'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:14:37,066 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1505402077029452 2017-09-14 15:14:37,066 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PROPAGATE{'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe63', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402077029452, 'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:14:37,066 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe63', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402077029452, 'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2'}, 'op': 'PROPAGATE', 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'}, 'Node1') 2017-09-14 15:14:37,067 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe63', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402077029452, 'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:14:37,067 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe63'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402077029452, 'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:14:37,067 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe63'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402077029452, 'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2'} 2017-09-14 15:14:37,067 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe63'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402077029452, 'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2'} to its replicas since already forwarded 2017-09-14 15:14:37,067 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe63', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402077029452, 'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:14:37,068 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe63', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402077029452, 'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2'} 2017-09-14 15:14:37,068 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe63', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402077029452, 'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2'} to its replicas since already forwarded 2017-09-14 15:14:37,069 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:14:37,069 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PREPREPARE{'instId': 0, 'discarded': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1505402077029452]], 'digest': 'defbbfd4c1fa5e018f6bd533c9ead325d16bf5ff804d0c53735b190f68f6dbd2', 'txnRootHash': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'stateRootHash': 'DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8', 'ppSeqNo': 3, 'ppTime': 1505402077, 'ledgerId': 1, 'viewNo': 0} 2017-09-14 15:14:37,069 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'discarded': 1, 'instId': 0, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1505402077029452]], 'stateRootHash': 'DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8', 'digest': 'defbbfd4c1fa5e018f6bd533c9ead325d16bf5ff804d0c53735b190f68f6dbd2', 'txnRootHash': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'op': 'PREPREPARE', 'ppTime': 1505402077, 'ppSeqNo': 3, 'ledgerId': 1, 'viewNo': 0}, 'Node1') 2017-09-14 15:14:37,069 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPREPARE{'instId': 0, 'discarded': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1505402077029452]], 'digest': 'defbbfd4c1fa5e018f6bd533c9ead325d16bf5ff804d0c53735b190f68f6dbd2', 'txnRootHash': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'stateRootHash': 'DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8', 'ppSeqNo': 3, 'ppTime': 1505402077, 'ledgerId': 1, 'viewNo': 0} 2017-09-14 15:14:37,070 | DEBUG | replica.py ( 823) | processPrePrepare | Node2:0 received PRE-PREPARE(0, 3) from Node1:0 at 13453.92193861 2017-09-14 15:14:37,071 | DEBUG | replica.py (1034) | validate_pre_prepare | Node2:0 state root before processing PREPREPARE{'instId': 0, 'discarded': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402077029452)], 'digest': 'defbbfd4c1fa5e018f6bd533c9ead325d16bf5ff804d0c53735b190f68f6dbd2', 'txnRootHash': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'stateRootHash': 'DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8', 'ppSeqNo': 3, 'ppTime': 1505402077, 'ledgerId': 1, 'viewNo': 0} is b'\x98MV\xf7\xcf\xb2W\xefE\xef\t*Q\xe7\xf9\xa6tK\xed\xcd\xc4\xc7\xf9\x85\xf7\xcea\xf0\xce\xc8\xb2`', C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby 2017-09-14 15:14:37,072 | DEBUG | replica.py ( 934) | doPrepare | Node2:0 Sending PREPARE(0, 3) at 13453.923638407 2017-09-14 15:14:37,072 | DEBUG | replica.py (2193) | send | Node2:0 sending Prepare 2017-09-14 15:14:37,072 | TRACE | replica.py (2194) | send | Node2:0 sending PREPARE{'instId': 0, 'stateRootHash': 'DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8', 'digest': 'defbbfd4c1fa5e018f6bd533c9ead325d16bf5ff804d0c53735b190f68f6dbd2', 'txnRootHash': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ppTime': 1505402077, 'viewNo': 0} 2017-09-14 15:14:37,072 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 0, 'stateRootHash': 'DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8', 'digest': 'defbbfd4c1fa5e018f6bd533c9ead325d16bf5ff804d0c53735b190f68f6dbd2', 'txnRootHash': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ppTime': 1505402077, 'viewNo': 0} 2017-09-14 15:14:37,072 | DEBUG | replica.py ( 614) | trackBatches | Node2:0 tracking batch for PREPREPARE{'instId': 0, 'discarded': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402077029452)], 'digest': 'defbbfd4c1fa5e018f6bd533c9ead325d16bf5ff804d0c53735b190f68f6dbd2', 'txnRootHash': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'stateRootHash': 'DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8', 'ppSeqNo': 3, 'ppTime': 1505402077, 'ledgerId': 1, 'viewNo': 0} with state root b'\x98MV\xf7\xcf\xb2W\xefE\xef\t*Q\xe7\xf9\xa6tK\xed\xcd\xc4\xc7\xf9\x85\xf7\xcea\xf0\xce\xc8\xb2`' 2017-09-14 15:14:37,073 | DEBUG | replica.py ( 842) | processPrePrepare | Node2:0 processed incoming PRE-PREPARE(0, 3) 2017-09-14 15:14:37,073 | DEBUG | node.py (2608) | send | Node2 sending message PREPARE{'instId': 0, 'stateRootHash': 'DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8', 'digest': 'defbbfd4c1fa5e018f6bd533c9ead325d16bf5ff804d0c53735b190f68f6dbd2', 'txnRootHash': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ppTime': 1505402077, 'viewNo': 0} to all recipients: ['Node1', 'Node3'] 2017-09-14 15:14:37,073 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"ppTime":1505402077,"instId":0,"stateRootHash":"DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8","digest":"defbbfd4c1fa5e018f6bd533c9ead325d16bf5ff804d0c53735b190f68f6dbd2","txnRootHash":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","op":"PREPARE","ppSeqNo":3,"viewNo":0}' to Node1 2017-09-14 15:14:37,074 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"ppTime":1505402077,"instId":0,"stateRootHash":"DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8","digest":"defbbfd4c1fa5e018f6bd533c9ead325d16bf5ff804d0c53735b190f68f6dbd2","txnRootHash":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","op":"PREPARE","ppSeqNo":3,"viewNo":0}' to Node1 2017-09-14 15:14:37,074 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"ppTime":1505402077,"instId":0,"stateRootHash":"DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8","digest":"defbbfd4c1fa5e018f6bd533c9ead325d16bf5ff804d0c53735b190f68f6dbd2","txnRootHash":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","op":"PREPARE","ppSeqNo":3,"viewNo":0}' to Node3 2017-09-14 15:14:37,074 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"ppTime":1505402077,"instId":0,"stateRootHash":"DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8","digest":"defbbfd4c1fa5e018f6bd533c9ead325d16bf5ff804d0c53735b190f68f6dbd2","txnRootHash":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","op":"PREPARE","ppSeqNo":3,"viewNo":0}' to Node3 2017-09-14 15:14:37,086 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:14:37,086 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'instId': 0, 'stateRootHash': 'DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8', 'digest': 'defbbfd4c1fa5e018f6bd533c9ead325d16bf5ff804d0c53735b190f68f6dbd2', 'txnRootHash': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ppTime': 1505402077, 'viewNo': 0} 2017-09-14 15:14:37,087 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'stateRootHash': 'DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8', 'digest': 'defbbfd4c1fa5e018f6bd533c9ead325d16bf5ff804d0c53735b190f68f6dbd2', 'txnRootHash': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'op': 'PREPARE', 'ppTime': 1505402077, 'ppSeqNo': 3, 'viewNo': 0}, 'Node3') 2017-09-14 15:14:37,087 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'instId': 0, 'stateRootHash': 'DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8', 'digest': 'defbbfd4c1fa5e018f6bd533c9ead325d16bf5ff804d0c53735b190f68f6dbd2', 'txnRootHash': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ppTime': 1505402077, 'viewNo': 0} 2017-09-14 15:14:37,087 | DEBUG | replica.py ( 953) | doCommit | Node2:0 Sending COMMIT(0, 3) at 13453.939283432 2017-09-14 15:14:37,087 | DEBUG | replica.py (2193) | send | Node2:0 sending Commit 2017-09-14 15:14:37,088 | TRACE | replica.py (2194) | send | Node2:0 sending COMMIT{'instId': 0, 'ppSeqNo': 3, 'viewNo': 0} 2017-09-14 15:14:37,088 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'ppSeqNo': 3, 'viewNo': 0} commits where f is 0 2017-09-14 15:14:37,088 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (0, 3) 2017-09-14 15:14:37,088 | DEBUG | node.py (2608) | send | Node2 sending message COMMIT{'instId': 0, 'ppSeqNo': 3, 'viewNo': 0} to all recipients: ['Node1', 'Node3'] 2017-09-14 15:14:37,088 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:14:37,089 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 0, 'ppSeqNo': 3, 'viewNo': 0} 2017-09-14 15:14:37,089 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'ppSeqNo': 3, 'viewNo': 0}, 'Node3') 2017-09-14 15:14:37,090 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 3, 'viewNo': 0} 2017-09-14 15:14:37,091 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","ppSeqNo":3,"viewNo":0}' to Node1 2017-09-14 15:14:37,092 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","ppSeqNo":3,"viewNo":0}' to Node1 2017-09-14 15:14:37,092 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","ppSeqNo":3,"viewNo":0}' to Node3 2017-09-14 15:14:37,092 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","ppSeqNo":3,"viewNo":0}' to Node3 2017-09-14 15:14:37,093 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(0, 3) from Node3:0 2017-09-14 15:14:37,093 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'ppSeqNo': 3, 'viewNo': 0} commits where f is 0 2017-09-14 15:14:37,093 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(0, 3) 2017-09-14 15:14:37,106 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:14:37,106 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 0, 'ppSeqNo': 3, 'viewNo': 0} 2017-09-14 15:14:37,107 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'ppSeqNo': 3, 'viewNo': 0}, 'Node1') 2017-09-14 15:14:37,107 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 3, 'viewNo': 0} 2017-09-14 15:14:37,107 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(0, 3) from Node1:0 2017-09-14 15:14:37,107 | TRACE | replica.py ( 925) | tryOrder | Node2:0 returning request to node 2017-09-14 15:14:37,107 | DEBUG | replica.py (1444) | doOrder | Node2:0 ordering COMMIT (0, 3) 2017-09-14 15:14:37,108 | DEBUG | replica.py ( 323) | last_ordered_3pc | Node2:0 set last ordered as (0, 3) 2017-09-14 15:14:37,108 | DEBUG | replica.py (2193) | send | Node2:0 sending Ordered 2017-09-14 15:14:37,108 | TRACE | replica.py (2194) | send | Node2:0 sending ORDERED{'instId': 0, 'stateRootHash': 'DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8', 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402077029452)], 'txnRootHash': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ppTime': 1505402077, 'ledgerId': 1, 'viewNo': 0} 2017-09-14 15:14:37,108 | INFO | replica.py (1483) | order_3pc_key | Node2:0 ordered batch request, view no 0, ppSeqNo 3, ledger 1, state root DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8, txn root 4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1505402077029452)], discarded [] 2017-09-14 15:14:37,108 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(0, 3) 2017-09-14 15:14:37,108 | DEBUG | node.py (1847) | processOrdered | Node2 executing Ordered batch 0 3 of 1 requests 2017-09-14 15:14:37,109 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-09-14 15:14:37,109 | DEBUG | node.py (2386) | sendReplyToClient | Node2 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1505402077029452) to client 2017-09-14 15:14:37,109 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"op":"REPLY","result":{"txnTime":1505402077,"reqId":1505402077029452,"seqNo":8,"rootHash":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","type":"1","identifier":"V4SGRU86Z58d6TV7PBUe6f","dest":"V4SGRU86Z58d6TV7PBUe63","signature":"vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2","auditPath":["BvyY1duMNWfjxw3JuAzptvrLmpLbN3W6vL68oL5CX86f","Dvt18TQs3z1emxZcnmW3jpcjdsycqSQwPFm63UUJzJbs","FA73SaKHCBxBXhf9gis5FxVgz3KxD5GJ1n12yyVYHhDj"]}}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:14:37,109 | INFO | node.py (2290) | executeBatch | Node2 committed batch request, view no 0, ppSeqNo 3, ledger 1, state root DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8, txn root 4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja, requests: [('V4SGRU86Z58d6TV7PBUe6f', 1505402077029452)] 2017-09-14 15:14:37,110 | DEBUG | node.py (2305) | executeBatch | Node2 storing 3PC key (0, 3) for ledger 1 range (8, 8) 2017-09-14 15:14:37,110 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:14:46,011 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 40 2017-09-14 15:14:46,012 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:14:46,012 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:14:46,012 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:14:46,012 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:14:46,012 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:14:46,013 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:14:46,013 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 41 to run in 10 seconds 2017-09-14 15:14:51,236 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:14:51,236 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:14:51,236 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:14:56,024 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 41 2017-09-14 15:14:56,024 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:14:56,024 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:14:56,024 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:14:56,024 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:14:56,025 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:14:56,025 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:14:56,025 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 42 to run in 10 seconds 2017-09-14 15:15:05,746 | TRACE | remote.py ( 119) | hasLostConnection | Remote Node1:HA(host='10.0.0.2', port=9701) has monitor events: [512, 4] 2017-09-14 15:15:05,746 | DEBUG | remote.py ( 123) | hasLostConnection | Node1:HA(host='10.0.0.2', port=9701) found disconnected event on monitor 2017-09-14 15:15:05,747 | DEBUG | keep_in_touch.py ( 68) | conns | Node2's connections changed from {'Node1', 'Node3'} to {'Node3'} 2017-09-14 15:15:05,747 | INFO | keep_in_touch.py ( 93) | _connsChanged | CONNECTION: Node2 disconnected from Node1 2017-09-14 15:15:05,747 | DEBUG | motor.py ( 34) | set_status | Node2 changing status from started to started_hungry 2017-09-14 15:15:05,747 | INFO | node.py ( 823) | onConnsChanged | Node2 lost connection to primary of master 2017-09-14 15:15:05,747 | DEBUG | node.py (2121) | lost_master_primary | Node2 scheduling a view change in 2 sec 2017-09-14 15:15:05,747 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action propose_view_change with id 43 to run in 2 seconds 2017-09-14 15:15:05,747 | DEBUG | node.py ( 941) | checkInstances | Node2 choosing to start election on the basis of count 2 and nodes {'Node3'} 2017-09-14 15:15:05,816 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 35 2017-09-14 15:15:05,816 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node2 checking its request amount 2017-09-14 15:15:05,816 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:15:05,817 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 44 to run in 60 seconds 2017-09-14 15:15:05,817 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 5 2017-09-14 15:15:05,817 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-09-14 15:15:05,817 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 6 to run in 60 seconds 2017-09-14 15:15:06,029 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 42 2017-09-14 15:15:06,029 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:15:06,029 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:15:06,029 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:15:06,029 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:15:06,029 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:15:06,029 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:15:06,030 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 45 to run in 10 seconds 2017-09-14 15:15:06,158 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action dump_json_file with id 37 2017-09-14 15:15:06,160 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action dump_json_file with id 46 to run in 60 seconds 2017-09-14 15:15:06,242 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:15:06,243 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:15:06,243 | DEBUG | zstack.py ( 645) | sendPingPong | Node2 pinged Node1 2017-09-14 15:15:06,244 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:15:06,258 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'pi' to Node1 2017-09-14 15:15:06,259 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'pi' to Node1 2017-09-14 15:15:07,755 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:15:07,756 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} 2017-09-14 15:15:07,756 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'INSTANCE_CHANGE', 'viewNo': 1, 'reason': 26}, 'Node3') 2017-09-14 15:15:07,756 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} 2017-09-14 15:15:07,756 | DEBUG | node.py (1915) | processInstanceChange | Node2 received instance change request: INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} from Node3 2017-09-14 15:15:07,756 | DEBUG | node.py (1958) | do_view_change_if_possible | Node2 has no quorum for view 1 2017-09-14 15:15:07,757 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:15:07,757 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:15:07,757 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:15:07,757 | DEBUG | node.py (1946) | processInstanceChange | Node2 received instance change message INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} but did not find the master to be slow or has already sent an instance change message 2017-09-14 15:15:07,757 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action propose_view_change with id 43 2017-09-14 15:15:07,757 | DEBUG | throttler.py ( 31) | acquire | now: 13484.609335968, len(actionsLog): 0 2017-09-14 15:15:07,758 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 0 2017-09-14 15:15:07,758 | DEBUG | throttler.py ( 39) | acquire | len(actionsLog) was 0, after append, len(actionsLog): 1 2017-09-14 15:15:07,758 | INFO | node.py (2048) | sendInstanceChange | VIEW CHANGE: Node2 sending an instance change with view_no 1 since Primary of master protocol instance disconnected 2017-09-14 15:15:07,758 | INFO | node.py (2051) | sendInstanceChange | MONITORING: Node2 metrics for monitor: Node2 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [13182.643704637] ordered request counts: {0: 3} ordered request durations: {0: 18.85095754899885} master request latencies: {} client avg request latencies: [{'V4SGRU86Z58d6TV7PBUe6f': (3, 6.28365251633295)}] throughput: {0: 0.1591431094257239} master throughput: 0.1591431094257239 total requests: 3 avg backup throughput: None master throughput ratio: None 2017-09-14 15:15:07,758 | DEBUG | node.py (2608) | send | Node2 sending message INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} to all recipients: ['Node1', 'Node3'] 2017-09-14 15:15:07,758 | INFO | node.py (1954) | do_view_change_if_possible | VIEW CHANGE: Node2 initiating a view change to 1 from 0 2017-09-14 15:15:07,759 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action _check_view_change_completed with id 47 to run in 60 seconds 2017-09-14 15:15:07,759 | DEBUG | replica.py ( 455) | on_view_change_start | Node2:0 setting last prepared for master to (0, 3) 2017-09-14 15:15:07,759 | DEBUG | node.py (2146) | startViewChange | Node2 resetting monitor stats after view change 2017-09-14 15:15:07,759 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-09-14 15:15:07,759 | DEBUG | replica.py ( 408) | primaryName | Node2:0 setting primaryName for view no 1 to: None 2017-09-14 15:15:07,760 | INFO | node.py (2159) | startViewChange | VIEW CHANGE: Node2 changed to view 1, will start catchup now 2017-09-14 15:15:07,760 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node2:0 going to remove 0 Ordered messages from outbox 2017-09-14 15:15:07,760 | DEBUG | node.py (1888) | force_process_ordered | Node2 processed 0 Ordered batches for instance 0 before starting catch up 2017-09-14 15:15:07,761 | DEBUG | node.py (2192) | start_catchup | Node2 reverted 0 batches before starting catch up 2017-09-14 15:15:07,761 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node1'] 2017-09-14 15:15:07,761 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node1 for ledger status of ledger 0 2017-09-14 15:15:07,761 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to all recipients: ['Node1', 'Node3'] 2017-09-14 15:15:07,761 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node2 for ledger status of ledger 0 2017-09-14 15:15:07,761 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node3'] 2017-09-14 15:15:07,761 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node3 for ledger status of ledger 0 2017-09-14 15:15:07,761 | INFO | node.py (2108) | propose_view_change | Node2 sent view change since was disconnected from primary for too long 2017-09-14 15:15:07,762 | DEBUG | node.py (1958) | do_view_change_if_possible | Node2 is in higher view more than 1 2017-09-14 15:15:07,762 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 3 msgs to Node1 into one transmission 2017-09-14 15:15:07,762 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"INSTANCE_CHANGE","viewNo":1,"reason":26}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']) 2017-09-14 15:15:07,762 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"INSTANCE_CHANGE\\",\\"viewNo\\":1,\\"reason\\":26}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' 2017-09-14 15:15:07,763 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"INSTANCE_CHANGE\\",\\"viewNo\\":1,\\"reason\\":26}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' to Node1 2017-09-14 15:15:07,763 | DEBUG | zstack.py ( 728) | transmit | Remote Node1 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-09-14 15:15:07,763 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 3 msgs to Node3 into one transmission 2017-09-14 15:15:07,763 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"INSTANCE_CHANGE","viewNo":1,"reason":26}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":0}}']) 2017-09-14 15:15:07,763 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"op\\":\\"INSTANCE_CHANGE\\",\\"viewNo\\":1,\\"reason\\":26}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' 2017-09-14 15:15:07,763 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"INSTANCE_CHANGE\\",\\"viewNo\\":1,\\"reason\\":26}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null}' to Node3 2017-09-14 15:15:07,788 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:15:07,788 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ppSeqNo":null,"txnSeqNo":3,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ppSeqNo":null,"txnSeqNo":3,"viewNo":null}}']} 2017-09-14 15:15:07,789 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ppSeqNo":null,"txnSeqNo":3,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ppSeqNo":null,"txnSeqNo":3,"viewNo":null}}']}, 'Node3') 2017-09-14 15:15:07,789 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ppSeqNo":null,"txnSeqNo":3,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ppSeqNo":null,"txnSeqNo":3,"viewNo":null}}']} 2017-09-14 15:15:07,789 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}, 'msg': {'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None}} 2017-09-14 15:15:07,789 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}, 'msg': {'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None}}, 'Node3') 2017-09-14 15:15:07,789 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}, 'msg': {'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None}} 2017-09-14 15:15:07,789 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}, 'msg': {'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None}} 2017-09-14 15:15:07,790 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}, 'msg': {'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None}}, 'Node3') 2017-09-14 15:15:07,790 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}, 'msg': {'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None}} 2017-09-14 15:15:07,790 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} from Node3 2017-09-14 15:15:07,790 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 3 with 3 2017-09-14 15:15:07,790 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 3 with 3 2017-09-14 15:15:07,791 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} from Node3 2017-09-14 15:15:07,792 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 3 with 3 2017-09-14 15:15:07,792 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 3 with 3 2017-09-14 15:15:10,587 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:15:10,588 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1505402110572745 2017-09-14 15:15:10,588 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe64'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402110572745, 'signature': 'JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:15:10,588 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe64'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402110572745, 'signature': 'JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox', 'op': 'PROPAGATE'}, 'Node3') 2017-09-14 15:15:10,588 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe64'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402110572745, 'signature': 'JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:15:10,588 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe64'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402110572745, 'signature': 'JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:15:10,589 | TRACE | propagator.py ( 165) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe64'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402110572745, 'signature': 'JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF'} 2017-09-14 15:15:10,589 | INFO | propagator.py ( 148) | propagate | Node2 propagating request ('V4SGRU86Z58d6TV7PBUe6f', 1505402110572745) from client v#ZljK34WWooA6%RgESn.Ox 2017-09-14 15:15:10,589 | DEBUG | node.py (2608) | send | Node2 sending message PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe64'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402110572745, 'signature': 'JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} to all recipients: ['Node1', 'Node3'] 2017-09-14 15:15:10,589 | DEBUG | propagator.py ( 214) | forward | Node2 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1505402110572745) to 0 replicas 2017-09-14 15:15:10,590 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2C got 1 messages through listener 2017-09-14 15:15:10,590 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1505402110572745 2017-09-14 15:15:10,590 | TRACE | node.py (1388) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe64'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402110572745, 'signature': 'JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF'} 2017-09-14 15:15:10,591 | DEBUG | node.py (1434) | processClientInBox | Node2C processing b'v#ZljK34WWooA6%RgESn.Ox' request SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe64'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402110572745, 'signature': 'JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF'} 2017-09-14 15:15:10,591 | DEBUG | node.py (1708) | processRequest | Node2 received client request: SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe64'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402110572745, 'signature': 'JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF'} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:15:10,591 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe64'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402110572745, 'signature': 'JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF'} 2017-09-14 15:15:10,593 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe64'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402110572745, 'signature': 'JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF'} to its replicas since already forwarded 2017-09-14 15:15:10,593 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402110572745}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:15:10,594 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"request":{"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe64"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402110572745,"signature":"JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node1 2017-09-14 15:15:10,594 | DEBUG | zstack.py ( 728) | transmit | Remote Node1 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-09-14 15:15:10,594 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"request":{"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe64"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402110572745,"signature":"JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node1 2017-09-14 15:15:10,594 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"request":{"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe64"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402110572745,"signature":"JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node3 2017-09-14 15:15:10,594 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"request":{"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe64"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402110572745,"signature":"JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node3 2017-09-14 15:15:16,038 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 45 2017-09-14 15:15:16,039 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:15:16,039 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 48 to run in 10 seconds 2017-09-14 15:15:21,250 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:15:21,250 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:15:21,250 | DEBUG | zstack.py ( 605) | reconnectRemote | Node2 reconnecting to Node1:HA(host='10.0.0.2', port=9701) 2017-09-14 15:15:21,250 | DEBUG | remote.py ( 89) | disconnect | disconnecting remote Node1:HA(host='10.0.0.2', port=9701) 2017-09-14 15:15:21,251 | TRACE | remote.py ( 92) | disconnect | disconnecting socket 78 41031728 2017-09-14 15:15:21,251 | TRACE | remote.py ( 95) | disconnect | Node1:HA(host='10.0.0.2', port=9701) closing monitor socket 2017-09-14 15:15:21,251 | TRACE | remote.py ( 86) | connect | connecting socket 79 41203248 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-09-14 15:15:21,251 | DEBUG | zstack.py ( 645) | sendPingPong | Node2 pinged Node1 2017-09-14 15:15:21,252 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:15:21,263 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'pi' to Node1 2017-09-14 15:15:21,263 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'pi' to Node1 2017-09-14 15:15:24,448 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:15:24,449 | DEBUG | zstack.py ( 663) | handlePingPong | Node2 got ping from Node1 2017-09-14 15:15:24,449 | DEBUG | zstack.py ( 645) | sendPingPong | Node2 ponged Node1 2017-09-14 15:15:24,449 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'po' to Node1 2017-09-14 15:15:24,449 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'po' to Node1 2017-09-14 15:15:24,496 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-09-14 15:15:24,497 | DEBUG | zstack.py ( 669) | handlePingPong | Node2 got pong from Node1 2017-09-14 15:15:24,497 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: BATCH{'signature': None, 'messages': ['{"primary":[],"op":"CURRENT_STATE","viewNo":0}', '{"txnSeqNo":3,"ppSeqNo":null,"op":"LEDGER_STATUS","viewNo":null,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ledgerId":0}']} 2017-09-14 15:15:24,497 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"primary":[],"op":"CURRENT_STATE","viewNo":0}', '{"txnSeqNo":3,"ppSeqNo":null,"op":"LEDGER_STATUS","viewNo":null,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ledgerId":0}']}, 'Node1') 2017-09-14 15:15:24,497 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"primary":[],"op":"CURRENT_STATE","viewNo":0}', '{"txnSeqNo":3,"ppSeqNo":null,"op":"LEDGER_STATUS","viewNo":null,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ledgerId":0}']} 2017-09-14 15:15:24,497 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: CURRENT_STATE{'primary': [], 'viewNo': 0} 2017-09-14 15:15:24,497 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'CURRENT_STATE', 'primary': [], 'viewNo': 0}, 'Node1') 2017-09-14 15:15:24,497 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox CURRENT_STATE{'primary': [], 'viewNo': 0} 2017-09-14 15:15:24,498 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} 2017-09-14 15:15:24,498 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'ledgerId': 0, 'op': 'LEDGER_STATUS', 'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'viewNo': None}, 'Node1') 2017-09-14 15:15:24,498 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} 2017-09-14 15:15:24,498 | DEBUG | node.py ( 907) | process_current_state_message | Node2 processing current state CURRENT_STATE{'primary': [], 'viewNo': 0} from Node1 2017-09-14 15:15:24,498 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} from Node1 2017-09-14 15:15:24,498 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 3 with 3 2017-09-14 15:15:24,498 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 3 with 3 2017-09-14 15:15:24,499 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node1', 'Node3'} that its ledger of type 0 is latest 2017-09-14 15:15:24,499 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node2 found from ledger status LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} that it does not need catchup 2017-09-14 15:15:24,499 | DEBUG | node.py (1479) | preLedgerCatchUp | Node2 going to process any ordered requests before starting catchup. 2017-09-14 15:15:24,499 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node2:0 going to remove 0 Ordered messages from outbox 2017-09-14 15:15:24,499 | DEBUG | node.py (1888) | force_process_ordered | Node2 processed 0 Ordered batches for instance 0 before starting catch up 2017-09-14 15:15:24,499 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-09-14 15:15:24,499 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-09-14 15:15:24,499 | INFO | node.py (1489) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 0 2017-09-14 15:15:24,499 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 3 with 3 2017-09-14 15:15:24,500 | DEBUG | node.py ( 941) | checkInstances | Node2 choosing to start election on the basis of count 2 and nodes {'Node3'} 2017-09-14 15:15:24,500 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node1'] 2017-09-14 15:15:24,500 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node1 for ledger status of ledger 2 2017-09-14 15:15:24,500 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to all recipients: ['Node1', 'Node3'] 2017-09-14 15:15:24,500 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node2 for ledger status of ledger 2 2017-09-14 15:15:24,500 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-09-14 15:15:24,500 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node3 for ledger status of ledger 2 2017-09-14 15:15:24,501 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node2 going to process 0 stashed ledger statuses for ledger 2 2017-09-14 15:15:24,501 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 0, caught up 0 in total 2017-09-14 15:15:24,501 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-09-14 15:15:24,501 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-09-14 15:15:24,501 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' 2017-09-14 15:15:24,501 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' to Node1 2017-09-14 15:15:24,502 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-09-14 15:15:24,502 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']) 2017-09-14 15:15:24,502 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' 2017-09-14 15:15:24,502 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null}' to Node3 2017-09-14 15:15:24,503 | DEBUG | keep_in_touch.py ( 68) | conns | Node2's connections changed from {'Node3'} to {'Node1', 'Node3'} 2017-09-14 15:15:24,503 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node2 now connected to Node1 2017-09-14 15:15:24,503 | DEBUG | motor.py ( 34) | set_status | Node2 changing status from started_hungry to started 2017-09-14 15:15:24,503 | DEBUG | node.py ( 941) | checkInstances | Node2 choosing to start election on the basis of count 3 and nodes {'Node1', 'Node3'} 2017-09-14 15:15:24,503 | DEBUG | primary_selector.py ( 77) | get_msgs_for_lagged_nodes | Node2 has no ViewChangeDone message to send for view 1 2017-09-14 15:15:24,503 | DEBUG | node.py ( 902) | send_current_state_to_lagging_node | Node2 sending current state CURRENT_STATE{'primary': [], 'viewNo': 1} to lagged node Node1 2017-09-14 15:15:24,504 | DEBUG | node.py (2608) | send | Node2 sending message CURRENT_STATE{'primary': [], 'viewNo': 1} to 1 recipients: ['Node1'] 2017-09-14 15:15:24,504 | DEBUG | node.py (2608) | send | Node2 sending message LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} to 1 recipients: ['Node1'] 2017-09-14 15:15:24,504 | DEBUG | node.py (2608) | send | Node2 sending message LEDGER_STATUS{'ppSeqNo': 3, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'txnSeqNo': 8, 'ledgerId': 1, 'viewNo': 0} to 1 recipients: ['Node1'] 2017-09-14 15:15:24,504 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:15:24,505 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1505402110572745 2017-09-14 15:15:24,506 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe64'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402110572745, 'signature': 'JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:15:24,506 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe64'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402110572745, 'signature': 'JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox', 'op': 'PROPAGATE'}, 'Node1') 2017-09-14 15:15:24,506 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe64'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402110572745, 'signature': 'JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:15:24,506 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe64'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402110572745, 'signature': 'JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:15:24,506 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe64'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402110572745, 'signature': 'JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF'} 2017-09-14 15:15:24,506 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe64'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402110572745, 'signature': 'JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF'} to its replicas since already forwarded 2017-09-14 15:15:24,507 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 3 msgs to Node1 into one transmission 2017-09-14 15:15:24,507 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"CURRENT_STATE","primary":[],"viewNo":1}', b'{"ledgerId":0,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","txnSeqNo":3,"viewNo":null}', b'{"ledgerId":1,"op":"LEDGER_STATUS","ppSeqNo":3,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","txnSeqNo":8,"viewNo":0}']) 2017-09-14 15:15:24,507 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[],\\"viewNo\\":1}","{\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":null,\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"txnSeqNo\\":3,\\"viewNo\\":null}","{\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":3,\\"merkleRoot\\":\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\",\\"txnSeqNo\\":8,\\"viewNo\\":0}"],"signature":null}' 2017-09-14 15:15:24,507 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[],\\"viewNo\\":1}","{\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":null,\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"txnSeqNo\\":3,\\"viewNo\\":null}","{\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":3,\\"merkleRoot\\":\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\",\\"txnSeqNo\\":8,\\"viewNo\\":0}"],"signature":null}' to Node1 2017-09-14 15:15:24,519 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-09-14 15:15:24,520 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']} 2017-09-14 15:15:24,520 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']}, 'Node3') 2017-09-14 15:15:24,520 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']} 2017-09-14 15:15:24,520 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}} 2017-09-14 15:15:24,521 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}}, 'Node3') 2017-09-14 15:15:24,521 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}} 2017-09-14 15:15:24,521 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}} 2017-09-14 15:15:24,521 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}}, 'Node3') 2017-09-14 15:15:24,521 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}} 2017-09-14 15:15:24,521 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}']} 2017-09-14 15:15:24,521 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}']}, 'Node1') 2017-09-14 15:15:24,521 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}']} 2017-09-14 15:15:24,522 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}} 2017-09-14 15:15:24,522 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}}, 'Node1') 2017-09-14 15:15:24,522 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}} 2017-09-14 15:15:24,522 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}} 2017-09-14 15:15:24,522 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}}, 'Node1') 2017-09-14 15:15:24,522 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}} 2017-09-14 15:15:24,523 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None} from Node3 2017-09-14 15:15:24,523 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-09-14 15:15:24,523 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-09-14 15:15:24,523 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None} from Node3 2017-09-14 15:15:24,523 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-09-14 15:15:24,523 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-09-14 15:15:24,524 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None} from Node1 2017-09-14 15:15:24,524 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-09-14 15:15:24,524 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-09-14 15:15:24,524 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node1', 'Node3'} that its ledger of type 2 is latest 2017-09-14 15:15:24,524 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node2 found from ledger status LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None} that it does not need catchup 2017-09-14 15:15:24,524 | DEBUG | node.py (1479) | preLedgerCatchUp | Node2 going to process any ordered requests before starting catchup. 2017-09-14 15:15:24,524 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node2:0 going to remove 0 Ordered messages from outbox 2017-09-14 15:15:24,524 | DEBUG | node.py (1888) | force_process_ordered | Node2 processed 0 Ordered batches for instance 0 before starting catch up 2017-09-14 15:15:24,524 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-09-14 15:15:24,525 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-09-14 15:15:24,525 | INFO | node.py (1489) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 2 2017-09-14 15:15:24,525 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-09-14 15:15:24,525 | DEBUG | pool_config.py ( 35) | processLedger | processing config ledger for any POOL_CONFIGs 2017-09-14 15:15:24,525 | DEBUG | upgrader.py ( 208) | processLedger | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb processing config ledger for any upgrades 2017-09-14 15:15:24,525 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node1'] 2017-09-14 15:15:24,525 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node1 for ledger status of ledger 1 2017-09-14 15:15:24,525 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to all recipients: ['Node1', 'Node3'] 2017-09-14 15:15:24,526 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node2 for ledger status of ledger 1 2017-09-14 15:15:24,526 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-09-14 15:15:24,526 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node3 for ledger status of ledger 1 2017-09-14 15:15:24,526 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node2 going to process 0 stashed ledger statuses for ledger 1 2017-09-14 15:15:24,526 | INFO | upgrader.py ( 150) | should_notify_about_upgrade_result | Node's 'Node2' last upgrade txn is None 2017-09-14 15:15:24,526 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 2, caught up 0 in total 2017-09-14 15:15:24,526 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None} from Node1 2017-09-14 15:15:24,526 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-09-14 15:15:24,527 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-09-14 15:15:24,527 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-09-14 15:15:24,527 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-09-14 15:15:24,527 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' 2017-09-14 15:15:24,527 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' to Node1 2017-09-14 15:15:24,528 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-09-14 15:15:24,528 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', b'{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}']) 2017-09-14 15:15:24,528 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' 2017-09-14 15:15:24,528 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' to Node3 2017-09-14 15:15:24,540 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:15:24,540 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}']} 2017-09-14 15:15:24,540 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}']}, 'Node3') 2017-09-14 15:15:24,540 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}']} 2017-09-14 15:15:24,541 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'ppSeqNo': 3, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'txnSeqNo': 8, 'ledgerId': 1, 'viewNo': 0}} 2017-09-14 15:15:24,541 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'ppSeqNo': 3, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'txnSeqNo': 8, 'ledgerId': 1, 'viewNo': 0}}, 'Node3') 2017-09-14 15:15:24,541 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'ppSeqNo': 3, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'txnSeqNo': 8, 'ledgerId': 1, 'viewNo': 0}} 2017-09-14 15:15:24,541 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'ppSeqNo': 3, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'txnSeqNo': 8, 'ledgerId': 1, 'viewNo': 0}} 2017-09-14 15:15:24,541 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'ppSeqNo': 3, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'txnSeqNo': 8, 'ledgerId': 1, 'viewNo': 0}}, 'Node3') 2017-09-14 15:15:24,541 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'ppSeqNo': 3, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'txnSeqNo': 8, 'ledgerId': 1, 'viewNo': 0}} 2017-09-14 15:15:24,541 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'ppSeqNo': 3, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'txnSeqNo': 8, 'ledgerId': 1, 'viewNo': 0} from Node3 2017-09-14 15:15:24,542 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 8 with 8 2017-09-14 15:15:24,542 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 8 with 8 2017-09-14 15:15:24,542 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'ppSeqNo': 3, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'txnSeqNo': 8, 'ledgerId': 1, 'viewNo': 0} from Node3 2017-09-14 15:15:24,542 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 8 with 8 2017-09-14 15:15:24,542 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 8 with 8 2017-09-14 15:15:24,565 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:15:24,566 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":null,"txnSeqNo":8,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":null,"txnSeqNo":8,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}']} 2017-09-14 15:15:24,566 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":null,"txnSeqNo":8,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":null,"txnSeqNo":8,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}']}, 'Node1') 2017-09-14 15:15:24,566 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":null,"txnSeqNo":8,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":null,"txnSeqNo":8,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}']} 2017-09-14 15:15:24,566 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,566 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node1') 2017-09-14 15:15:24,566 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,566 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,566 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node1') 2017-09-14 15:15:24,567 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,567 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'ppSeqNo': None, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'txnSeqNo': 8, 'ledgerId': 1, 'viewNo': None}} 2017-09-14 15:15:24,567 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'ppSeqNo': None, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'txnSeqNo': 8, 'ledgerId': 1, 'viewNo': None}}, 'Node1') 2017-09-14 15:15:24,567 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'ppSeqNo': None, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'txnSeqNo': 8, 'ledgerId': 1, 'viewNo': None}} 2017-09-14 15:15:24,567 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'ppSeqNo': None, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'txnSeqNo': 8, 'ledgerId': 1, 'viewNo': None}} 2017-09-14 15:15:24,567 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_RESPONSE', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'ppSeqNo': None, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'txnSeqNo': 8, 'ledgerId': 1, 'viewNo': None}}, 'Node1') 2017-09-14 15:15:24,567 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'ppSeqNo': None, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'txnSeqNo': 8, 'ledgerId': 1, 'viewNo': None}} 2017-09-14 15:15:24,568 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}} to 1 recipients: ['Node1'] 2017-09-14 15:15:24,568 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}} to 1 recipients: ['Node1'] 2017-09-14 15:15:24,568 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'txnSeqNo': 8, 'ledgerId': 1, 'viewNo': None} from Node1 2017-09-14 15:15:24,568 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 8 with 8 2017-09-14 15:15:24,569 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 8 with 8 2017-09-14 15:15:24,569 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node1', 'Node3'} that its ledger of type 1 is latest 2017-09-14 15:15:24,569 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node2 found from ledger status LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'txnSeqNo': 8, 'ledgerId': 1, 'viewNo': None} that it does not need catchup 2017-09-14 15:15:24,569 | DEBUG | node.py (1479) | preLedgerCatchUp | Node2 going to process any ordered requests before starting catchup. 2017-09-14 15:15:24,569 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node2:0 going to remove 0 Ordered messages from outbox 2017-09-14 15:15:24,569 | DEBUG | node.py (1888) | force_process_ordered | Node2 processed 0 Ordered batches for instance 0 before starting catch up 2017-09-14 15:15:24,569 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-09-14 15:15:24,569 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-09-14 15:15:24,569 | INFO | node.py (1489) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 1 2017-09-14 15:15:24,570 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 8 with 8 2017-09-14 15:15:24,570 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 1, caught up 0 in total 2017-09-14 15:15:24,570 | DEBUG | node.py (1609) | num_txns_caught_up_in_last_catchup | Node2 caught up to 0 txns in the last catchup 2017-09-14 15:15:24,570 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-09-14 15:15:24,570 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-09-14 15:15:24,570 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node2 needs 3 ViewChangeDone messages 2017-09-14 15:15:24,570 | DEBUG | node.py (1568) | caught_up_for_current_view | Node2 does not have view change quorum for view 1 2017-09-14 15:15:24,570 | DEBUG | node.py (1552) | is_catchup_needed | Node2 is not caught up for the current view 1 2017-09-14 15:15:24,570 | DEBUG | node.py (1609) | num_txns_caught_up_in_last_catchup | Node2 caught up to 0 txns in the last catchup 2017-09-14 15:15:24,570 | DEBUG | node.py (1557) | is_catchup_needed | Node2 ordered till last prepared certificate 2017-09-14 15:15:24,571 | INFO | node.py (1537) | allLedgersCaughtUp | CATCH-UP: Node2 does not need any more catchups 2017-09-14 15:15:24,571 | DEBUG | primary_decider.py ( 134) | send | Node2's elector sending VIEW_CHANGE_DONE{'ledgerInfo': [(0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'name': 'Node2', 'viewNo': 1} 2017-09-14 15:15:24,571 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node2 needs 2 ViewChangeDone messages 2017-09-14 15:15:24,571 | DEBUG | primary_selector.py ( 266) | _startSelection | Node2 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-09-14 15:15:24,571 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'txnSeqNo': 8, 'ledgerId': 1, 'viewNo': None} from Node1 2017-09-14 15:15:24,572 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 8 with 8 2017-09-14 15:15:24,572 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 8 with 8 2017-09-14 15:15:24,572 | DEBUG | node.py (2608) | send | Node2 sending message VIEW_CHANGE_DONE{'ledgerInfo': [(0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'name': 'Node2', 'viewNo': 1} to all recipients: ['Node1', 'Node3'] 2017-09-14 15:15:24,573 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 3 msgs to Node1 into one transmission 2017-09-14 15:15:24,573 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', b'{"op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,8,"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node2","viewNo":1}']) 2017-09-14 15:15:24,573 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"VIEW_CHANGE_DONE\\",\\"ledgerInfo\\":[[0,3,\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\"],[1,8,\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"name\\":\\"Node2\\",\\"viewNo\\":1}"],"signature":null}' 2017-09-14 15:15:24,573 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"VIEW_CHANGE_DONE\\",\\"ledgerInfo\\":[[0,3,\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\"],[1,8,\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"name\\":\\"Node2\\",\\"viewNo\\":1}"],"signature":null}' to Node1 2017-09-14 15:15:24,573 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,8,"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node2","viewNo":1}' to Node3 2017-09-14 15:15:24,574 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,8,"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node2","viewNo":1}' to Node3 2017-09-14 15:15:24,585 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:15:24,586 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":0}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":0}}']} 2017-09-14 15:15:24,586 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":0}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":0}}']}, 'Node3') 2017-09-14 15:15:24,586 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":0}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":0}}']} 2017-09-14 15:15:24,586 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-09-14 15:15:24,586 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node3') 2017-09-14 15:15:24,586 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-09-14 15:15:24,586 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-09-14 15:15:24,586 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node3') 2017-09-14 15:15:24,587 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-09-14 15:15:24,587 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}, 'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None}} to 1 recipients: ['Node3'] 2017-09-14 15:15:24,587 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}, 'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None}} to 1 recipients: ['Node3'] 2017-09-14 15:15:24,588 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-09-14 15:15:24,588 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ppSeqNo":null,"txnSeqNo":3,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ppSeqNo":null,"txnSeqNo":3,"viewNo":null}}']) 2017-09-14 15:15:24,588 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0},\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":3,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0},\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":3,\\"viewNo\\":null}}"],"signature":null}' 2017-09-14 15:15:24,588 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0},\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":3,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0},\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":3,\\"viewNo\\":null}}"],"signature":null}' to Node3 2017-09-14 15:15:24,613 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:15:24,613 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"op":"MESSAGE_REQUEST"}', '{"name":"Node1","ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,8,"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","viewNo":0}']} 2017-09-14 15:15:24,613 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"op":"MESSAGE_REQUEST"}', '{"name":"Node1","ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,8,"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","viewNo":0}']}, 'Node1') 2017-09-14 15:15:24,614 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"op":"MESSAGE_REQUEST"}', '{"name":"Node1","ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,8,"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","viewNo":0}']} 2017-09-14 15:15:24,614 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,614 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node1') 2017-09-14 15:15:24,614 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,614 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,614 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node1') 2017-09-14 15:15:24,614 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,615 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-09-14 15:15:24,615 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node1') 2017-09-14 15:15:24,615 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-09-14 15:15:24,615 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-09-14 15:15:24,615 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}}, 'Node1') 2017-09-14 15:15:24,615 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-09-14 15:15:24,616 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-09-14 15:15:24,616 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'VIEW_CHANGE_DONE', 'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0}, 'Node1') 2017-09-14 15:15:24,616 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-09-14 15:15:24,616 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': LEDGER_STATUS{'ppSeqNo': 3, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'txnSeqNo': 8, 'ledgerId': 1, 'viewNo': 0}} to 1 recipients: ['Node1'] 2017-09-14 15:15:24,617 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': LEDGER_STATUS{'ppSeqNo': 3, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'txnSeqNo': 8, 'ledgerId': 1, 'viewNo': 0}} to 1 recipients: ['Node1'] 2017-09-14 15:15:24,617 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}, 'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None}} to 1 recipients: ['Node1'] 2017-09-14 15:15:24,617 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}, 'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None}} to 1 recipients: ['Node1'] 2017-09-14 15:15:24,618 | DEBUG | node.py (1205) | sendToElector | Node2 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0}, 'Node1') 2017-09-14 15:15:24,618 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0}, 'Node1') because its view no 0 is less than the elector's 1 2017-09-14 15:15:24,618 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 4 msgs to Node1 into one transmission 2017-09-14 15:15:24,618 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}', b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}', b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ppSeqNo":null,"txnSeqNo":3,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ppSeqNo":null,"txnSeqNo":3,"viewNo":null}}']) 2017-09-14 15:15:24,618 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\",\\"ppSeqNo\\":3,\\"txnSeqNo\\":8,\\"viewNo\\":0}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\",\\"ppSeqNo\\":3,\\"txnSeqNo\\":8,\\"viewNo\\":0}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0},\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":3,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0},\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":3,\\"viewNo\\":null}}"],"signature":null}' 2017-09-14 15:15:24,619 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\",\\"ppSeqNo\\":3,\\"txnSeqNo\\":8,\\"viewNo\\":0}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\",\\"ppSeqNo\\":3,\\"txnSeqNo\\":8,\\"viewNo\\":0}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0},\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":3,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0},\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":3,\\"viewNo\\":null}}"],"signature":null}' to Node1 2017-09-14 15:15:24,631 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:15:24,631 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}']} 2017-09-14 15:15:24,631 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}']}, 'Node3') 2017-09-14 15:15:24,631 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}']} 2017-09-14 15:15:24,631 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,632 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node3') 2017-09-14 15:15:24,632 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,632 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,632 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node3') 2017-09-14 15:15:24,632 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,632 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}} to 1 recipients: ['Node3'] 2017-09-14 15:15:24,633 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}} to 1 recipients: ['Node3'] 2017-09-14 15:15:24,633 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-09-14 15:15:24,633 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']) 2017-09-14 15:15:24,633 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}"],"signature":null}' 2017-09-14 15:15:24,634 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}"],"signature":null}' to Node3 2017-09-14 15:15:24,636 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:15:24,637 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}']} 2017-09-14 15:15:24,637 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}']}, 'Node1') 2017-09-14 15:15:24,637 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}']} 2017-09-14 15:15:24,637 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,637 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node1') 2017-09-14 15:15:24,637 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,637 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,638 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node1') 2017-09-14 15:15:24,638 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,638 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}} to 1 recipients: ['Node1'] 2017-09-14 15:15:24,638 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}} to 1 recipients: ['Node1'] 2017-09-14 15:15:24,639 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-09-14 15:15:24,639 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']) 2017-09-14 15:15:24,639 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}"],"signature":null}' 2017-09-14 15:15:24,639 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}"],"signature":null}' to Node1 2017-09-14 15:15:24,651 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:15:24,651 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}']} 2017-09-14 15:15:24,652 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}']}, 'Node3') 2017-09-14 15:15:24,652 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}']} 2017-09-14 15:15:24,652 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,652 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node3') 2017-09-14 15:15:24,652 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,652 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,652 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node3') 2017-09-14 15:15:24,652 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,653 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': LEDGER_STATUS{'ppSeqNo': 3, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'txnSeqNo': 8, 'ledgerId': 1, 'viewNo': 0}} to 1 recipients: ['Node3'] 2017-09-14 15:15:24,653 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': LEDGER_STATUS{'ppSeqNo': 3, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'txnSeqNo': 8, 'ledgerId': 1, 'viewNo': 0}} to 1 recipients: ['Node3'] 2017-09-14 15:15:24,654 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-09-14 15:15:24,654 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}', b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}']) 2017-09-14 15:15:24,654 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\",\\"ppSeqNo\\":3,\\"txnSeqNo\\":8,\\"viewNo\\":0}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\",\\"ppSeqNo\\":3,\\"txnSeqNo\\":8,\\"viewNo\\":0}}"],"signature":null}' 2017-09-14 15:15:24,654 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\",\\"ppSeqNo\\":3,\\"txnSeqNo\\":8,\\"viewNo\\":0}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\",\\"ppSeqNo\\":3,\\"txnSeqNo\\":8,\\"viewNo\\":0}}"],"signature":null}' to Node3 2017-09-14 15:15:24,681 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:15:24,681 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"name":"Node2","ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,8,"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","viewNo":1}']} 2017-09-14 15:15:24,682 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"name":"Node2","ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,8,"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","viewNo":1}']}, 'Node1') 2017-09-14 15:15:24,682 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"name":"Node2","ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,8,"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","viewNo":1}']} 2017-09-14 15:15:24,682 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,682 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node1') 2017-09-14 15:15:24,682 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,682 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,682 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node1') 2017-09-14 15:15:24,682 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,683 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1} 2017-09-14 15:15:24,683 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'VIEW_CHANGE_DONE', 'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1}, 'Node1') 2017-09-14 15:15:24,683 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1} 2017-09-14 15:15:24,683 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': LEDGER_STATUS{'ppSeqNo': 3, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'txnSeqNo': 8, 'ledgerId': 1, 'viewNo': 0}} to 1 recipients: ['Node1'] 2017-09-14 15:15:24,684 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': LEDGER_STATUS{'ppSeqNo': 3, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'txnSeqNo': 8, 'ledgerId': 1, 'viewNo': 0}} to 1 recipients: ['Node1'] 2017-09-14 15:15:24,684 | DEBUG | node.py (1205) | sendToElector | Node2 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1}, 'Node1') 2017-09-14 15:15:24,684 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node2's primary selector started processing of ViewChangeDone msg from Node1 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1} 2017-09-14 15:15:24,684 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node2 needs 1 ViewChangeDone messages 2017-09-14 15:15:24,684 | DEBUG | primary_selector.py ( 266) | _startSelection | Node2 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-09-14 15:15:24,685 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-09-14 15:15:24,685 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}', b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}']) 2017-09-14 15:15:24,685 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\",\\"ppSeqNo\\":3,\\"txnSeqNo\\":8,\\"viewNo\\":0}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\",\\"ppSeqNo\\":3,\\"txnSeqNo\\":8,\\"viewNo\\":0}}"],"signature":null}' 2017-09-14 15:15:24,685 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\",\\"ppSeqNo\\":3,\\"txnSeqNo\\":8,\\"viewNo\\":0}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\",\\"ppSeqNo\\":3,\\"txnSeqNo\\":8,\\"viewNo\\":0}}"],"signature":null}' to Node1 2017-09-14 15:15:24,697 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:15:24,698 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1} 2017-09-14 15:15:24,698 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'VIEW_CHANGE_DONE', 'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1}, 'Node3') 2017-09-14 15:15:24,698 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1} 2017-09-14 15:15:24,698 | DEBUG | node.py (1205) | sendToElector | Node2 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1}, 'Node3') 2017-09-14 15:15:24,699 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node2's primary selector started processing of ViewChangeDone msg from Node3 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1} 2017-09-14 15:15:24,699 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node2 got view change quorum (3 >= 3) 2017-09-14 15:15:24,699 | DEBUG | primary_selector.py ( 215) | has_view_change_from_primary | Node2 received ViewChangeDone from primary Node2 2017-09-14 15:15:24,699 | DEBUG | primary_selector.py ( 238) | has_sufficient_same_view_change_done_messages | Node2 found acceptable primary Node2 and ledger info ((0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')) 2017-09-14 15:15:24,699 | DEBUG | primary_selector.py ( 281) | _startSelection | Node2 starting selection 2017-09-14 15:15:24,699 | DISPLAY | primary_selector.py ( 291) | _startSelection | PRIMARY SELECTION: Node2:0 selected primary Node2:0 for instance 0 (view 1) 2017-09-14 15:15:24,699 | INFO | node.py ( 510) | start_participating | Node2 started participating 2017-09-14 15:15:24,700 | DEBUG | replica.py ( 408) | primaryName | Node2:0 setting primaryName for view no 1 to: Node2:0 2017-09-14 15:15:24,700 | DEBUG | replica.py (1694) | _gc | Node2:0 cleaning up till (0, 3) 2017-09-14 15:15:24,700 | DEBUG | replica.py (1709) | _gc | Node2:0 found 3 3-phase keys to clean 2017-09-14 15:15:24,700 | DEBUG | replica.py (1711) | _gc | Node2:0 found 3 request keys to clean 2017-09-14 15:15:24,700 | DEBUG | replica.py (1733) | _gc | Node2:0 clearing request ('V4SGRU86Z58d6TV7PBUe6f', 1505402077029452) from previous checkpoints 2017-09-14 15:15:24,700 | DEBUG | replica.py (1733) | _gc | Node2:0 clearing request ('V4SGRU86Z58d6TV7PBUe6f', 1505402052949993) from previous checkpoints 2017-09-14 15:15:24,700 | DEBUG | replica.py (1733) | _gc | Node2:0 clearing request ('V4SGRU86Z58d6TV7PBUe6f', 1505402016669500) from previous checkpoints 2017-09-14 15:15:24,700 | DEBUG | replica.py ( 313) | h | Node2:0 set watermarks as 0 300 2017-09-14 15:15:24,701 | DISPLAY | primary_selector.py ( 315) | _startSelection | VIEW CHANGE: Node2:0 declares view change 1 as completed for instance 0, new primary is Node2:0, ledger info is [(0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-09-14 15:15:24,701 | DEBUG | replica.py ( 667) | create3PCBatch | Node2:0 creating batch 1 for ledger 1 with state root b'\xba\xacPz1\x97K\xccr\xbe\xdc\x96\x95\x0b\xdb\x85y2\x03\xb9\xeeI\xda\xbd\x87\x8e\xf8:Gc\xa3S' 2017-09-14 15:15:24,703 | DEBUG | replica.py ( 700) | create3PCBatch | Node2:0 created a PRE-PREPARE with 1 requests for ledger 1 2017-09-14 15:15:24,703 | DEBUG | replica.py (2193) | send | Node2:0 sending PrePrepare 2017-09-14 15:15:24,703 | TRACE | replica.py (2194) | send | Node2:0 sending PREPREPARE{'instId': 0, 'discarded': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402110572745)], 'digest': '22c727caf41ca2e8f715bcfcf11f8730a7dc29d6ccc2553d64dc9a1e97f34261', 'txnRootHash': '8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm', 'stateRootHash': '5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq', 'ppSeqNo': 1, 'ppTime': 1505402124, 'ledgerId': 1, 'viewNo': 1} 2017-09-14 15:15:24,704 | DEBUG | replica.py ( 614) | trackBatches | Node2:0 tracking batch for PREPREPARE{'instId': 0, 'discarded': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402110572745)], 'digest': '22c727caf41ca2e8f715bcfcf11f8730a7dc29d6ccc2553d64dc9a1e97f34261', 'txnRootHash': '8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm', 'stateRootHash': '5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq', 'ppSeqNo': 1, 'ppTime': 1505402124, 'ledgerId': 1, 'viewNo': 1} with state root b'\xba\xacPz1\x97K\xccr\xbe\xdc\x96\x95\x0b\xdb\x85y2\x03\xb9\xeeI\xda\xbd\x87\x8e\xf8:Gc\xa3S' 2017-09-14 15:15:24,705 | DEBUG | node.py (2608) | send | Node2 sending message PREPREPARE{'instId': 0, 'discarded': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402110572745)], 'digest': '22c727caf41ca2e8f715bcfcf11f8730a7dc29d6ccc2553d64dc9a1e97f34261', 'txnRootHash': '8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm', 'stateRootHash': '5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq', 'ppSeqNo': 1, 'ppTime': 1505402124, 'ledgerId': 1, 'viewNo': 1} to all recipients: ['Node1', 'Node3'] 2017-09-14 15:15:24,706 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"discarded":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1505402110572745]],"ppTime":1505402124,"instId":0,"stateRootHash":"5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq","digest":"22c727caf41ca2e8f715bcfcf11f8730a7dc29d6ccc2553d64dc9a1e97f34261","txnRootHash":"8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm","op":"PREPREPARE","ppSeqNo":1,"ledgerId":1,"viewNo":1}' to Node1 2017-09-14 15:15:24,706 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"discarded":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1505402110572745]],"ppTime":1505402124,"instId":0,"stateRootHash":"5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq","digest":"22c727caf41ca2e8f715bcfcf11f8730a7dc29d6ccc2553d64dc9a1e97f34261","txnRootHash":"8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm","op":"PREPREPARE","ppSeqNo":1,"ledgerId":1,"viewNo":1}' to Node1 2017-09-14 15:15:24,707 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"discarded":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1505402110572745]],"ppTime":1505402124,"instId":0,"stateRootHash":"5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq","digest":"22c727caf41ca2e8f715bcfcf11f8730a7dc29d6ccc2553d64dc9a1e97f34261","txnRootHash":"8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm","op":"PREPREPARE","ppSeqNo":1,"ledgerId":1,"viewNo":1}' to Node3 2017-09-14 15:15:24,707 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"discarded":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1505402110572745]],"ppTime":1505402124,"instId":0,"stateRootHash":"5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq","digest":"22c727caf41ca2e8f715bcfcf11f8730a7dc29d6ccc2553d64dc9a1e97f34261","txnRootHash":"8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm","op":"PREPREPARE","ppSeqNo":1,"ledgerId":1,"viewNo":1}' to Node3 2017-09-14 15:15:24,718 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:15:24,719 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PREPARE{'instId': 0, 'stateRootHash': '5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq', 'digest': '22c727caf41ca2e8f715bcfcf11f8730a7dc29d6ccc2553d64dc9a1e97f34261', 'txnRootHash': '8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm', 'ppSeqNo': 1, 'ppTime': 1505402124, 'viewNo': 1} 2017-09-14 15:15:24,719 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'stateRootHash': '5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq', 'digest': '22c727caf41ca2e8f715bcfcf11f8730a7dc29d6ccc2553d64dc9a1e97f34261', 'txnRootHash': '8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm', 'op': 'PREPARE', 'ppSeqNo': 1, 'ppTime': 1505402124, 'viewNo': 1}, 'Node1') 2017-09-14 15:15:24,719 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'instId': 0, 'stateRootHash': '5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq', 'digest': '22c727caf41ca2e8f715bcfcf11f8730a7dc29d6ccc2553d64dc9a1e97f34261', 'txnRootHash': '8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm', 'ppSeqNo': 1, 'ppTime': 1505402124, 'viewNo': 1} 2017-09-14 15:15:24,720 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 0, 'stateRootHash': '5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq', 'digest': '22c727caf41ca2e8f715bcfcf11f8730a7dc29d6ccc2553d64dc9a1e97f34261', 'txnRootHash': '8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm', 'ppSeqNo': 1, 'ppTime': 1505402124, 'viewNo': 1} 2017-09-14 15:15:24,720 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (1, 1) 2017-09-14 15:15:24,733 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-09-14 15:15:24,733 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'instId': 0, 'stateRootHash': '5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq', 'digest': '22c727caf41ca2e8f715bcfcf11f8730a7dc29d6ccc2553d64dc9a1e97f34261', 'txnRootHash': '8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm', 'ppSeqNo': 1, 'ppTime': 1505402124, 'viewNo': 1} 2017-09-14 15:15:24,733 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'stateRootHash': '5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq', 'digest': '22c727caf41ca2e8f715bcfcf11f8730a7dc29d6ccc2553d64dc9a1e97f34261', 'txnRootHash': '8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm', 'op': 'PREPARE', 'ppTime': 1505402124, 'ppSeqNo': 1, 'viewNo': 1}, 'Node3') 2017-09-14 15:15:24,733 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'instId': 0, 'stateRootHash': '5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq', 'digest': '22c727caf41ca2e8f715bcfcf11f8730a7dc29d6ccc2553d64dc9a1e97f34261', 'txnRootHash': '8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm', 'ppSeqNo': 1, 'ppTime': 1505402124, 'viewNo': 1} 2017-09-14 15:15:24,734 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 1} 2017-09-14 15:15:24,734 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'ppSeqNo': 1, 'viewNo': 1}, 'Node3') 2017-09-14 15:15:24,734 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 1} 2017-09-14 15:15:24,735 | DEBUG | replica.py ( 953) | doCommit | Node2:0 Sending COMMIT(1, 1) at 13501.586464871 2017-09-14 15:15:24,735 | DEBUG | replica.py (2193) | send | Node2:0 sending Commit 2017-09-14 15:15:24,735 | TRACE | replica.py (2194) | send | Node2:0 sending COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 1} 2017-09-14 15:15:24,735 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 1} commits where f is 0 2017-09-14 15:15:24,735 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (1, 1) 2017-09-14 15:15:24,735 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(1, 1) from Node3:0 2017-09-14 15:15:24,735 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 1} commits where f is 0 2017-09-14 15:15:24,735 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(1, 1) 2017-09-14 15:15:24,735 | DEBUG | node.py (2608) | send | Node2 sending message COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 1} to all recipients: ['Node1', 'Node3'] 2017-09-14 15:15:24,736 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","ppSeqNo":1,"viewNo":1}' to Node1 2017-09-14 15:15:24,736 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","ppSeqNo":1,"viewNo":1}' to Node1 2017-09-14 15:15:24,736 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","ppSeqNo":1,"viewNo":1}' to Node3 2017-09-14 15:15:24,737 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","ppSeqNo":1,"viewNo":1}' to Node3 2017-09-14 15:15:24,749 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:15:24,750 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 1} 2017-09-14 15:15:24,750 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'ppSeqNo': 1, 'viewNo': 1}, 'Node1') 2017-09-14 15:15:24,750 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 1} 2017-09-14 15:15:24,751 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(1, 1) from Node1:0 2017-09-14 15:15:24,751 | TRACE | replica.py ( 925) | tryOrder | Node2:0 returning request to node 2017-09-14 15:15:24,751 | DEBUG | replica.py (1444) | doOrder | Node2:0 ordering COMMIT (1, 1) 2017-09-14 15:15:24,751 | DEBUG | replica.py ( 323) | last_ordered_3pc | Node2:0 set last ordered as (1, 1) 2017-09-14 15:15:24,751 | DEBUG | replica.py (2193) | send | Node2:0 sending Ordered 2017-09-14 15:15:24,751 | TRACE | replica.py (2194) | send | Node2:0 sending ORDERED{'instId': 0, 'stateRootHash': '5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq', 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402110572745)], 'txnRootHash': '8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm', 'ppSeqNo': 1, 'ppTime': 1505402124, 'ledgerId': 1, 'viewNo': 1} 2017-09-14 15:15:24,751 | INFO | replica.py (1483) | order_3pc_key | Node2:0 ordered batch request, view no 1, ppSeqNo 1, ledger 1, state root 5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq, txn root 8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1505402110572745)], discarded [] 2017-09-14 15:15:24,752 | DEBUG | replica.py (1556) | _newCheckpointState | Node2:0 adding new checkpoint state for (1, 100) 2017-09-14 15:15:24,752 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(1, 1) 2017-09-14 15:15:24,752 | DEBUG | node.py (1847) | processOrdered | Node2 executing Ordered batch 1 1 of 1 requests 2017-09-14 15:15:24,752 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-09-14 15:15:24,753 | DEBUG | node.py (2386) | sendReplyToClient | Node2 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1505402110572745) to client 2017-09-14 15:15:24,753 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"op":"REPLY","result":{"txnTime":1505402124,"reqId":1505402110572745,"seqNo":9,"rootHash":"8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm","type":"1","identifier":"V4SGRU86Z58d6TV7PBUe6f","dest":"V4SGRU86Z58d6TV7PBUe64","signature":"JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF","auditPath":["4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja"]}}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:15:24,753 | INFO | node.py (2290) | executeBatch | Node2 committed batch request, view no 1, ppSeqNo 1, ledger 1, state root 5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq, txn root 8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm, requests: [('V4SGRU86Z58d6TV7PBUe6f', 1505402110572745)] 2017-09-14 15:15:24,753 | DEBUG | node.py (2305) | executeBatch | Node2 storing 3PC key (1, 1) for ledger 1 range (9, 9) 2017-09-14 15:15:24,753 | DEBUG | monitor.py ( 233) | requestOrdered | Got ordered request with identifier V4SGRU86Z58d6TV7PBUe6f and reqId 1505402110572745 but it was from a previous view 2017-09-14 15:15:24,753 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:15:26,043 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 48 2017-09-14 15:15:26,043 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:15:26,043 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:15:26,044 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:15:26,044 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:15:26,044 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:15:26,044 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:15:26,044 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 49 to run in 10 seconds 2017-09-14 15:15:27,743 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-09-14 15:15:27,744 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1505402127709038 2017-09-14 15:15:27,744 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe65'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402127709038, 'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:15:27,744 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe65'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402127709038, 'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox', 'op': 'PROPAGATE'}, 'Node3') 2017-09-14 15:15:27,744 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe65'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402127709038, 'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:15:27,746 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1505402127709038 2017-09-14 15:15:27,746 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe65'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402127709038, 'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:15:27,747 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe65'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402127709038, 'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox', 'op': 'PROPAGATE'}, 'Node1') 2017-09-14 15:15:27,747 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe65'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402127709038, 'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:15:27,747 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe65'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402127709038, 'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:15:27,747 | TRACE | propagator.py ( 165) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe65'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402127709038, 'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D'} 2017-09-14 15:15:27,747 | INFO | propagator.py ( 148) | propagate | Node2 propagating request ('V4SGRU86Z58d6TV7PBUe6f', 1505402127709038) from client v#ZljK34WWooA6%RgESn.Ox 2017-09-14 15:15:27,747 | DEBUG | node.py (2608) | send | Node2 sending message PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe65'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402127709038, 'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} to all recipients: ['Node1', 'Node3'] 2017-09-14 15:15:27,747 | DEBUG | propagator.py ( 214) | forward | Node2 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1505402127709038) to 0 replicas 2017-09-14 15:15:27,748 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe65'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402127709038, 'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:15:27,748 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe65'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402127709038, 'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D'} 2017-09-14 15:15:27,748 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe65'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402127709038, 'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D'} to its replicas since already forwarded 2017-09-14 15:15:27,748 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2C got 1 messages through listener 2017-09-14 15:15:27,749 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1505402127709038 2017-09-14 15:15:27,749 | TRACE | node.py (1388) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe65'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402127709038, 'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D'} 2017-09-14 15:15:27,749 | DEBUG | node.py (1434) | processClientInBox | Node2C processing b'v#ZljK34WWooA6%RgESn.Ox' request SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe65'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402127709038, 'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D'} 2017-09-14 15:15:27,749 | DEBUG | node.py (1708) | processRequest | Node2 received client request: SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe65'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402127709038, 'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D'} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:15:27,750 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe65'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402127709038, 'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D'} 2017-09-14 15:15:27,750 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe65'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402127709038, 'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D'} to its replicas since already forwarded 2017-09-14 15:15:27,750 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402127709038}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:15:27,750 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"request":{"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe65"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402127709038,"signature":"Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node1 2017-09-14 15:15:27,750 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"request":{"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe65"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402127709038,"signature":"Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node1 2017-09-14 15:15:27,751 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"request":{"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe65"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402127709038,"signature":"Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node3 2017-09-14 15:15:27,751 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"request":{"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe65"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402127709038,"signature":"Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node3 2017-09-14 15:15:27,751 | DEBUG | replica.py ( 667) | create3PCBatch | Node2:0 creating batch 2 for ledger 1 with state root b'Ek\x11\xa6O\xa30u A\x81\x91\x97)\x94\x88}e\xd6\x17\xcfO\x13\xda\xe5\x91\x9e\x0e\xa4<\x93\xa6' 2017-09-14 15:15:27,753 | DEBUG | replica.py ( 700) | create3PCBatch | Node2:0 created a PRE-PREPARE with 1 requests for ledger 1 2017-09-14 15:15:27,753 | DEBUG | replica.py (2193) | send | Node2:0 sending PrePrepare 2017-09-14 15:15:27,753 | TRACE | replica.py (2194) | send | Node2:0 sending PREPREPARE{'instId': 0, 'discarded': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402127709038)], 'digest': '89ccc072336b1421ffb726dffb332d7d90ed24649d0dc981ddfc174581b650de', 'txnRootHash': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'stateRootHash': '2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf', 'ppSeqNo': 2, 'ppTime': 1505402127, 'ledgerId': 1, 'viewNo': 1} 2017-09-14 15:15:27,753 | DEBUG | replica.py ( 614) | trackBatches | Node2:0 tracking batch for PREPREPARE{'instId': 0, 'discarded': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402127709038)], 'digest': '89ccc072336b1421ffb726dffb332d7d90ed24649d0dc981ddfc174581b650de', 'txnRootHash': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'stateRootHash': '2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf', 'ppSeqNo': 2, 'ppTime': 1505402127, 'ledgerId': 1, 'viewNo': 1} with state root b'Ek\x11\xa6O\xa30u A\x81\x91\x97)\x94\x88}e\xd6\x17\xcfO\x13\xda\xe5\x91\x9e\x0e\xa4<\x93\xa6' 2017-09-14 15:15:27,753 | DEBUG | node.py (2608) | send | Node2 sending message PREPREPARE{'instId': 0, 'discarded': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402127709038)], 'digest': '89ccc072336b1421ffb726dffb332d7d90ed24649d0dc981ddfc174581b650de', 'txnRootHash': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'stateRootHash': '2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf', 'ppSeqNo': 2, 'ppTime': 1505402127, 'ledgerId': 1, 'viewNo': 1} to all recipients: ['Node1', 'Node3'] 2017-09-14 15:15:27,754 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"discarded":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1505402127709038]],"ppTime":1505402127,"instId":0,"stateRootHash":"2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf","digest":"89ccc072336b1421ffb726dffb332d7d90ed24649d0dc981ddfc174581b650de","txnRootHash":"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq","op":"PREPREPARE","ppSeqNo":2,"ledgerId":1,"viewNo":1}' to Node1 2017-09-14 15:15:27,754 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"discarded":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1505402127709038]],"ppTime":1505402127,"instId":0,"stateRootHash":"2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf","digest":"89ccc072336b1421ffb726dffb332d7d90ed24649d0dc981ddfc174581b650de","txnRootHash":"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq","op":"PREPREPARE","ppSeqNo":2,"ledgerId":1,"viewNo":1}' to Node1 2017-09-14 15:15:27,754 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"discarded":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1505402127709038]],"ppTime":1505402127,"instId":0,"stateRootHash":"2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf","digest":"89ccc072336b1421ffb726dffb332d7d90ed24649d0dc981ddfc174581b650de","txnRootHash":"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq","op":"PREPREPARE","ppSeqNo":2,"ledgerId":1,"viewNo":1}' to Node3 2017-09-14 15:15:27,755 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"discarded":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1505402127709038]],"ppTime":1505402127,"instId":0,"stateRootHash":"2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf","digest":"89ccc072336b1421ffb726dffb332d7d90ed24649d0dc981ddfc174581b650de","txnRootHash":"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq","op":"PREPREPARE","ppSeqNo":2,"ledgerId":1,"viewNo":1}' to Node3 2017-09-14 15:15:27,785 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 3 messages through listener 2017-09-14 15:15:27,785 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'instId': 0, 'stateRootHash': '2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf', 'digest': '89ccc072336b1421ffb726dffb332d7d90ed24649d0dc981ddfc174581b650de', 'txnRootHash': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'ppSeqNo': 2, 'ppTime': 1505402127, 'viewNo': 1} 2017-09-14 15:15:27,785 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'stateRootHash': '2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf', 'digest': '89ccc072336b1421ffb726dffb332d7d90ed24649d0dc981ddfc174581b650de', 'txnRootHash': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'op': 'PREPARE', 'ppTime': 1505402127, 'ppSeqNo': 2, 'viewNo': 1}, 'Node3') 2017-09-14 15:15:27,786 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'instId': 0, 'stateRootHash': '2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf', 'digest': '89ccc072336b1421ffb726dffb332d7d90ed24649d0dc981ddfc174581b650de', 'txnRootHash': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'ppSeqNo': 2, 'ppTime': 1505402127, 'viewNo': 1} 2017-09-14 15:15:27,786 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PREPARE{'instId': 0, 'stateRootHash': '2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf', 'digest': '89ccc072336b1421ffb726dffb332d7d90ed24649d0dc981ddfc174581b650de', 'txnRootHash': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'ppSeqNo': 2, 'ppTime': 1505402127, 'viewNo': 1} 2017-09-14 15:15:27,786 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'stateRootHash': '2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf', 'digest': '89ccc072336b1421ffb726dffb332d7d90ed24649d0dc981ddfc174581b650de', 'txnRootHash': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'op': 'PREPARE', 'ppSeqNo': 2, 'ppTime': 1505402127, 'viewNo': 1}, 'Node1') 2017-09-14 15:15:27,786 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'instId': 0, 'stateRootHash': '2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf', 'digest': '89ccc072336b1421ffb726dffb332d7d90ed24649d0dc981ddfc174581b650de', 'txnRootHash': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'ppSeqNo': 2, 'ppTime': 1505402127, 'viewNo': 1} 2017-09-14 15:15:27,786 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 1} 2017-09-14 15:15:27,786 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'ppSeqNo': 2, 'viewNo': 1}, 'Node1') 2017-09-14 15:15:27,786 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 1} 2017-09-14 15:15:27,787 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 0, 'stateRootHash': '2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf', 'digest': '89ccc072336b1421ffb726dffb332d7d90ed24649d0dc981ddfc174581b650de', 'txnRootHash': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'ppSeqNo': 2, 'ppTime': 1505402127, 'viewNo': 1} 2017-09-14 15:15:27,787 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (1, 2) 2017-09-14 15:15:27,787 | DEBUG | replica.py ( 953) | doCommit | Node2:0 Sending COMMIT(1, 2) at 13504.639373887 2017-09-14 15:15:27,788 | DEBUG | replica.py (2193) | send | Node2:0 sending Commit 2017-09-14 15:15:27,788 | TRACE | replica.py (2194) | send | Node2:0 sending COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 1} 2017-09-14 15:15:27,788 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 1} commits where f is 0 2017-09-14 15:15:27,788 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (1, 2) 2017-09-14 15:15:27,788 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(1, 2) from Node1:0 2017-09-14 15:15:27,788 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 1} commits where f is 0 2017-09-14 15:15:27,788 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(1, 2) 2017-09-14 15:15:27,789 | DEBUG | node.py (2608) | send | Node2 sending message COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 1} to all recipients: ['Node1', 'Node3'] 2017-09-14 15:15:27,789 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:15:27,789 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 1} 2017-09-14 15:15:27,789 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'ppSeqNo': 2, 'viewNo': 1}, 'Node3') 2017-09-14 15:15:27,790 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 1} 2017-09-14 15:15:27,790 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","ppSeqNo":2,"viewNo":1}' to Node1 2017-09-14 15:15:27,790 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","ppSeqNo":2,"viewNo":1}' to Node1 2017-09-14 15:15:27,790 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","ppSeqNo":2,"viewNo":1}' to Node3 2017-09-14 15:15:27,790 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","ppSeqNo":2,"viewNo":1}' to Node3 2017-09-14 15:15:27,791 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(1, 2) from Node3:0 2017-09-14 15:15:27,796 | TRACE | replica.py ( 925) | tryOrder | Node2:0 returning request to node 2017-09-14 15:15:27,796 | DEBUG | replica.py (1444) | doOrder | Node2:0 ordering COMMIT (1, 2) 2017-09-14 15:15:27,796 | DEBUG | replica.py ( 323) | last_ordered_3pc | Node2:0 set last ordered as (1, 2) 2017-09-14 15:15:27,796 | DEBUG | replica.py (2193) | send | Node2:0 sending Ordered 2017-09-14 15:15:27,796 | TRACE | replica.py (2194) | send | Node2:0 sending ORDERED{'instId': 0, 'stateRootHash': '2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf', 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402127709038)], 'txnRootHash': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'ppSeqNo': 2, 'ppTime': 1505402127, 'ledgerId': 1, 'viewNo': 1} 2017-09-14 15:15:27,796 | INFO | replica.py (1483) | order_3pc_key | Node2:0 ordered batch request, view no 1, ppSeqNo 2, ledger 1, state root 2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf, txn root 3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1505402127709038)], discarded [] 2017-09-14 15:15:27,797 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(1, 2) 2017-09-14 15:15:27,797 | DEBUG | node.py (1847) | processOrdered | Node2 executing Ordered batch 1 2 of 1 requests 2017-09-14 15:15:27,797 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-09-14 15:15:27,798 | DEBUG | node.py (2386) | sendReplyToClient | Node2 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1505402127709038) to client 2017-09-14 15:15:27,798 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"op":"REPLY","result":{"txnTime":1505402127,"reqId":1505402127709038,"seqNo":10,"rootHash":"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq","type":"1","identifier":"V4SGRU86Z58d6TV7PBUe6f","dest":"V4SGRU86Z58d6TV7PBUe65","signature":"Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D","auditPath":["CkpbpBJkZfJw4NsSEot9TP4CKGonBTKj66VXAwRUSbUy","4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja"]}}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:15:27,798 | INFO | node.py (2290) | executeBatch | Node2 committed batch request, view no 1, ppSeqNo 2, ledger 1, state root 2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf, txn root 3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq, requests: [('V4SGRU86Z58d6TV7PBUe6f', 1505402127709038)] 2017-09-14 15:15:27,798 | DEBUG | node.py (2305) | executeBatch | Node2 storing 3PC key (1, 2) for ledger 1 range (10, 10) 2017-09-14 15:15:27,798 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:15:36,046 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 49 2017-09-14 15:15:36,047 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:15:36,047 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:15:36,047 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:15:36,047 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:15:36,047 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:15:36,047 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:15:36,047 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 50 to run in 10 seconds 2017-09-14 15:15:36,260 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:15:36,260 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:15:36,261 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:15:46,049 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 50 2017-09-14 15:15:46,050 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:15:46,050 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:15:46,050 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:15:46,050 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:15:46,050 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:15:46,050 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:15:46,050 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 51 to run in 10 seconds 2017-09-14 15:15:51,267 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:15:51,267 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:15:51,268 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:15:56,055 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 51 2017-09-14 15:15:56,055 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:15:56,055 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:15:56,055 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:15:56,055 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:15:56,055 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:15:56,056 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:15:56,056 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 52 to run in 10 seconds 2017-09-14 15:16:04,826 | TRACE | remote.py ( 119) | hasLostConnection | Remote Node1:HA(host='10.0.0.2', port=9701) has monitor events: [512, 4] 2017-09-14 15:16:04,826 | DEBUG | remote.py ( 123) | hasLostConnection | Node1:HA(host='10.0.0.2', port=9701) found disconnected event on monitor 2017-09-14 15:16:04,826 | DEBUG | keep_in_touch.py ( 68) | conns | Node2's connections changed from {'Node1', 'Node3'} to {'Node3'} 2017-09-14 15:16:04,826 | INFO | keep_in_touch.py ( 93) | _connsChanged | CONNECTION: Node2 disconnected from Node1 2017-09-14 15:16:04,826 | DEBUG | motor.py ( 34) | set_status | Node2 changing status from started to started_hungry 2017-09-14 15:16:04,826 | DEBUG | node.py ( 941) | checkInstances | Node2 choosing to start election on the basis of count 2 and nodes {'Node3'} 2017-09-14 15:16:05,825 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 44 2017-09-14 15:16:05,825 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node2 checking its request amount 2017-09-14 15:16:05,825 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:16:05,825 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 53 to run in 60 seconds 2017-09-14 15:16:05,825 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 6 2017-09-14 15:16:05,825 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-09-14 15:16:05,825 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 7 to run in 60 seconds 2017-09-14 15:16:06,058 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 52 2017-09-14 15:16:06,059 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:16:06,059 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:16:06,059 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:16:06,059 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:16:06,059 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:16:06,059 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:16:06,059 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 54 to run in 10 seconds 2017-09-14 15:16:06,169 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action dump_json_file with id 46 2017-09-14 15:16:06,170 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action dump_json_file with id 55 to run in 60 seconds 2017-09-14 15:16:06,268 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:16:06,269 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:16:06,269 | DEBUG | zstack.py ( 645) | sendPingPong | Node2 pinged Node1 2017-09-14 15:16:06,269 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:16:06,280 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'pi' to Node1 2017-09-14 15:16:06,281 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'pi' to Node1 2017-09-14 15:16:07,763 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action _check_view_change_completed with id 47 2017-09-14 15:16:07,763 | DEBUG | node.py (1011) | _check_view_change_completed | Node2 running the scheduled check for view change completion 2017-09-14 15:16:07,763 | DEBUG | node.py (1013) | _check_view_change_completed | Node2 already completion view change 2017-09-14 15:16:08,885 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2C got 1 messages through listener 2017-09-14 15:16:08,886 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1505402168868387 2017-09-14 15:16:08,886 | TRACE | node.py (1388) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe66'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} 2017-09-14 15:16:08,886 | DEBUG | node.py (1434) | processClientInBox | Node2C processing b'v#ZljK34WWooA6%RgESn.Ox' request SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe66'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} 2017-09-14 15:16:08,886 | DEBUG | node.py (1708) | processRequest | Node2 received client request: SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe66'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:16:08,887 | TRACE | propagator.py ( 165) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe66'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} 2017-09-14 15:16:08,887 | INFO | propagator.py ( 148) | propagate | Node2 propagating request ('V4SGRU86Z58d6TV7PBUe6f', 1505402168868387) from client b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:16:08,887 | DEBUG | node.py (2608) | send | Node2 sending message PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe66'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} to all recipients: ['Node1', 'Node3'] 2017-09-14 15:16:08,887 | DEBUG | propagator.py ( 214) | forward | Node2 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1505402168868387) to 0 replicas 2017-09-14 15:16:08,887 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402168868387}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:16:08,887 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"request":{"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe66"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402168868387,"signature":"36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node1 2017-09-14 15:16:08,888 | DEBUG | zstack.py ( 728) | transmit | Remote Node1 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-09-14 15:16:08,888 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"request":{"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe66"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402168868387,"signature":"36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node1 2017-09-14 15:16:08,888 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"request":{"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe66"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402168868387,"signature":"36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node3 2017-09-14 15:16:08,888 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"request":{"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe66"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402168868387,"signature":"36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node3 2017-09-14 15:16:08,889 | DEBUG | replica.py ( 667) | create3PCBatch | Node2:0 creating batch 3 for ledger 1 with state root b'\x11J\x1d.C\xe9\xb1*\xa9\xb5Z\xb5\xf3\xe5h\xb7\xf4\xc9\xbe\xaf[\x01\xee\x89|T\x06\xffQG\x9cx' 2017-09-14 15:16:08,890 | DEBUG | replica.py ( 700) | create3PCBatch | Node2:0 created a PRE-PREPARE with 1 requests for ledger 1 2017-09-14 15:16:08,890 | DEBUG | replica.py (2193) | send | Node2:0 sending PrePrepare 2017-09-14 15:16:08,890 | TRACE | replica.py (2194) | send | Node2:0 sending PREPREPARE{'instId': 0, 'discarded': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402168868387)], 'digest': '9ecbd5039951ac63430121b1efcd09398a11361343dec5681f72386f62408e30', 'txnRootHash': 'J58EEZUkgZpjDMen6f7zj1jgYhj5GTsw2YxQRFUZ8XNS', 'stateRootHash': '2YkpdGxx16PJ9cKVYHjcGBvBFvAcnWtYHTTMMQznVMo6', 'ppSeqNo': 3, 'ppTime': 1505402168, 'ledgerId': 1, 'viewNo': 1} 2017-09-14 15:16:08,890 | DEBUG | replica.py ( 614) | trackBatches | Node2:0 tracking batch for PREPREPARE{'instId': 0, 'discarded': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402168868387)], 'digest': '9ecbd5039951ac63430121b1efcd09398a11361343dec5681f72386f62408e30', 'txnRootHash': 'J58EEZUkgZpjDMen6f7zj1jgYhj5GTsw2YxQRFUZ8XNS', 'stateRootHash': '2YkpdGxx16PJ9cKVYHjcGBvBFvAcnWtYHTTMMQznVMo6', 'ppSeqNo': 3, 'ppTime': 1505402168, 'ledgerId': 1, 'viewNo': 1} with state root b'\x11J\x1d.C\xe9\xb1*\xa9\xb5Z\xb5\xf3\xe5h\xb7\xf4\xc9\xbe\xaf[\x01\xee\x89|T\x06\xffQG\x9cx' 2017-09-14 15:16:08,890 | DEBUG | node.py (2608) | send | Node2 sending message PREPREPARE{'instId': 0, 'discarded': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402168868387)], 'digest': '9ecbd5039951ac63430121b1efcd09398a11361343dec5681f72386f62408e30', 'txnRootHash': 'J58EEZUkgZpjDMen6f7zj1jgYhj5GTsw2YxQRFUZ8XNS', 'stateRootHash': '2YkpdGxx16PJ9cKVYHjcGBvBFvAcnWtYHTTMMQznVMo6', 'ppSeqNo': 3, 'ppTime': 1505402168, 'ledgerId': 1, 'viewNo': 1} to all recipients: ['Node1', 'Node3'] 2017-09-14 15:16:08,891 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:16:08,892 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1505402168868387 2017-09-14 15:16:08,892 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe66'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:16:08,892 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe66'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox', 'op': 'PROPAGATE'}, 'Node3') 2017-09-14 15:16:08,892 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe66'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:16:08,892 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe66'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:16:08,892 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe66'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} 2017-09-14 15:16:08,892 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe66'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} to its replicas since already forwarded 2017-09-14 15:16:08,893 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"discarded":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1505402168868387]],"ppTime":1505402168,"instId":0,"stateRootHash":"2YkpdGxx16PJ9cKVYHjcGBvBFvAcnWtYHTTMMQznVMo6","digest":"9ecbd5039951ac63430121b1efcd09398a11361343dec5681f72386f62408e30","txnRootHash":"J58EEZUkgZpjDMen6f7zj1jgYhj5GTsw2YxQRFUZ8XNS","op":"PREPREPARE","ppSeqNo":3,"ledgerId":1,"viewNo":1}' to Node1 2017-09-14 15:16:08,893 | DEBUG | zstack.py ( 728) | transmit | Remote Node1 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-09-14 15:16:08,893 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"discarded":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1505402168868387]],"ppTime":1505402168,"instId":0,"stateRootHash":"2YkpdGxx16PJ9cKVYHjcGBvBFvAcnWtYHTTMMQznVMo6","digest":"9ecbd5039951ac63430121b1efcd09398a11361343dec5681f72386f62408e30","txnRootHash":"J58EEZUkgZpjDMen6f7zj1jgYhj5GTsw2YxQRFUZ8XNS","op":"PREPREPARE","ppSeqNo":3,"ledgerId":1,"viewNo":1}' to Node1 2017-09-14 15:16:08,893 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"discarded":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1505402168868387]],"ppTime":1505402168,"instId":0,"stateRootHash":"2YkpdGxx16PJ9cKVYHjcGBvBFvAcnWtYHTTMMQznVMo6","digest":"9ecbd5039951ac63430121b1efcd09398a11361343dec5681f72386f62408e30","txnRootHash":"J58EEZUkgZpjDMen6f7zj1jgYhj5GTsw2YxQRFUZ8XNS","op":"PREPREPARE","ppSeqNo":3,"ledgerId":1,"viewNo":1}' to Node3 2017-09-14 15:16:08,893 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"discarded":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1505402168868387]],"ppTime":1505402168,"instId":0,"stateRootHash":"2YkpdGxx16PJ9cKVYHjcGBvBFvAcnWtYHTTMMQznVMo6","digest":"9ecbd5039951ac63430121b1efcd09398a11361343dec5681f72386f62408e30","txnRootHash":"J58EEZUkgZpjDMen6f7zj1jgYhj5GTsw2YxQRFUZ8XNS","op":"PREPREPARE","ppSeqNo":3,"ledgerId":1,"viewNo":1}' to Node3 2017-09-14 15:16:08,919 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:16:08,920 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'instId': 0, 'stateRootHash': '2YkpdGxx16PJ9cKVYHjcGBvBFvAcnWtYHTTMMQznVMo6', 'digest': '9ecbd5039951ac63430121b1efcd09398a11361343dec5681f72386f62408e30', 'txnRootHash': 'J58EEZUkgZpjDMen6f7zj1jgYhj5GTsw2YxQRFUZ8XNS', 'ppSeqNo': 3, 'ppTime': 1505402168, 'viewNo': 1} 2017-09-14 15:16:08,920 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'stateRootHash': '2YkpdGxx16PJ9cKVYHjcGBvBFvAcnWtYHTTMMQznVMo6', 'digest': '9ecbd5039951ac63430121b1efcd09398a11361343dec5681f72386f62408e30', 'txnRootHash': 'J58EEZUkgZpjDMen6f7zj1jgYhj5GTsw2YxQRFUZ8XNS', 'op': 'PREPARE', 'ppTime': 1505402168, 'ppSeqNo': 3, 'viewNo': 1}, 'Node3') 2017-09-14 15:16:08,920 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'instId': 0, 'stateRootHash': '2YkpdGxx16PJ9cKVYHjcGBvBFvAcnWtYHTTMMQznVMo6', 'digest': '9ecbd5039951ac63430121b1efcd09398a11361343dec5681f72386f62408e30', 'txnRootHash': 'J58EEZUkgZpjDMen6f7zj1jgYhj5GTsw2YxQRFUZ8XNS', 'ppSeqNo': 3, 'ppTime': 1505402168, 'viewNo': 1} 2017-09-14 15:16:08,921 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 0, 'stateRootHash': '2YkpdGxx16PJ9cKVYHjcGBvBFvAcnWtYHTTMMQznVMo6', 'digest': '9ecbd5039951ac63430121b1efcd09398a11361343dec5681f72386f62408e30', 'txnRootHash': 'J58EEZUkgZpjDMen6f7zj1jgYhj5GTsw2YxQRFUZ8XNS', 'ppSeqNo': 3, 'ppTime': 1505402168, 'viewNo': 1} 2017-09-14 15:16:08,921 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (1, 3) 2017-09-14 15:16:16,061 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 54 2017-09-14 15:16:16,061 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:16:16,061 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:16:16,062 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:16:16,062 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:16:16,062 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:16:16,062 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:16:16,062 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 56 to run in 10 seconds 2017-09-14 15:16:18,825 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:16:18,825 | DEBUG | zstack.py ( 663) | handlePingPong | Node2 got ping from Node1 2017-09-14 15:16:18,825 | DEBUG | zstack.py ( 645) | sendPingPong | Node2 ponged Node1 2017-09-14 15:16:18,825 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'po' to Node1 2017-09-14 15:16:18,826 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'po' to Node1 2017-09-14 15:16:18,885 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-09-14 15:16:18,886 | DEBUG | zstack.py ( 669) | handlePingPong | Node2 got pong from Node1 2017-09-14 15:16:18,886 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: BATCH{'signature': None, 'messages': ['{"request":{"operation":{"dest":"V4SGRU86Z58d6TV7PBUe66","type":"1"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402168868387,"signature":"36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}']} 2017-09-14 15:16:18,886 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"request":{"operation":{"dest":"V4SGRU86Z58d6TV7PBUe66","type":"1"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402168868387,"signature":"36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}']}, 'Node1') 2017-09-14 15:16:18,886 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"request":{"operation":{"dest":"V4SGRU86Z58d6TV7PBUe66","type":"1"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402168868387,"signature":"36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}']} 2017-09-14 15:16:18,887 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1505402168868387 2017-09-14 15:16:18,887 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe66'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:16:18,887 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe66'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox', 'op': 'PROPAGATE'}, 'Node1') 2017-09-14 15:16:18,887 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe66'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:16:18,888 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: BATCH{'signature': None, 'messages': ['{"viewNo":0,"op":"CURRENT_STATE","primary":[]}', '{"ledgerId":0,"op":"LEDGER_STATUS","viewNo":null,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ppSeqNo":null,"txnSeqNo":3}']} 2017-09-14 15:16:18,888 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"viewNo":0,"op":"CURRENT_STATE","primary":[]}', '{"ledgerId":0,"op":"LEDGER_STATUS","viewNo":null,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ppSeqNo":null,"txnSeqNo":3}']}, 'Node1') 2017-09-14 15:16:18,888 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"viewNo":0,"op":"CURRENT_STATE","primary":[]}', '{"ledgerId":0,"op":"LEDGER_STATUS","viewNo":null,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ppSeqNo":null,"txnSeqNo":3}']} 2017-09-14 15:16:18,889 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: CURRENT_STATE{'primary': [], 'viewNo': 0} 2017-09-14 15:16:18,889 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'CURRENT_STATE', 'primary': [], 'viewNo': 0}, 'Node1') 2017-09-14 15:16:18,889 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox CURRENT_STATE{'primary': [], 'viewNo': 0} 2017-09-14 15:16:18,889 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} 2017-09-14 15:16:18,889 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'txnSeqNo': 3, 'op': 'LEDGER_STATUS', 'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ledgerId': 0, 'viewNo': None}, 'Node1') 2017-09-14 15:16:18,889 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} 2017-09-14 15:16:18,889 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe66'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:16:18,890 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe66'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} 2017-09-14 15:16:18,890 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe66'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} to its replicas since already forwarded 2017-09-14 15:16:18,890 | DEBUG | node.py ( 907) | process_current_state_message | Node2 processing current state CURRENT_STATE{'primary': [], 'viewNo': 0} from Node1 2017-09-14 15:16:18,890 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} from Node1 2017-09-14 15:16:18,890 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 3 with 3 2017-09-14 15:16:18,890 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 3 with 3 2017-09-14 15:16:18,892 | TRACE | remote.py ( 119) | hasLostConnection | Remote Node1:HA(host='10.0.0.2', port=9701) has monitor events: [2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 1] 2017-09-14 15:16:18,892 | DEBUG | remote.py ( 123) | hasLostConnection | Node1:HA(host='10.0.0.2', port=9701) found disconnected event on monitor 2017-09-14 15:16:18,892 | DEBUG | keep_in_touch.py ( 68) | conns | Node2's connections changed from {'Node3'} to {'Node1', 'Node3'} 2017-09-14 15:16:18,892 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node2 now connected to Node1 2017-09-14 15:16:18,892 | DEBUG | motor.py ( 34) | set_status | Node2 changing status from started_hungry to started 2017-09-14 15:16:18,892 | DEBUG | node.py ( 941) | checkInstances | Node2 choosing to start election on the basis of count 3 and nodes {'Node1', 'Node3'} 2017-09-14 15:16:18,893 | DEBUG | node.py ( 902) | send_current_state_to_lagging_node | Node2 sending current state CURRENT_STATE{'primary': [VIEW_CHANGE_DONE{'ledgerInfo': ((0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')), 'name': 'Node2', 'viewNo': 1}], 'viewNo': 1} to lagged node Node1 2017-09-14 15:16:18,893 | DEBUG | node.py (2608) | send | Node2 sending message CURRENT_STATE{'primary': [VIEW_CHANGE_DONE{'ledgerInfo': ((0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')), 'name': 'Node2', 'viewNo': 1}], 'viewNo': 1} to 1 recipients: ['Node1'] 2017-09-14 15:16:18,893 | DEBUG | node.py (2608) | send | Node2 sending message LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} to 1 recipients: ['Node1'] 2017-09-14 15:16:18,893 | DEBUG | node.py (2608) | send | Node2 sending message LEDGER_STATUS{'ppSeqNo': 2, 'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'txnSeqNo': 10, 'ledgerId': 1, 'viewNo': 1} to 1 recipients: ['Node1'] 2017-09-14 15:16:18,894 | DEBUG | node.py (2608) | send | Node2 sending message LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None} to 1 recipients: ['Node1'] 2017-09-14 15:16:18,894 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 4 msgs to Node1 into one transmission 2017-09-14 15:16:18,894 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"CURRENT_STATE","primary":[{"ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,8,"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node2","viewNo":1}],"viewNo":1}', b'{"ledgerId":0,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","txnSeqNo":3,"viewNo":null}', b'{"ledgerId":1,"op":"LEDGER_STATUS","ppSeqNo":2,"merkleRoot":"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq","txnSeqNo":10,"viewNo":1}', b'{"ledgerId":2,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","txnSeqNo":0,"viewNo":null}']) 2017-09-14 15:16:18,894 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[{\\"ledgerInfo\\":[[0,3,\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\"],[1,8,\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"name\\":\\"Node2\\",\\"viewNo\\":1}],\\"viewNo\\":1}","{\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":null,\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"txnSeqNo\\":3,\\"viewNo\\":null}","{\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":2,\\"merkleRoot\\":\\"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq\\",\\"txnSeqNo\\":10,\\"viewNo\\":1}","{\\"ledgerId\\":2,\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":null,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"txnSeqNo\\":0,\\"viewNo\\":null}"],"signature":null}' 2017-09-14 15:16:18,895 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[{\\"ledgerInfo\\":[[0,3,\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\"],[1,8,\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"name\\":\\"Node2\\",\\"viewNo\\":1}],\\"viewNo\\":1}","{\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":null,\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"txnSeqNo\\":3,\\"viewNo\\":null}","{\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":2,\\"merkleRoot\\":\\"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq\\",\\"txnSeqNo\\":10,\\"viewNo\\":1}","{\\"ledgerId\\":2,\\"op\\":\\"LEDGER_STATUS\\",\\"ppSeqNo\\":null,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"txnSeqNo\\":0,\\"viewNo\\":null}"],"signature":null}' to Node1 2017-09-14 15:16:18,980 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:16:18,980 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,10,"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node2","op":"VIEW_CHANGE_DONE","viewNo":1}']} 2017-09-14 15:16:18,980 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,10,"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node2","op":"VIEW_CHANGE_DONE","viewNo":1}']}, 'Node1') 2017-09-14 15:16:18,980 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,10,"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node2","op":"VIEW_CHANGE_DONE","viewNo":1}']} 2017-09-14 15:16:18,981 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:16:18,981 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node1') 2017-09-14 15:16:18,981 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:16:18,981 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:16:18,981 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}, 'Node1') 2017-09-14 15:16:18,981 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:16:18,981 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:16:18,982 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node1') 2017-09-14 15:16:18,982 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:16:18,982 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:16:18,982 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'MESSAGE_REQUEST', 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}, 'Node1') 2017-09-14 15:16:18,982 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:16:18,982 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 10, '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1} 2017-09-14 15:16:18,982 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'op': 'VIEW_CHANGE_DONE', 'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 10, '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1}, 'Node1') 2017-09-14 15:16:18,983 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 10, '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1} 2017-09-14 15:16:18,983 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}} to 1 recipients: ['Node1'] 2017-09-14 15:16:18,983 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None}} to 1 recipients: ['Node1'] 2017-09-14 15:16:18,984 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': LEDGER_STATUS{'ppSeqNo': 2, 'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'txnSeqNo': 10, 'ledgerId': 1, 'viewNo': 1}} to 1 recipients: ['Node1'] 2017-09-14 15:16:18,984 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': LEDGER_STATUS{'ppSeqNo': 2, 'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'txnSeqNo': 10, 'ledgerId': 1, 'viewNo': 1}} to 1 recipients: ['Node1'] 2017-09-14 15:16:18,984 | DEBUG | node.py (1205) | sendToElector | Node2 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 10, '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1}, 'Node1') 2017-09-14 15:16:18,985 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node2's primary selector started processing of ViewChangeDone msg from Node1 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 10, '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1} 2017-09-14 15:16:18,985 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 10, '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1} because it already decided primary which is Node2:0 2017-09-14 15:16:18,985 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 4 msgs to Node1 into one transmission 2017-09-14 15:16:18,985 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq","ppSeqNo":2,"txnSeqNo":10,"viewNo":1}}', b'{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq","ppSeqNo":2,"txnSeqNo":10,"viewNo":1}}']) 2017-09-14 15:16:18,985 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq\\",\\"ppSeqNo\\":2,\\"txnSeqNo\\":10,\\"viewNo\\":1}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq\\",\\"ppSeqNo\\":2,\\"txnSeqNo\\":10,\\"viewNo\\":1}}"],"signature":null}' 2017-09-14 15:16:18,985 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq\\",\\"ppSeqNo\\":2,\\"txnSeqNo\\":10,\\"viewNo\\":1}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq\\",\\"ppSeqNo\\":2,\\"txnSeqNo\\":10,\\"viewNo\\":1}}"],"signature":null}' to Node1 2017-09-14 15:16:18,999 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-09-14 15:16:18,999 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PREPARE{'instId': 0, 'stateRootHash': '2YkpdGxx16PJ9cKVYHjcGBvBFvAcnWtYHTTMMQznVMo6', 'digest': '9ecbd5039951ac63430121b1efcd09398a11361343dec5681f72386f62408e30', 'txnRootHash': 'J58EEZUkgZpjDMen6f7zj1jgYhj5GTsw2YxQRFUZ8XNS', 'ppSeqNo': 3, 'ppTime': 1505402168, 'viewNo': 1} 2017-09-14 15:16:18,999 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'stateRootHash': '2YkpdGxx16PJ9cKVYHjcGBvBFvAcnWtYHTTMMQznVMo6', 'digest': '9ecbd5039951ac63430121b1efcd09398a11361343dec5681f72386f62408e30', 'txnRootHash': 'J58EEZUkgZpjDMen6f7zj1jgYhj5GTsw2YxQRFUZ8XNS', 'op': 'PREPARE', 'ppTime': 1505402168, 'ppSeqNo': 3, 'viewNo': 1}, 'Node1') 2017-09-14 15:16:18,999 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'instId': 0, 'stateRootHash': '2YkpdGxx16PJ9cKVYHjcGBvBFvAcnWtYHTTMMQznVMo6', 'digest': '9ecbd5039951ac63430121b1efcd09398a11361343dec5681f72386f62408e30', 'txnRootHash': 'J58EEZUkgZpjDMen6f7zj1jgYhj5GTsw2YxQRFUZ8XNS', 'ppSeqNo': 3, 'ppTime': 1505402168, 'viewNo': 1} 2017-09-14 15:16:19,000 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 0, 'ppSeqNo': 3, 'viewNo': 1} 2017-09-14 15:16:19,000 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'ppSeqNo': 3, 'viewNo': 1}, 'Node3') 2017-09-14 15:16:19,000 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 3, 'viewNo': 1} 2017-09-14 15:16:19,001 | DEBUG | replica.py ( 953) | doCommit | Node2:0 Sending COMMIT(1, 3) at 13555.852675902 2017-09-14 15:16:19,001 | DEBUG | replica.py (2193) | send | Node2:0 sending Commit 2017-09-14 15:16:19,001 | TRACE | replica.py (2194) | send | Node2:0 sending COMMIT{'instId': 0, 'ppSeqNo': 3, 'viewNo': 1} 2017-09-14 15:16:19,001 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'ppSeqNo': 3, 'viewNo': 1} commits where f is 0 2017-09-14 15:16:19,001 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (1, 3) 2017-09-14 15:16:19,001 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(1, 3) from Node3:0 2017-09-14 15:16:19,001 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'ppSeqNo': 3, 'viewNo': 1} commits where f is 0 2017-09-14 15:16:19,002 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(1, 3) 2017-09-14 15:16:19,002 | DEBUG | node.py (2608) | send | Node2 sending message COMMIT{'instId': 0, 'ppSeqNo': 3, 'viewNo': 1} to all recipients: ['Node1', 'Node3'] 2017-09-14 15:16:19,002 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","ppSeqNo":3,"viewNo":1}' to Node1 2017-09-14 15:16:19,003 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","ppSeqNo":3,"viewNo":1}' to Node1 2017-09-14 15:16:19,003 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","ppSeqNo":3,"viewNo":1}' to Node3 2017-09-14 15:16:19,003 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","ppSeqNo":3,"viewNo":1}' to Node3 2017-09-14 15:16:21,279 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:16:21,279 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:16:21,279 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:16:22,364 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2C got 1 messages through listener 2017-09-14 15:16:22,365 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1505402182349581 2017-09-14 15:16:22,365 | TRACE | node.py (1388) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe67'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:16:22,365 | DEBUG | node.py (1434) | processClientInBox | Node2C processing b'v#ZljK34WWooA6%RgESn.Ox' request SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe67'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:16:22,366 | DEBUG | node.py (1708) | processRequest | Node2 received client request: SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe67'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:16:22,366 | TRACE | propagator.py ( 165) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe67'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:16:22,366 | INFO | propagator.py ( 148) | propagate | Node2 propagating request ('V4SGRU86Z58d6TV7PBUe6f', 1505402182349581) from client b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:16:22,366 | DEBUG | node.py (2608) | send | Node2 sending message PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe67'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} to all recipients: ['Node1', 'Node3'] 2017-09-14 15:16:22,366 | DEBUG | propagator.py ( 214) | forward | Node2 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1505402182349581) to 0 replicas 2017-09-14 15:16:22,366 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402182349581}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:16:22,367 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"request":{"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe67"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402182349581,"signature":"5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node1 2017-09-14 15:16:22,367 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"request":{"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe67"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402182349581,"signature":"5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node1 2017-09-14 15:16:22,367 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"request":{"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe67"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402182349581,"signature":"5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node3 2017-09-14 15:16:22,367 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"request":{"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe67"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402182349581,"signature":"5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node3 2017-09-14 15:16:22,368 | DEBUG | replica.py ( 667) | create3PCBatch | Node2:0 creating batch 4 for ledger 1 with state root b'\x16\xfe=\x9f\x84{g1\xa0\xe7I\x8bc\xb3\xe8\xb5\x82\xdd\xf1x\x8f\xcf k\xf0\x9b\xef\x1f\x99\xaf;\xd1' 2017-09-14 15:16:22,369 | DEBUG | replica.py ( 700) | create3PCBatch | Node2:0 created a PRE-PREPARE with 1 requests for ledger 1 2017-09-14 15:16:22,370 | DEBUG | replica.py (2193) | send | Node2:0 sending PrePrepare 2017-09-14 15:16:22,370 | TRACE | replica.py (2194) | send | Node2:0 sending PREPREPARE{'instId': 0, 'discarded': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402182349581)], 'digest': 'cfbf1f06b34b99be51c2768e434b24584e26aaf349bdcdfe774d173c848d4247', 'txnRootHash': '6yaNKaDqtduU2k7wmoJcdHdp7Jf3BBjwUEFJAyyukC1T', 'stateRootHash': 'EoqS54wgE8uye5WQ1rHrFj1stwrnnTFZi9y3t6aNY4nv', 'ppSeqNo': 4, 'ppTime': 1505402182, 'ledgerId': 1, 'viewNo': 1} 2017-09-14 15:16:22,370 | DEBUG | replica.py ( 614) | trackBatches | Node2:0 tracking batch for PREPREPARE{'instId': 0, 'discarded': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402182349581)], 'digest': 'cfbf1f06b34b99be51c2768e434b24584e26aaf349bdcdfe774d173c848d4247', 'txnRootHash': '6yaNKaDqtduU2k7wmoJcdHdp7Jf3BBjwUEFJAyyukC1T', 'stateRootHash': 'EoqS54wgE8uye5WQ1rHrFj1stwrnnTFZi9y3t6aNY4nv', 'ppSeqNo': 4, 'ppTime': 1505402182, 'ledgerId': 1, 'viewNo': 1} with state root b'\x16\xfe=\x9f\x84{g1\xa0\xe7I\x8bc\xb3\xe8\xb5\x82\xdd\xf1x\x8f\xcf k\xf0\x9b\xef\x1f\x99\xaf;\xd1' 2017-09-14 15:16:22,370 | DEBUG | node.py (2608) | send | Node2 sending message PREPREPARE{'instId': 0, 'discarded': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402182349581)], 'digest': 'cfbf1f06b34b99be51c2768e434b24584e26aaf349bdcdfe774d173c848d4247', 'txnRootHash': '6yaNKaDqtduU2k7wmoJcdHdp7Jf3BBjwUEFJAyyukC1T', 'stateRootHash': 'EoqS54wgE8uye5WQ1rHrFj1stwrnnTFZi9y3t6aNY4nv', 'ppSeqNo': 4, 'ppTime': 1505402182, 'ledgerId': 1, 'viewNo': 1} to all recipients: ['Node1', 'Node3'] 2017-09-14 15:16:22,370 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:16:22,371 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1505402182349581 2017-09-14 15:16:22,371 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe67'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:16:22,371 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe67'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox', 'op': 'PROPAGATE'}, 'Node1') 2017-09-14 15:16:22,371 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe67'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:16:22,372 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe67'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:16:22,372 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe67'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:16:22,372 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe67'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} to its replicas since already forwarded 2017-09-14 15:16:22,375 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"discarded":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1505402182349581]],"ppTime":1505402182,"instId":0,"stateRootHash":"EoqS54wgE8uye5WQ1rHrFj1stwrnnTFZi9y3t6aNY4nv","digest":"cfbf1f06b34b99be51c2768e434b24584e26aaf349bdcdfe774d173c848d4247","txnRootHash":"6yaNKaDqtduU2k7wmoJcdHdp7Jf3BBjwUEFJAyyukC1T","op":"PREPREPARE","ppSeqNo":4,"ledgerId":1,"viewNo":1}' to Node1 2017-09-14 15:16:22,376 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"discarded":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1505402182349581]],"ppTime":1505402182,"instId":0,"stateRootHash":"EoqS54wgE8uye5WQ1rHrFj1stwrnnTFZi9y3t6aNY4nv","digest":"cfbf1f06b34b99be51c2768e434b24584e26aaf349bdcdfe774d173c848d4247","txnRootHash":"6yaNKaDqtduU2k7wmoJcdHdp7Jf3BBjwUEFJAyyukC1T","op":"PREPREPARE","ppSeqNo":4,"ledgerId":1,"viewNo":1}' to Node1 2017-09-14 15:16:22,376 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"discarded":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1505402182349581]],"ppTime":1505402182,"instId":0,"stateRootHash":"EoqS54wgE8uye5WQ1rHrFj1stwrnnTFZi9y3t6aNY4nv","digest":"cfbf1f06b34b99be51c2768e434b24584e26aaf349bdcdfe774d173c848d4247","txnRootHash":"6yaNKaDqtduU2k7wmoJcdHdp7Jf3BBjwUEFJAyyukC1T","op":"PREPREPARE","ppSeqNo":4,"ledgerId":1,"viewNo":1}' to Node3 2017-09-14 15:16:22,377 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"discarded":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1505402182349581]],"ppTime":1505402182,"instId":0,"stateRootHash":"EoqS54wgE8uye5WQ1rHrFj1stwrnnTFZi9y3t6aNY4nv","digest":"cfbf1f06b34b99be51c2768e434b24584e26aaf349bdcdfe774d173c848d4247","txnRootHash":"6yaNKaDqtduU2k7wmoJcdHdp7Jf3BBjwUEFJAyyukC1T","op":"PREPREPARE","ppSeqNo":4,"ledgerId":1,"viewNo":1}' to Node3 2017-09-14 15:16:22,389 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:16:22,390 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1505402182349581 2017-09-14 15:16:22,390 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe67'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:16:22,390 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe67'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox', 'op': 'PROPAGATE'}, 'Node3') 2017-09-14 15:16:22,390 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe67'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:16:22,390 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'request': {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe67'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:16:22,390 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe67'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:16:22,390 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe67'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} to its replicas since already forwarded 2017-09-14 15:16:22,391 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:16:22,392 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PREPARE{'instId': 0, 'stateRootHash': 'EoqS54wgE8uye5WQ1rHrFj1stwrnnTFZi9y3t6aNY4nv', 'digest': 'cfbf1f06b34b99be51c2768e434b24584e26aaf349bdcdfe774d173c848d4247', 'txnRootHash': '6yaNKaDqtduU2k7wmoJcdHdp7Jf3BBjwUEFJAyyukC1T', 'ppSeqNo': 4, 'ppTime': 1505402182, 'viewNo': 1} 2017-09-14 15:16:22,392 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'stateRootHash': 'EoqS54wgE8uye5WQ1rHrFj1stwrnnTFZi9y3t6aNY4nv', 'digest': 'cfbf1f06b34b99be51c2768e434b24584e26aaf349bdcdfe774d173c848d4247', 'txnRootHash': '6yaNKaDqtduU2k7wmoJcdHdp7Jf3BBjwUEFJAyyukC1T', 'op': 'PREPARE', 'ppTime': 1505402182, 'ppSeqNo': 4, 'viewNo': 1}, 'Node1') 2017-09-14 15:16:22,392 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'instId': 0, 'stateRootHash': 'EoqS54wgE8uye5WQ1rHrFj1stwrnnTFZi9y3t6aNY4nv', 'digest': 'cfbf1f06b34b99be51c2768e434b24584e26aaf349bdcdfe774d173c848d4247', 'txnRootHash': '6yaNKaDqtduU2k7wmoJcdHdp7Jf3BBjwUEFJAyyukC1T', 'ppSeqNo': 4, 'ppTime': 1505402182, 'viewNo': 1} 2017-09-14 15:16:22,393 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 0, 'stateRootHash': 'EoqS54wgE8uye5WQ1rHrFj1stwrnnTFZi9y3t6aNY4nv', 'digest': 'cfbf1f06b34b99be51c2768e434b24584e26aaf349bdcdfe774d173c848d4247', 'txnRootHash': '6yaNKaDqtduU2k7wmoJcdHdp7Jf3BBjwUEFJAyyukC1T', 'ppSeqNo': 4, 'ppTime': 1505402182, 'viewNo': 1} 2017-09-14 15:16:22,393 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (1, 4) 2017-09-14 15:16:22,405 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-09-14 15:16:22,406 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'instId': 0, 'stateRootHash': 'EoqS54wgE8uye5WQ1rHrFj1stwrnnTFZi9y3t6aNY4nv', 'digest': 'cfbf1f06b34b99be51c2768e434b24584e26aaf349bdcdfe774d173c848d4247', 'txnRootHash': '6yaNKaDqtduU2k7wmoJcdHdp7Jf3BBjwUEFJAyyukC1T', 'ppSeqNo': 4, 'ppTime': 1505402182, 'viewNo': 1} 2017-09-14 15:16:22,406 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'stateRootHash': 'EoqS54wgE8uye5WQ1rHrFj1stwrnnTFZi9y3t6aNY4nv', 'digest': 'cfbf1f06b34b99be51c2768e434b24584e26aaf349bdcdfe774d173c848d4247', 'txnRootHash': '6yaNKaDqtduU2k7wmoJcdHdp7Jf3BBjwUEFJAyyukC1T', 'op': 'PREPARE', 'ppTime': 1505402182, 'ppSeqNo': 4, 'viewNo': 1}, 'Node3') 2017-09-14 15:16:22,406 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'instId': 0, 'stateRootHash': 'EoqS54wgE8uye5WQ1rHrFj1stwrnnTFZi9y3t6aNY4nv', 'digest': 'cfbf1f06b34b99be51c2768e434b24584e26aaf349bdcdfe774d173c848d4247', 'txnRootHash': '6yaNKaDqtduU2k7wmoJcdHdp7Jf3BBjwUEFJAyyukC1T', 'ppSeqNo': 4, 'ppTime': 1505402182, 'viewNo': 1} 2017-09-14 15:16:22,406 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1} 2017-09-14 15:16:22,407 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'ppSeqNo': 4, 'viewNo': 1}, 'Node3') 2017-09-14 15:16:22,407 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1} 2017-09-14 15:16:22,407 | DEBUG | replica.py ( 953) | doCommit | Node2:0 Sending COMMIT(1, 4) at 13559.259384516 2017-09-14 15:16:22,408 | DEBUG | replica.py (2193) | send | Node2:0 sending Commit 2017-09-14 15:16:22,408 | TRACE | replica.py (2194) | send | Node2:0 sending COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1} 2017-09-14 15:16:22,408 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1} commits where f is 0 2017-09-14 15:16:22,408 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (1, 4) 2017-09-14 15:16:22,408 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(1, 4) from Node3:0 2017-09-14 15:16:22,408 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1} commits where f is 0 2017-09-14 15:16:22,408 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(1, 4) 2017-09-14 15:16:22,409 | DEBUG | node.py (2608) | send | Node2 sending message COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1} to all recipients: ['Node1', 'Node3'] 2017-09-14 15:16:22,409 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-09-14 15:16:22,409 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1} 2017-09-14 15:16:22,409 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'ppSeqNo': 4, 'viewNo': 1}, 'Node1') 2017-09-14 15:16:22,409 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1} 2017-09-14 15:16:22,410 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","ppSeqNo":4,"viewNo":1}' to Node1 2017-09-14 15:16:22,410 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","ppSeqNo":4,"viewNo":1}' to Node1 2017-09-14 15:16:22,410 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","ppSeqNo":4,"viewNo":1}' to Node3 2017-09-14 15:16:22,410 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","ppSeqNo":4,"viewNo":1}' to Node3 2017-09-14 15:16:22,411 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(1, 4) from Node1:0 2017-09-14 15:16:22,411 | DEBUG | has_action_queue.py ( 79) | startRepeating | process_stashed_out_of_order_commits will be repeating every 1 seconds 2017-09-14 15:16:22,411 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 1 to run in 1 seconds 2017-09-14 15:16:22,411 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: stashing COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1} since out of order 2017-09-14 15:16:22,411 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(1, 4) 2017-09-14 15:16:23,426 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 1 2017-09-14 15:16:23,426 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:23,426 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 2 to run in 1 seconds 2017-09-14 15:16:23,898 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2C got 1 messages through listener 2017-09-14 15:16:23,899 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1505402168868387 2017-09-14 15:16:23,899 | TRACE | node.py (1388) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe66', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} 2017-09-14 15:16:23,899 | DEBUG | node.py (1434) | processClientInBox | Node2C processing b'v#ZljK34WWooA6%RgESn.Ox' request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe66', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} 2017-09-14 15:16:23,899 | DEBUG | node.py (1708) | processRequest | Node2 received client request: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe66', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:16:23,899 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe66', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} 2017-09-14 15:16:23,899 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe66', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} to its replicas since already forwarded 2017-09-14 15:16:23,900 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402168868387}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:16:24,435 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 2 2017-09-14 15:16:24,435 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:24,435 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 3 to run in 1 seconds 2017-09-14 15:16:25,437 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 3 2017-09-14 15:16:25,437 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:25,437 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 4 to run in 1 seconds 2017-09-14 15:16:26,064 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 56 2017-09-14 15:16:26,065 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:16:26,065 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:16:26,065 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:16:26,065 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:16:26,065 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:16:26,065 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:16:26,065 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 57 to run in 10 seconds 2017-09-14 15:16:26,448 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 4 2017-09-14 15:16:26,449 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:26,449 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 5 to run in 1 seconds 2017-09-14 15:16:27,458 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 5 2017-09-14 15:16:27,458 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:27,458 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 6 to run in 1 seconds 2017-09-14 15:16:28,469 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 6 2017-09-14 15:16:28,469 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:28,469 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 7 to run in 1 seconds 2017-09-14 15:16:29,481 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 7 2017-09-14 15:16:29,481 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:29,481 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 8 to run in 1 seconds 2017-09-14 15:16:30,482 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 8 2017-09-14 15:16:30,482 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:30,482 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 9 to run in 1 seconds 2017-09-14 15:16:31,491 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 9 2017-09-14 15:16:31,492 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:31,492 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 10 to run in 1 seconds 2017-09-14 15:16:32,497 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 10 2017-09-14 15:16:32,497 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:32,497 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 11 to run in 1 seconds 2017-09-14 15:16:33,502 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 11 2017-09-14 15:16:33,502 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:33,503 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 12 to run in 1 seconds 2017-09-14 15:16:34,508 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 12 2017-09-14 15:16:34,509 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:34,509 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 13 to run in 1 seconds 2017-09-14 15:16:35,519 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 13 2017-09-14 15:16:35,519 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:35,520 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 14 to run in 1 seconds 2017-09-14 15:16:36,074 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 57 2017-09-14 15:16:36,074 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:16:36,075 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:16:36,075 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:16:36,075 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:16:36,075 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:16:36,075 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:16:36,075 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 58 to run in 10 seconds 2017-09-14 15:16:36,288 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:16:36,288 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:16:36,289 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:16:36,522 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 14 2017-09-14 15:16:36,523 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:36,523 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 15 to run in 1 seconds 2017-09-14 15:16:37,529 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 15 2017-09-14 15:16:37,529 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:37,530 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 16 to run in 1 seconds 2017-09-14 15:16:38,541 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 16 2017-09-14 15:16:38,541 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:38,541 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 17 to run in 1 seconds 2017-09-14 15:16:38,922 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2C got 1 messages through listener 2017-09-14 15:16:38,922 | TRACE | node.py (1388) | validateClientMsg | Node2C received CLIENT message: BATCH{'signature': None, 'messages': ['{"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402182349581,"signature":"5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb","operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe67"}}', '{"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402168868387,"signature":"36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8","operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe66"}}']} 2017-09-14 15:16:38,923 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1505402182349581 2017-09-14 15:16:38,923 | TRACE | node.py (1388) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:16:38,924 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1505402168868387 2017-09-14 15:16:38,924 | TRACE | node.py (1388) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe66', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} 2017-09-14 15:16:38,924 | DEBUG | node.py (1434) | processClientInBox | Node2C processing b'v#ZljK34WWooA6%RgESn.Ox' request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:16:38,924 | DEBUG | node.py (1708) | processRequest | Node2 received client request: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:16:38,924 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:16:38,924 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} to its replicas since already forwarded 2017-09-14 15:16:38,924 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402182349581}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:16:38,925 | DEBUG | node.py (1434) | processClientInBox | Node2C processing b'v#ZljK34WWooA6%RgESn.Ox' request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe66', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} 2017-09-14 15:16:38,925 | DEBUG | node.py (1708) | processRequest | Node2 received client request: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe66', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:16:38,925 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe66', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} 2017-09-14 15:16:38,925 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe66', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} to its replicas since already forwarded 2017-09-14 15:16:38,925 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402168868387}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:16:39,549 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 17 2017-09-14 15:16:39,550 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:39,550 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 18 to run in 1 seconds 2017-09-14 15:16:40,551 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 18 2017-09-14 15:16:40,552 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:40,552 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 19 to run in 1 seconds 2017-09-14 15:16:41,558 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 19 2017-09-14 15:16:41,558 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:41,558 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 20 to run in 1 seconds 2017-09-14 15:16:42,561 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 20 2017-09-14 15:16:42,561 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:42,561 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 21 to run in 1 seconds 2017-09-14 15:16:43,568 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 21 2017-09-14 15:16:43,568 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:43,568 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 22 to run in 1 seconds 2017-09-14 15:16:44,575 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 22 2017-09-14 15:16:44,576 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:44,576 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 23 to run in 1 seconds 2017-09-14 15:16:45,579 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 23 2017-09-14 15:16:45,580 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:45,580 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 24 to run in 1 seconds 2017-09-14 15:16:46,075 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 58 2017-09-14 15:16:46,075 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:16:46,075 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:16:46,076 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:16:46,076 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:16:46,076 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:16:46,076 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:16:46,076 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 59 to run in 10 seconds 2017-09-14 15:16:46,581 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 24 2017-09-14 15:16:46,581 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:46,581 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 25 to run in 1 seconds 2017-09-14 15:16:47,588 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 25 2017-09-14 15:16:47,588 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:47,588 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 26 to run in 1 seconds 2017-09-14 15:16:48,590 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 26 2017-09-14 15:16:48,591 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:48,591 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 27 to run in 1 seconds 2017-09-14 15:16:49,600 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 27 2017-09-14 15:16:49,601 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:49,601 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 28 to run in 1 seconds 2017-09-14 15:16:50,602 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 28 2017-09-14 15:16:50,602 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:50,602 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 29 to run in 1 seconds 2017-09-14 15:16:51,297 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:16:51,297 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:16:51,297 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:16:51,608 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 29 2017-09-14 15:16:51,608 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:51,608 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 30 to run in 1 seconds 2017-09-14 15:16:52,618 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 30 2017-09-14 15:16:52,618 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:52,618 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 31 to run in 1 seconds 2017-09-14 15:16:53,629 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 31 2017-09-14 15:16:53,629 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:53,629 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 32 to run in 1 seconds 2017-09-14 15:16:53,939 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2C got 1 messages through listener 2017-09-14 15:16:53,939 | TRACE | node.py (1388) | validateClientMsg | Node2C received CLIENT message: BATCH{'signature': None, 'messages': ['{"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402182349581,"signature":"5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb","operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe67"}}', '{"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402168868387,"signature":"36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8","operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe66"}}']} 2017-09-14 15:16:53,940 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1505402182349581 2017-09-14 15:16:53,940 | TRACE | node.py (1388) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:16:53,941 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1505402168868387 2017-09-14 15:16:53,942 | TRACE | node.py (1388) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe66', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} 2017-09-14 15:16:53,942 | DEBUG | node.py (1434) | processClientInBox | Node2C processing b'v#ZljK34WWooA6%RgESn.Ox' request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:16:53,942 | DEBUG | node.py (1708) | processRequest | Node2 received client request: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:16:53,942 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:16:53,942 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} to its replicas since already forwarded 2017-09-14 15:16:53,943 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402182349581}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:16:53,943 | DEBUG | node.py (1434) | processClientInBox | Node2C processing b'v#ZljK34WWooA6%RgESn.Ox' request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe66', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} 2017-09-14 15:16:53,943 | DEBUG | node.py (1708) | processRequest | Node2 received client request: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe66', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:16:53,943 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe66', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} 2017-09-14 15:16:53,943 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe66', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} to its replicas since already forwarded 2017-09-14 15:16:53,944 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402168868387}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:16:54,637 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 32 2017-09-14 15:16:54,637 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:54,637 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 33 to run in 1 seconds 2017-09-14 15:16:55,638 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 33 2017-09-14 15:16:55,638 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:55,638 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 34 to run in 1 seconds 2017-09-14 15:16:56,081 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 59 2017-09-14 15:16:56,081 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:16:56,082 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:16:56,082 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:16:56,082 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:16:56,082 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:16:56,082 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:16:56,082 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 60 to run in 10 seconds 2017-09-14 15:16:56,640 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 34 2017-09-14 15:16:56,640 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:56,640 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 35 to run in 1 seconds 2017-09-14 15:16:57,647 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 35 2017-09-14 15:16:57,647 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:57,647 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 36 to run in 1 seconds 2017-09-14 15:16:58,649 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 36 2017-09-14 15:16:58,649 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:58,649 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 37 to run in 1 seconds 2017-09-14 15:16:59,652 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 37 2017-09-14 15:16:59,652 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:16:59,652 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 38 to run in 1 seconds 2017-09-14 15:17:00,654 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 38 2017-09-14 15:17:00,654 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:00,654 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 39 to run in 1 seconds 2017-09-14 15:17:01,660 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 39 2017-09-14 15:17:01,660 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:01,661 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 40 to run in 1 seconds 2017-09-14 15:17:02,666 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 40 2017-09-14 15:17:02,666 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:02,666 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 41 to run in 1 seconds 2017-09-14 15:17:03,669 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 41 2017-09-14 15:17:03,669 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:03,669 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 42 to run in 1 seconds 2017-09-14 15:17:04,677 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 42 2017-09-14 15:17:04,678 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:04,678 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 43 to run in 1 seconds 2017-09-14 15:17:05,684 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 43 2017-09-14 15:17:05,684 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:05,684 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 44 to run in 1 seconds 2017-09-14 15:17:05,830 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 53 2017-09-14 15:17:05,830 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node2 checking its request amount 2017-09-14 15:17:05,830 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:17:05,830 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 61 to run in 60 seconds 2017-09-14 15:17:05,830 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 7 2017-09-14 15:17:05,830 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-09-14 15:17:05,830 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 8 to run in 60 seconds 2017-09-14 15:17:06,083 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 60 2017-09-14 15:17:06,084 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:17:06,084 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:17:06,084 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:17:06,084 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:17:06,085 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:17:06,085 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:17:06,085 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 62 to run in 10 seconds 2017-09-14 15:17:06,177 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action dump_json_file with id 55 2017-09-14 15:17:06,178 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action dump_json_file with id 63 to run in 60 seconds 2017-09-14 15:17:06,297 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:17:06,297 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:17:06,297 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:17:06,688 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 44 2017-09-14 15:17:06,689 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:06,689 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 45 to run in 1 seconds 2017-09-14 15:17:07,693 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 45 2017-09-14 15:17:07,694 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:07,694 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 46 to run in 1 seconds 2017-09-14 15:17:08,694 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 46 2017-09-14 15:17:08,694 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:08,694 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 47 to run in 1 seconds 2017-09-14 15:17:08,956 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2C got 1 messages through listener 2017-09-14 15:17:08,957 | TRACE | node.py (1388) | validateClientMsg | Node2C received CLIENT message: BATCH{'signature': None, 'messages': ['{"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402182349581,"signature":"5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb","operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe67"}}', '{"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402168868387,"signature":"36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8","operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe66"}}']} 2017-09-14 15:17:08,958 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1505402182349581 2017-09-14 15:17:08,958 | TRACE | node.py (1388) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:17:08,959 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1505402168868387 2017-09-14 15:17:08,959 | TRACE | node.py (1388) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe66', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} 2017-09-14 15:17:08,959 | DEBUG | node.py (1434) | processClientInBox | Node2C processing b'v#ZljK34WWooA6%RgESn.Ox' request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:17:08,960 | DEBUG | node.py (1708) | processRequest | Node2 received client request: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:17:08,960 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:17:08,960 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} to its replicas since already forwarded 2017-09-14 15:17:08,960 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402182349581}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:17:08,961 | DEBUG | node.py (1434) | processClientInBox | Node2C processing b'v#ZljK34WWooA6%RgESn.Ox' request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe66', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} 2017-09-14 15:17:08,961 | DEBUG | node.py (1708) | processRequest | Node2 received client request: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe66', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:17:08,961 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe66', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} 2017-09-14 15:17:08,961 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe66', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} to its replicas since already forwarded 2017-09-14 15:17:08,961 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402168868387}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:17:09,699 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 47 2017-09-14 15:17:09,699 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:09,699 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 48 to run in 1 seconds 2017-09-14 15:17:10,699 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 48 2017-09-14 15:17:10,700 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:10,700 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 49 to run in 1 seconds 2017-09-14 15:17:11,713 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 49 2017-09-14 15:17:11,714 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:11,714 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 50 to run in 1 seconds 2017-09-14 15:17:12,718 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 50 2017-09-14 15:17:12,718 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:12,718 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 51 to run in 1 seconds 2017-09-14 15:17:13,719 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 51 2017-09-14 15:17:13,720 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:13,720 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 52 to run in 1 seconds 2017-09-14 15:17:14,726 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 52 2017-09-14 15:17:14,726 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:14,726 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 53 to run in 1 seconds 2017-09-14 15:17:15,738 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 53 2017-09-14 15:17:15,738 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:15,738 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 54 to run in 1 seconds 2017-09-14 15:17:16,089 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 62 2017-09-14 15:17:16,089 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:17:16,089 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:17:16,089 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:17:16,089 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:17:16,090 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:17:16,090 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:17:16,090 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 64 to run in 10 seconds 2017-09-14 15:17:16,740 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 54 2017-09-14 15:17:16,740 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:16,741 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 55 to run in 1 seconds 2017-09-14 15:17:17,749 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 55 2017-09-14 15:17:17,750 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:17,750 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 56 to run in 1 seconds 2017-09-14 15:17:18,761 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 56 2017-09-14 15:17:18,761 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:18,761 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 57 to run in 1 seconds 2017-09-14 15:17:19,767 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 57 2017-09-14 15:17:19,768 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:19,768 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 58 to run in 1 seconds 2017-09-14 15:17:20,780 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 58 2017-09-14 15:17:20,780 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:20,780 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 59 to run in 1 seconds 2017-09-14 15:17:21,308 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:17:21,308 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:17:21,309 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:17:21,789 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 59 2017-09-14 15:17:21,790 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:21,790 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 60 to run in 1 seconds 2017-09-14 15:17:22,796 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 60 2017-09-14 15:17:22,796 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:22,796 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 61 to run in 1 seconds 2017-09-14 15:17:23,800 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 61 2017-09-14 15:17:23,801 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:23,801 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 62 to run in 1 seconds 2017-09-14 15:17:23,981 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2C got 1 messages through listener 2017-09-14 15:17:23,981 | TRACE | node.py (1388) | validateClientMsg | Node2C received CLIENT message: BATCH{'signature': None, 'messages': ['{"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402182349581,"signature":"5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb","operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe67"}}', '{"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402168868387,"signature":"36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8","operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe66"}}']} 2017-09-14 15:17:23,982 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1505402182349581 2017-09-14 15:17:23,982 | TRACE | node.py (1388) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:17:23,983 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1505402168868387 2017-09-14 15:17:23,984 | TRACE | node.py (1388) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe66', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} 2017-09-14 15:17:23,984 | DEBUG | node.py (1434) | processClientInBox | Node2C processing b'v#ZljK34WWooA6%RgESn.Ox' request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:17:23,984 | DEBUG | node.py (1708) | processRequest | Node2 received client request: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:17:23,984 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:17:23,984 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} to its replicas since already forwarded 2017-09-14 15:17:23,985 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402182349581}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:17:23,986 | DEBUG | node.py (1434) | processClientInBox | Node2C processing b'v#ZljK34WWooA6%RgESn.Ox' request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe66', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} 2017-09-14 15:17:23,987 | DEBUG | node.py (1708) | processRequest | Node2 received client request: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe66', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:17:23,987 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe66', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} 2017-09-14 15:17:23,987 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe66', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} to its replicas since already forwarded 2017-09-14 15:17:23,987 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402168868387}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:17:24,807 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 62 2017-09-14 15:17:24,808 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:24,808 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 63 to run in 1 seconds 2017-09-14 15:17:25,813 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 63 2017-09-14 15:17:25,813 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:25,813 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 64 to run in 1 seconds 2017-09-14 15:17:26,100 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 64 2017-09-14 15:17:26,100 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:17:26,100 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:17:26,101 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:17:26,101 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:17:26,101 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:17:26,101 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:17:26,101 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 65 to run in 10 seconds 2017-09-14 15:17:26,818 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 64 2017-09-14 15:17:26,819 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:26,819 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 65 to run in 1 seconds 2017-09-14 15:17:27,821 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 65 2017-09-14 15:17:27,821 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:27,821 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 66 to run in 1 seconds 2017-09-14 15:17:28,827 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 66 2017-09-14 15:17:28,827 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:28,827 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 67 to run in 1 seconds 2017-09-14 15:17:29,834 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 67 2017-09-14 15:17:29,834 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:29,834 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 68 to run in 1 seconds 2017-09-14 15:17:30,844 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 68 2017-09-14 15:17:30,844 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:30,844 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 69 to run in 1 seconds 2017-09-14 15:17:31,854 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 69 2017-09-14 15:17:31,855 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:31,855 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 70 to run in 1 seconds 2017-09-14 15:17:32,863 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 70 2017-09-14 15:17:32,864 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:32,864 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 71 to run in 1 seconds 2017-09-14 15:17:33,866 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 71 2017-09-14 15:17:33,867 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:33,867 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 72 to run in 1 seconds 2017-09-14 15:17:34,871 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 72 2017-09-14 15:17:34,872 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:34,872 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 73 to run in 1 seconds 2017-09-14 15:17:35,882 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 73 2017-09-14 15:17:35,883 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:35,883 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 74 to run in 1 seconds 2017-09-14 15:17:36,105 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 65 2017-09-14 15:17:36,105 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:17:36,105 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:17:36,106 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:17:36,106 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:17:36,106 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:17:36,106 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:17:36,106 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 66 to run in 10 seconds 2017-09-14 15:17:36,313 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:17:36,313 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:17:36,313 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:17:36,891 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 74 2017-09-14 15:17:36,891 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:36,891 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 75 to run in 1 seconds 2017-09-14 15:17:37,901 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 75 2017-09-14 15:17:37,901 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:37,902 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 76 to run in 1 seconds 2017-09-14 15:17:38,912 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 76 2017-09-14 15:17:38,912 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:38,912 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 77 to run in 1 seconds 2017-09-14 15:17:38,994 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2C got 1 messages through listener 2017-09-14 15:17:38,995 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1505402182349581 2017-09-14 15:17:38,996 | TRACE | node.py (1388) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:17:38,996 | DEBUG | node.py (1434) | processClientInBox | Node2C processing b'v#ZljK34WWooA6%RgESn.Ox' request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:17:38,996 | DEBUG | node.py (1708) | processRequest | Node2 received client request: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:17:38,996 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:17:38,997 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} to its replicas since already forwarded 2017-09-14 15:17:38,997 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402182349581}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:17:39,917 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 77 2017-09-14 15:17:39,917 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:39,917 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 78 to run in 1 seconds 2017-09-14 15:17:40,929 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 78 2017-09-14 15:17:40,930 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:40,930 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 79 to run in 1 seconds 2017-09-14 15:17:41,932 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 79 2017-09-14 15:17:41,932 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:41,932 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 80 to run in 1 seconds 2017-09-14 15:17:42,938 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 80 2017-09-14 15:17:42,938 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:42,938 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 81 to run in 1 seconds 2017-09-14 15:17:43,941 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 81 2017-09-14 15:17:43,942 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:43,942 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 82 to run in 1 seconds 2017-09-14 15:17:44,950 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 82 2017-09-14 15:17:44,950 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:44,950 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 83 to run in 1 seconds 2017-09-14 15:17:45,953 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 83 2017-09-14 15:17:45,954 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:45,954 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 84 to run in 1 seconds 2017-09-14 15:17:46,109 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 66 2017-09-14 15:17:46,109 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:17:46,110 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:17:46,110 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:17:46,110 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:17:46,110 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:17:46,110 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:17:46,110 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 67 to run in 10 seconds 2017-09-14 15:17:46,954 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 84 2017-09-14 15:17:46,954 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:46,954 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 85 to run in 1 seconds 2017-09-14 15:17:47,961 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 85 2017-09-14 15:17:47,961 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:47,961 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 86 to run in 1 seconds 2017-09-14 15:17:48,972 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 86 2017-09-14 15:17:48,973 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:48,973 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 87 to run in 1 seconds 2017-09-14 15:17:49,974 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 87 2017-09-14 15:17:49,974 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:49,974 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 88 to run in 1 seconds 2017-09-14 15:17:50,984 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 88 2017-09-14 15:17:50,984 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:50,985 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 89 to run in 1 seconds 2017-09-14 15:17:51,319 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:17:51,319 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:17:51,320 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:17:51,985 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 89 2017-09-14 15:17:51,986 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:51,986 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 90 to run in 1 seconds 2017-09-14 15:17:52,989 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 90 2017-09-14 15:17:52,989 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:52,989 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 91 to run in 1 seconds 2017-09-14 15:17:54,000 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 91 2017-09-14 15:17:54,000 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:54,001 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 92 to run in 1 seconds 2017-09-14 15:17:55,006 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 92 2017-09-14 15:17:55,006 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:55,007 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 93 to run in 1 seconds 2017-09-14 15:17:56,018 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 93 2017-09-14 15:17:56,018 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:56,018 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 94 to run in 1 seconds 2017-09-14 15:17:56,118 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 67 2017-09-14 15:17:56,118 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:17:56,118 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:17:56,118 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:17:56,119 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:17:56,119 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:17:56,119 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:17:56,119 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 68 to run in 10 seconds 2017-09-14 15:17:57,023 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 94 2017-09-14 15:17:57,023 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:57,023 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 95 to run in 1 seconds 2017-09-14 15:17:58,028 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 95 2017-09-14 15:17:58,029 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:58,029 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 96 to run in 1 seconds 2017-09-14 15:17:59,038 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 96 2017-09-14 15:17:59,039 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:17:59,039 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 97 to run in 1 seconds 2017-09-14 15:18:00,048 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 97 2017-09-14 15:18:00,048 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:00,048 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 98 to run in 1 seconds 2017-09-14 15:18:01,049 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 98 2017-09-14 15:18:01,050 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:01,050 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 99 to run in 1 seconds 2017-09-14 15:18:02,051 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 99 2017-09-14 15:18:02,051 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:02,051 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 100 to run in 1 seconds 2017-09-14 15:18:03,058 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 100 2017-09-14 15:18:03,058 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:03,058 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 101 to run in 1 seconds 2017-09-14 15:18:04,060 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 101 2017-09-14 15:18:04,061 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:04,061 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 102 to run in 1 seconds 2017-09-14 15:18:05,065 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 102 2017-09-14 15:18:05,065 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:05,065 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 103 to run in 1 seconds 2017-09-14 15:18:05,837 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 61 2017-09-14 15:18:05,837 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node2 checking its request amount 2017-09-14 15:18:05,838 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:18:05,838 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 69 to run in 60 seconds 2017-09-14 15:18:05,838 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 8 2017-09-14 15:18:05,838 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-09-14 15:18:05,838 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 9 to run in 60 seconds 2017-09-14 15:18:06,069 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 103 2017-09-14 15:18:06,069 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:06,069 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 104 to run in 1 seconds 2017-09-14 15:18:06,120 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 68 2017-09-14 15:18:06,120 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:18:06,120 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:18:06,120 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:18:06,120 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:18:06,120 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:18:06,121 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:18:06,121 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 70 to run in 10 seconds 2017-09-14 15:18:06,179 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action dump_json_file with id 63 2017-09-14 15:18:06,181 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action dump_json_file with id 71 to run in 60 seconds 2017-09-14 15:18:06,325 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:18:06,325 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:18:06,325 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:18:07,080 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 104 2017-09-14 15:18:07,080 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:07,081 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 105 to run in 1 seconds 2017-09-14 15:18:08,086 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 105 2017-09-14 15:18:08,087 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:08,087 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 106 to run in 1 seconds 2017-09-14 15:18:09,093 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 106 2017-09-14 15:18:09,093 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:09,093 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 107 to run in 1 seconds 2017-09-14 15:18:10,102 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 107 2017-09-14 15:18:10,103 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:10,103 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 108 to run in 1 seconds 2017-09-14 15:18:11,111 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 108 2017-09-14 15:18:11,112 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:11,112 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 109 to run in 1 seconds 2017-09-14 15:18:12,113 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 109 2017-09-14 15:18:12,113 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:12,113 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 110 to run in 1 seconds 2017-09-14 15:18:13,118 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 110 2017-09-14 15:18:13,118 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:13,119 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 111 to run in 1 seconds 2017-09-14 15:18:14,124 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 111 2017-09-14 15:18:14,124 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:14,124 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 112 to run in 1 seconds 2017-09-14 15:18:15,134 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 112 2017-09-14 15:18:15,135 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:15,135 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 113 to run in 1 seconds 2017-09-14 15:18:16,128 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 70 2017-09-14 15:18:16,128 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:18:16,128 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:18:16,128 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:18:16,128 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:18:16,129 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:18:16,129 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:18:16,129 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 72 to run in 10 seconds 2017-09-14 15:18:16,141 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 113 2017-09-14 15:18:16,141 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:16,141 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 114 to run in 1 seconds 2017-09-14 15:18:17,145 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 114 2017-09-14 15:18:17,146 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:17,146 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 115 to run in 1 seconds 2017-09-14 15:18:18,152 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 115 2017-09-14 15:18:18,152 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:18,153 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 116 to run in 1 seconds 2017-09-14 15:18:19,164 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 116 2017-09-14 15:18:19,165 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:19,165 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 117 to run in 1 seconds 2017-09-14 15:18:20,176 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 117 2017-09-14 15:18:20,177 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:20,177 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 118 to run in 1 seconds 2017-09-14 15:18:21,182 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 118 2017-09-14 15:18:21,182 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:21,182 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 119 to run in 1 seconds 2017-09-14 15:18:21,325 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:18:21,325 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:18:21,325 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:18:22,191 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 119 2017-09-14 15:18:22,192 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:22,192 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 120 to run in 1 seconds 2017-09-14 15:18:23,197 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 120 2017-09-14 15:18:23,197 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:23,197 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 121 to run in 1 seconds 2017-09-14 15:18:24,206 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 121 2017-09-14 15:18:24,206 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:24,206 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 122 to run in 1 seconds 2017-09-14 15:18:25,213 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 122 2017-09-14 15:18:25,214 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:25,214 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 123 to run in 1 seconds 2017-09-14 15:18:26,140 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 72 2017-09-14 15:18:26,141 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:18:26,141 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:18:26,141 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:18:26,141 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:18:26,141 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:18:26,141 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:18:26,142 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 73 to run in 10 seconds 2017-09-14 15:18:26,216 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 123 2017-09-14 15:18:26,216 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:26,216 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 124 to run in 1 seconds 2017-09-14 15:18:27,222 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 124 2017-09-14 15:18:27,223 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:27,223 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 125 to run in 1 seconds 2017-09-14 15:18:28,227 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 125 2017-09-14 15:18:28,228 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:28,228 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 126 to run in 1 seconds 2017-09-14 15:18:29,233 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 126 2017-09-14 15:18:29,233 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:29,233 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 127 to run in 1 seconds 2017-09-14 15:18:30,235 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 127 2017-09-14 15:18:30,235 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:30,235 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 128 to run in 1 seconds 2017-09-14 15:18:31,237 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 128 2017-09-14 15:18:31,238 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:31,238 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 129 to run in 1 seconds 2017-09-14 15:18:32,243 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 129 2017-09-14 15:18:32,243 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:32,243 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 130 to run in 1 seconds 2017-09-14 15:18:33,251 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 130 2017-09-14 15:18:33,251 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:33,251 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 131 to run in 1 seconds 2017-09-14 15:18:34,257 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 131 2017-09-14 15:18:34,258 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:34,258 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 132 to run in 1 seconds 2017-09-14 15:18:35,263 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 132 2017-09-14 15:18:35,263 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:35,263 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 133 to run in 1 seconds 2017-09-14 15:18:36,145 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 73 2017-09-14 15:18:36,145 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:18:36,145 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:18:36,145 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:18:36,145 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:18:36,145 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:18:36,145 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:18:36,145 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 74 to run in 10 seconds 2017-09-14 15:18:36,266 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 133 2017-09-14 15:18:36,267 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:36,267 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 134 to run in 1 seconds 2017-09-14 15:18:36,329 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:18:36,329 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:18:36,329 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:18:37,269 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 134 2017-09-14 15:18:37,269 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:37,269 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 135 to run in 1 seconds 2017-09-14 15:18:38,279 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 135 2017-09-14 15:18:38,280 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:38,280 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 136 to run in 1 seconds 2017-09-14 15:18:39,282 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 136 2017-09-14 15:18:39,282 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:39,282 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 137 to run in 1 seconds 2017-09-14 15:18:40,286 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 137 2017-09-14 15:18:40,286 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:40,286 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 138 to run in 1 seconds 2017-09-14 15:18:41,290 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 138 2017-09-14 15:18:41,290 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:41,290 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 139 to run in 1 seconds 2017-09-14 15:18:42,295 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 139 2017-09-14 15:18:42,295 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:42,296 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 140 to run in 1 seconds 2017-09-14 15:18:43,302 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 140 2017-09-14 15:18:43,302 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:43,302 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 141 to run in 1 seconds 2017-09-14 15:18:44,312 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 141 2017-09-14 15:18:44,312 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:44,312 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 142 to run in 1 seconds 2017-09-14 15:18:45,316 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 142 2017-09-14 15:18:45,317 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:45,317 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 143 to run in 1 seconds 2017-09-14 15:18:46,156 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 74 2017-09-14 15:18:46,156 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:18:46,156 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:18:46,156 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:18:46,157 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:18:46,157 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:18:46,157 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:18:46,157 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 75 to run in 10 seconds 2017-09-14 15:18:46,320 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 143 2017-09-14 15:18:46,320 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:46,320 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 144 to run in 1 seconds 2017-09-14 15:18:47,323 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 144 2017-09-14 15:18:47,324 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:47,324 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 145 to run in 1 seconds 2017-09-14 15:18:48,326 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 145 2017-09-14 15:18:48,327 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:48,327 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 146 to run in 1 seconds 2017-09-14 15:18:49,331 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 146 2017-09-14 15:18:49,331 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:49,331 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 147 to run in 1 seconds 2017-09-14 15:18:50,341 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 147 2017-09-14 15:18:50,341 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:50,342 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 148 to run in 1 seconds 2017-09-14 15:18:51,333 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:18:51,334 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:18:51,334 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:18:51,346 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 148 2017-09-14 15:18:51,347 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:51,347 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 149 to run in 1 seconds 2017-09-14 15:18:52,354 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 149 2017-09-14 15:18:52,355 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:52,355 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 150 to run in 1 seconds 2017-09-14 15:18:53,358 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 150 2017-09-14 15:18:53,359 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:53,359 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 151 to run in 1 seconds 2017-09-14 15:18:54,370 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 151 2017-09-14 15:18:54,370 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:54,370 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 152 to run in 1 seconds 2017-09-14 15:18:55,374 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 152 2017-09-14 15:18:55,374 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:55,374 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 153 to run in 1 seconds 2017-09-14 15:18:56,161 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 75 2017-09-14 15:18:56,162 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:18:56,162 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:18:56,162 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:18:56,162 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:18:56,162 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:18:56,162 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:18:56,163 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 76 to run in 10 seconds 2017-09-14 15:18:56,378 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 153 2017-09-14 15:18:56,378 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:56,379 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 154 to run in 1 seconds 2017-09-14 15:18:57,387 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 154 2017-09-14 15:18:57,387 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:57,387 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 155 to run in 1 seconds 2017-09-14 15:18:58,396 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 155 2017-09-14 15:18:58,397 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:58,397 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 156 to run in 1 seconds 2017-09-14 15:18:59,406 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 156 2017-09-14 15:18:59,406 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:18:59,407 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 157 to run in 1 seconds 2017-09-14 15:19:00,413 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 157 2017-09-14 15:19:00,413 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:00,414 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 158 to run in 1 seconds 2017-09-14 15:19:01,422 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 158 2017-09-14 15:19:01,422 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:01,423 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 159 to run in 1 seconds 2017-09-14 15:19:02,423 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 159 2017-09-14 15:19:02,424 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:02,424 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 160 to run in 1 seconds 2017-09-14 15:19:03,424 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 160 2017-09-14 15:19:03,425 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:03,425 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 161 to run in 1 seconds 2017-09-14 15:19:04,433 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 161 2017-09-14 15:19:04,434 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:04,434 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 162 to run in 1 seconds 2017-09-14 15:19:05,444 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 162 2017-09-14 15:19:05,446 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:05,446 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 163 to run in 1 seconds 2017-09-14 15:19:05,840 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 69 2017-09-14 15:19:05,840 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node2 checking its request amount 2017-09-14 15:19:05,840 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:19:05,840 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 77 to run in 60 seconds 2017-09-14 15:19:05,840 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 9 2017-09-14 15:19:05,840 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-09-14 15:19:05,840 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 10 to run in 60 seconds 2017-09-14 15:19:06,167 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 76 2017-09-14 15:19:06,167 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:19:06,167 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:19:06,167 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:19:06,167 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:19:06,168 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:19:06,168 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:19:06,168 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 78 to run in 10 seconds 2017-09-14 15:19:06,193 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action dump_json_file with id 71 2017-09-14 15:19:06,194 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action dump_json_file with id 79 to run in 60 seconds 2017-09-14 15:19:06,338 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:19:06,339 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:19:06,339 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:19:06,448 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 163 2017-09-14 15:19:06,448 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:06,448 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 164 to run in 1 seconds 2017-09-14 15:19:07,455 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 164 2017-09-14 15:19:07,455 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:07,455 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 165 to run in 1 seconds 2017-09-14 15:19:08,466 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 165 2017-09-14 15:19:08,466 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:08,467 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 166 to run in 1 seconds 2017-09-14 15:19:09,474 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 166 2017-09-14 15:19:09,475 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:09,475 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 167 to run in 1 seconds 2017-09-14 15:19:10,486 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 167 2017-09-14 15:19:10,487 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:10,488 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 168 to run in 1 seconds 2017-09-14 15:19:11,489 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 168 2017-09-14 15:19:11,489 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:11,489 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 169 to run in 1 seconds 2017-09-14 15:19:12,492 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 169 2017-09-14 15:19:12,492 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:12,493 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 170 to run in 1 seconds 2017-09-14 15:19:13,503 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 170 2017-09-14 15:19:13,503 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:13,504 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 171 to run in 1 seconds 2017-09-14 15:19:14,513 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 171 2017-09-14 15:19:14,513 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:14,513 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 172 to run in 1 seconds 2017-09-14 15:19:15,521 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 172 2017-09-14 15:19:15,521 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:15,521 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 173 to run in 1 seconds 2017-09-14 15:19:16,179 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 78 2017-09-14 15:19:16,179 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:19:16,179 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:19:16,179 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:19:16,179 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:19:16,180 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:19:16,180 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:19:16,180 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 80 to run in 10 seconds 2017-09-14 15:19:16,531 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 173 2017-09-14 15:19:16,531 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:16,531 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 174 to run in 1 seconds 2017-09-14 15:19:17,541 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 174 2017-09-14 15:19:17,542 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:17,542 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 175 to run in 1 seconds 2017-09-14 15:19:18,553 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 175 2017-09-14 15:19:18,553 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:18,553 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 176 to run in 1 seconds 2017-09-14 15:19:19,557 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 176 2017-09-14 15:19:19,557 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:19,558 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 177 to run in 1 seconds 2017-09-14 15:19:20,563 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 177 2017-09-14 15:19:20,563 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:20,564 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 178 to run in 1 seconds 2017-09-14 15:19:21,342 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:19:21,342 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:19:21,342 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:19:21,565 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 178 2017-09-14 15:19:21,565 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:21,565 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 179 to run in 1 seconds 2017-09-14 15:19:22,572 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 179 2017-09-14 15:19:22,572 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:22,573 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 180 to run in 1 seconds 2017-09-14 15:19:23,576 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 180 2017-09-14 15:19:23,576 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:23,576 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 181 to run in 1 seconds 2017-09-14 15:19:24,576 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 181 2017-09-14 15:19:24,577 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:24,577 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 182 to run in 1 seconds 2017-09-14 15:19:25,586 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 182 2017-09-14 15:19:25,586 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:25,586 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 183 to run in 1 seconds 2017-09-14 15:19:26,183 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 80 2017-09-14 15:19:26,183 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:19:26,183 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:19:26,184 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:19:26,184 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:19:26,184 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:19:26,184 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:19:26,184 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 81 to run in 10 seconds 2017-09-14 15:19:26,592 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 183 2017-09-14 15:19:26,593 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:26,593 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 184 to run in 1 seconds 2017-09-14 15:19:27,601 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 184 2017-09-14 15:19:27,602 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:27,602 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 185 to run in 1 seconds 2017-09-14 15:19:28,607 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 185 2017-09-14 15:19:28,607 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:28,607 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 186 to run in 1 seconds 2017-09-14 15:19:29,614 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 186 2017-09-14 15:19:29,614 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:29,615 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 187 to run in 1 seconds 2017-09-14 15:19:30,623 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 187 2017-09-14 15:19:30,623 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:30,624 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 188 to run in 1 seconds 2017-09-14 15:19:31,632 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 188 2017-09-14 15:19:31,632 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:31,633 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 189 to run in 1 seconds 2017-09-14 15:19:32,635 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 189 2017-09-14 15:19:32,635 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:32,635 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 190 to run in 1 seconds 2017-09-14 15:19:33,640 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 190 2017-09-14 15:19:33,641 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:33,641 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 191 to run in 1 seconds 2017-09-14 15:19:34,642 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 191 2017-09-14 15:19:34,643 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:34,643 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 192 to run in 1 seconds 2017-09-14 15:19:35,654 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 192 2017-09-14 15:19:35,655 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:35,655 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 193 to run in 1 seconds 2017-09-14 15:19:36,194 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 81 2017-09-14 15:19:36,194 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:19:36,194 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:19:36,194 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:19:36,195 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:19:36,195 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:19:36,195 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:19:36,195 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 82 to run in 10 seconds 2017-09-14 15:19:36,353 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:19:36,354 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:19:36,354 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:19:36,665 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 193 2017-09-14 15:19:36,666 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:36,666 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 194 to run in 1 seconds 2017-09-14 15:19:37,675 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 194 2017-09-14 15:19:37,676 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:37,676 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 195 to run in 1 seconds 2017-09-14 15:19:38,687 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 195 2017-09-14 15:19:38,688 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:38,688 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 196 to run in 1 seconds 2017-09-14 15:19:39,696 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 196 2017-09-14 15:19:39,697 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:39,697 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 197 to run in 1 seconds 2017-09-14 15:19:40,700 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 197 2017-09-14 15:19:40,701 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:40,701 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 198 to run in 1 seconds 2017-09-14 15:19:41,704 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 198 2017-09-14 15:19:41,704 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:41,704 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 199 to run in 1 seconds 2017-09-14 15:19:42,708 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 199 2017-09-14 15:19:42,708 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:42,708 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 200 to run in 1 seconds 2017-09-14 15:19:43,710 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 200 2017-09-14 15:19:43,711 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:43,711 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 201 to run in 1 seconds 2017-09-14 15:19:44,718 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 201 2017-09-14 15:19:44,719 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:44,719 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 202 to run in 1 seconds 2017-09-14 15:19:45,730 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 202 2017-09-14 15:19:45,730 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:45,730 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 203 to run in 1 seconds 2017-09-14 15:19:46,195 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 82 2017-09-14 15:19:46,196 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:19:46,196 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:19:46,196 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:19:46,196 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:19:46,196 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:19:46,196 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:19:46,197 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 83 to run in 10 seconds 2017-09-14 15:19:46,742 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 203 2017-09-14 15:19:46,742 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:46,742 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 204 to run in 1 seconds 2017-09-14 15:19:47,753 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 204 2017-09-14 15:19:47,753 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:47,753 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 205 to run in 1 seconds 2017-09-14 15:19:48,754 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 205 2017-09-14 15:19:48,754 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:48,755 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 206 to run in 1 seconds 2017-09-14 15:19:49,758 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 206 2017-09-14 15:19:49,759 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:49,759 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 207 to run in 1 seconds 2017-09-14 15:19:50,766 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 207 2017-09-14 15:19:50,766 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:50,767 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 208 to run in 1 seconds 2017-09-14 15:19:51,354 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:19:51,354 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:19:51,354 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:19:51,776 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 208 2017-09-14 15:19:51,776 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:51,777 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 209 to run in 1 seconds 2017-09-14 15:19:52,777 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 209 2017-09-14 15:19:52,777 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:52,777 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 210 to run in 1 seconds 2017-09-14 15:19:53,782 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 210 2017-09-14 15:19:53,782 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:53,782 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 211 to run in 1 seconds 2017-09-14 15:19:54,794 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 211 2017-09-14 15:19:54,794 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:54,794 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 212 to run in 1 seconds 2017-09-14 15:19:55,807 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 212 2017-09-14 15:19:55,808 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:55,808 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 213 to run in 1 seconds 2017-09-14 15:19:56,208 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 83 2017-09-14 15:19:56,208 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:19:56,209 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:19:56,209 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:19:56,209 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:19:56,209 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:19:56,209 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:19:56,209 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 84 to run in 10 seconds 2017-09-14 15:19:56,811 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 213 2017-09-14 15:19:56,811 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:56,811 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 214 to run in 1 seconds 2017-09-14 15:19:57,817 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 214 2017-09-14 15:19:57,818 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:57,818 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 215 to run in 1 seconds 2017-09-14 15:19:58,824 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 215 2017-09-14 15:19:58,824 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:58,824 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 216 to run in 1 seconds 2017-09-14 15:19:59,837 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 216 2017-09-14 15:19:59,837 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:19:59,837 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 217 to run in 1 seconds 2017-09-14 15:20:00,842 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 217 2017-09-14 15:20:00,842 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:00,843 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 218 to run in 1 seconds 2017-09-14 15:20:01,851 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 218 2017-09-14 15:20:01,851 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:01,852 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 219 to run in 1 seconds 2017-09-14 15:20:02,861 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 219 2017-09-14 15:20:02,861 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:02,862 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 220 to run in 1 seconds 2017-09-14 15:20:03,876 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 220 2017-09-14 15:20:03,876 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:03,876 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 221 to run in 1 seconds 2017-09-14 15:20:04,876 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 221 2017-09-14 15:20:04,876 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:04,876 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 222 to run in 1 seconds 2017-09-14 15:20:05,844 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 77 2017-09-14 15:20:05,844 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node2 checking its request amount 2017-09-14 15:20:05,844 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:20:05,845 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 85 to run in 60 seconds 2017-09-14 15:20:05,845 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 10 2017-09-14 15:20:05,846 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-09-14 15:20:05,846 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 11 to run in 60 seconds 2017-09-14 15:20:05,881 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 222 2017-09-14 15:20:05,882 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:05,882 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 223 to run in 1 seconds 2017-09-14 15:20:06,196 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action dump_json_file with id 79 2017-09-14 15:20:06,197 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action dump_json_file with id 86 to run in 60 seconds 2017-09-14 15:20:06,209 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 84 2017-09-14 15:20:06,210 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:20:06,210 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:20:06,210 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:20:06,210 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:20:06,210 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:20:06,210 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:20:06,210 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 87 to run in 10 seconds 2017-09-14 15:20:06,370 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:20:06,370 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:20:06,371 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:20:06,892 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 223 2017-09-14 15:20:06,892 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:06,892 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 224 to run in 1 seconds 2017-09-14 15:20:07,897 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 224 2017-09-14 15:20:07,897 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:07,897 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 225 to run in 1 seconds 2017-09-14 15:20:08,899 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 225 2017-09-14 15:20:08,899 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:08,899 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 226 to run in 1 seconds 2017-09-14 15:20:09,900 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 226 2017-09-14 15:20:09,901 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:09,901 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 227 to run in 1 seconds 2017-09-14 15:20:10,904 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 227 2017-09-14 15:20:10,904 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:10,904 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 228 to run in 1 seconds 2017-09-14 15:20:11,910 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 228 2017-09-14 15:20:11,910 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:11,910 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 229 to run in 1 seconds 2017-09-14 15:20:12,911 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 229 2017-09-14 15:20:12,911 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:12,911 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 230 to run in 1 seconds 2017-09-14 15:20:13,919 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 230 2017-09-14 15:20:13,919 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:13,919 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 231 to run in 1 seconds 2017-09-14 15:20:14,921 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 231 2017-09-14 15:20:14,921 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:14,922 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 232 to run in 1 seconds 2017-09-14 15:20:15,928 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 232 2017-09-14 15:20:15,928 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:15,928 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 233 to run in 1 seconds 2017-09-14 15:20:16,221 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 87 2017-09-14 15:20:16,222 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:20:16,222 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:20:16,222 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:20:16,222 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:20:16,222 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:20:16,222 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:20:16,222 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 88 to run in 10 seconds 2017-09-14 15:20:16,932 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 233 2017-09-14 15:20:16,932 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:16,933 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 234 to run in 1 seconds 2017-09-14 15:20:17,933 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 234 2017-09-14 15:20:17,933 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:17,933 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 235 to run in 1 seconds 2017-09-14 15:20:18,936 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 235 2017-09-14 15:20:18,937 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:18,937 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 236 to run in 1 seconds 2017-09-14 15:20:19,940 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 236 2017-09-14 15:20:19,940 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:19,940 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 237 to run in 1 seconds 2017-09-14 15:20:20,950 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 237 2017-09-14 15:20:20,950 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:20,950 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 238 to run in 1 seconds 2017-09-14 15:20:21,379 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:20:21,380 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:20:21,380 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:20:21,964 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 238 2017-09-14 15:20:21,965 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:21,965 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 239 to run in 1 seconds 2017-09-14 15:20:22,973 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 239 2017-09-14 15:20:22,973 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:22,973 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 240 to run in 1 seconds 2017-09-14 15:20:23,986 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 240 2017-09-14 15:20:23,986 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:23,986 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 241 to run in 1 seconds 2017-09-14 15:20:24,994 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 241 2017-09-14 15:20:24,994 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:24,994 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 242 to run in 1 seconds 2017-09-14 15:20:25,995 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 242 2017-09-14 15:20:25,996 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:25,996 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 243 to run in 1 seconds 2017-09-14 15:20:26,230 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 88 2017-09-14 15:20:26,230 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:20:26,230 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:20:26,231 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:20:26,231 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:20:26,231 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:20:26,231 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:20:26,231 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 89 to run in 10 seconds 2017-09-14 15:20:26,996 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 243 2017-09-14 15:20:26,997 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:26,997 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 244 to run in 1 seconds 2017-09-14 15:20:28,009 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 244 2017-09-14 15:20:28,009 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:28,009 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 245 to run in 1 seconds 2017-09-14 15:20:29,016 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 245 2017-09-14 15:20:29,016 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:29,016 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 246 to run in 1 seconds 2017-09-14 15:20:30,023 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 246 2017-09-14 15:20:30,023 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:30,023 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 247 to run in 1 seconds 2017-09-14 15:20:31,031 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 247 2017-09-14 15:20:31,031 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:31,032 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 248 to run in 1 seconds 2017-09-14 15:20:32,033 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 248 2017-09-14 15:20:32,033 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:32,033 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 249 to run in 1 seconds 2017-09-14 15:20:33,038 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 249 2017-09-14 15:20:33,038 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:33,038 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 250 to run in 1 seconds 2017-09-14 15:20:34,044 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 250 2017-09-14 15:20:34,044 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:34,045 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 251 to run in 1 seconds 2017-09-14 15:20:35,059 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 251 2017-09-14 15:20:35,059 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:35,060 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 252 to run in 1 seconds 2017-09-14 15:20:36,064 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 252 2017-09-14 15:20:36,065 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:36,065 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 253 to run in 1 seconds 2017-09-14 15:20:36,233 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 89 2017-09-14 15:20:36,234 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:20:36,234 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:20:36,234 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:20:36,234 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:20:36,234 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:20:36,234 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:20:36,234 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 90 to run in 10 seconds 2017-09-14 15:20:36,382 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:20:36,382 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:20:36,383 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:20:37,073 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 253 2017-09-14 15:20:37,073 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:37,073 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 254 to run in 1 seconds 2017-09-14 15:20:38,083 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 254 2017-09-14 15:20:38,083 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:38,083 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 255 to run in 1 seconds 2017-09-14 15:20:39,085 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 255 2017-09-14 15:20:39,085 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:39,085 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 256 to run in 1 seconds 2017-09-14 15:20:40,093 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 256 2017-09-14 15:20:40,094 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:40,094 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 257 to run in 1 seconds 2017-09-14 15:20:41,104 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 257 2017-09-14 15:20:41,104 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:41,104 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 258 to run in 1 seconds 2017-09-14 15:20:42,115 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 258 2017-09-14 15:20:42,116 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:42,116 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 259 to run in 1 seconds 2017-09-14 15:20:43,118 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 259 2017-09-14 15:20:43,118 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:43,118 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 260 to run in 1 seconds 2017-09-14 15:20:44,130 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 260 2017-09-14 15:20:44,130 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:44,130 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 261 to run in 1 seconds 2017-09-14 15:20:45,133 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 261 2017-09-14 15:20:45,133 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:45,133 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 262 to run in 1 seconds 2017-09-14 15:20:46,140 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 262 2017-09-14 15:20:46,142 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:46,142 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 263 to run in 1 seconds 2017-09-14 15:20:46,238 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 90 2017-09-14 15:20:46,239 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:20:46,239 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:20:46,239 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:20:46,239 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:20:46,239 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:20:46,239 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:20:46,239 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 91 to run in 10 seconds 2017-09-14 15:20:47,150 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 263 2017-09-14 15:20:47,150 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:47,150 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 264 to run in 1 seconds 2017-09-14 15:20:48,151 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 264 2017-09-14 15:20:48,151 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:48,152 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 265 to run in 1 seconds 2017-09-14 15:20:49,160 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 265 2017-09-14 15:20:49,160 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:49,160 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 266 to run in 1 seconds 2017-09-14 15:20:50,167 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 266 2017-09-14 15:20:50,167 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:50,168 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 267 to run in 1 seconds 2017-09-14 15:20:51,173 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 267 2017-09-14 15:20:51,173 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:51,174 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 268 to run in 1 seconds 2017-09-14 15:20:51,392 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:20:51,392 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:20:51,393 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:20:52,181 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 268 2017-09-14 15:20:52,181 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:52,181 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 269 to run in 1 seconds 2017-09-14 15:20:53,187 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 269 2017-09-14 15:20:53,187 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:53,187 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 270 to run in 1 seconds 2017-09-14 15:20:54,191 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 270 2017-09-14 15:20:54,191 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:54,191 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 271 to run in 1 seconds 2017-09-14 15:20:55,194 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 271 2017-09-14 15:20:55,194 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:55,194 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 272 to run in 1 seconds 2017-09-14 15:20:56,207 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 272 2017-09-14 15:20:56,208 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:56,208 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 273 to run in 1 seconds 2017-09-14 15:20:56,252 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 91 2017-09-14 15:20:56,252 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:20:56,252 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:20:56,252 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:20:56,252 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:20:56,252 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:20:56,252 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:20:56,253 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 92 to run in 10 seconds 2017-09-14 15:20:57,212 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 273 2017-09-14 15:20:57,212 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:57,212 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 274 to run in 1 seconds 2017-09-14 15:20:58,222 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 274 2017-09-14 15:20:58,222 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:58,222 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 275 to run in 1 seconds 2017-09-14 15:20:59,235 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 275 2017-09-14 15:20:59,235 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:20:59,235 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 276 to run in 1 seconds 2017-09-14 15:21:00,240 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 276 2017-09-14 15:21:00,241 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:00,241 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 277 to run in 1 seconds 2017-09-14 15:21:01,250 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 277 2017-09-14 15:21:01,250 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:01,250 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 278 to run in 1 seconds 2017-09-14 15:21:02,250 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 278 2017-09-14 15:21:02,251 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:02,251 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 279 to run in 1 seconds 2017-09-14 15:21:03,252 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 279 2017-09-14 15:21:03,253 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:03,253 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 280 to run in 1 seconds 2017-09-14 15:21:04,255 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 280 2017-09-14 15:21:04,256 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:04,256 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 281 to run in 1 seconds 2017-09-14 15:21:05,259 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 281 2017-09-14 15:21:05,260 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:05,260 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 282 to run in 1 seconds 2017-09-14 15:21:05,849 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 85 2017-09-14 15:21:05,850 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node2 checking its request amount 2017-09-14 15:21:05,850 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:21:05,850 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 93 to run in 60 seconds 2017-09-14 15:21:05,850 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 11 2017-09-14 15:21:05,850 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-09-14 15:21:05,850 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 12 to run in 60 seconds 2017-09-14 15:21:06,202 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action dump_json_file with id 86 2017-09-14 15:21:06,204 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action dump_json_file with id 94 to run in 60 seconds 2017-09-14 15:21:06,254 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 92 2017-09-14 15:21:06,254 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:21:06,254 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:21:06,254 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:21:06,254 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:21:06,255 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:21:06,255 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:21:06,255 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 95 to run in 10 seconds 2017-09-14 15:21:06,269 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 282 2017-09-14 15:21:06,269 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:06,269 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 283 to run in 1 seconds 2017-09-14 15:21:06,394 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:21:06,394 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:21:06,395 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:21:07,274 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 283 2017-09-14 15:21:07,274 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:07,274 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 284 to run in 1 seconds 2017-09-14 15:21:08,283 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 284 2017-09-14 15:21:08,284 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:08,284 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 285 to run in 1 seconds 2017-09-14 15:21:09,293 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 285 2017-09-14 15:21:09,294 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:09,295 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 286 to run in 1 seconds 2017-09-14 15:21:10,301 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 286 2017-09-14 15:21:10,301 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:10,302 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 287 to run in 1 seconds 2017-09-14 15:21:11,308 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 287 2017-09-14 15:21:11,309 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:11,309 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 288 to run in 1 seconds 2017-09-14 15:21:12,320 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 288 2017-09-14 15:21:12,320 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:12,320 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 289 to run in 1 seconds 2017-09-14 15:21:13,324 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 289 2017-09-14 15:21:13,325 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:13,325 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 290 to run in 1 seconds 2017-09-14 15:21:14,332 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 290 2017-09-14 15:21:14,333 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:14,333 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 291 to run in 1 seconds 2017-09-14 15:21:15,343 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 291 2017-09-14 15:21:15,344 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:15,344 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 292 to run in 1 seconds 2017-09-14 15:21:16,262 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 95 2017-09-14 15:21:16,262 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:21:16,262 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:21:16,262 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:21:16,263 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:21:16,263 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:21:16,263 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:21:16,263 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 96 to run in 10 seconds 2017-09-14 15:21:16,351 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 292 2017-09-14 15:21:16,351 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:16,351 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 293 to run in 1 seconds 2017-09-14 15:21:17,362 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 293 2017-09-14 15:21:17,363 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:17,363 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 294 to run in 1 seconds 2017-09-14 15:21:18,366 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 294 2017-09-14 15:21:18,366 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:18,366 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 295 to run in 1 seconds 2017-09-14 15:21:19,370 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 295 2017-09-14 15:21:19,370 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:19,370 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 296 to run in 1 seconds 2017-09-14 15:21:20,371 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 296 2017-09-14 15:21:20,372 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:20,372 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 297 to run in 1 seconds 2017-09-14 15:21:21,378 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 297 2017-09-14 15:21:21,379 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:21,379 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 298 to run in 1 seconds 2017-09-14 15:21:21,403 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:21:21,403 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:21:21,403 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:21:22,381 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 298 2017-09-14 15:21:22,381 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:22,381 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 299 to run in 1 seconds 2017-09-14 15:21:23,386 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 299 2017-09-14 15:21:23,386 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:23,386 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 300 to run in 1 seconds 2017-09-14 15:21:24,387 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 300 2017-09-14 15:21:24,387 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:24,387 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 301 to run in 1 seconds 2017-09-14 15:21:25,397 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 301 2017-09-14 15:21:25,397 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:25,397 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 302 to run in 1 seconds 2017-09-14 15:21:26,264 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 96 2017-09-14 15:21:26,264 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:21:26,265 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:21:26,265 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:21:26,265 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:21:26,265 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:21:26,265 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:21:26,265 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 97 to run in 10 seconds 2017-09-14 15:21:26,402 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 302 2017-09-14 15:21:26,402 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:26,402 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 303 to run in 1 seconds 2017-09-14 15:21:27,414 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 303 2017-09-14 15:21:27,415 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:27,415 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 304 to run in 1 seconds 2017-09-14 15:21:28,427 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 304 2017-09-14 15:21:28,427 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:28,427 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 305 to run in 1 seconds 2017-09-14 15:21:29,437 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 305 2017-09-14 15:21:29,437 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:29,437 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 306 to run in 1 seconds 2017-09-14 15:21:30,445 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 306 2017-09-14 15:21:30,445 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:30,446 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 307 to run in 1 seconds 2017-09-14 15:21:31,450 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 307 2017-09-14 15:21:31,450 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:31,450 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 308 to run in 1 seconds 2017-09-14 15:21:32,460 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 308 2017-09-14 15:21:32,460 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:32,460 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 309 to run in 1 seconds 2017-09-14 15:21:33,468 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 309 2017-09-14 15:21:33,468 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:33,468 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 310 to run in 1 seconds 2017-09-14 15:21:34,470 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 310 2017-09-14 15:21:34,471 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:34,471 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 311 to run in 1 seconds 2017-09-14 15:21:35,482 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 311 2017-09-14 15:21:35,483 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:35,483 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 312 to run in 1 seconds 2017-09-14 15:21:36,275 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 97 2017-09-14 15:21:36,275 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:21:36,275 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:21:36,275 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:21:36,275 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:21:36,275 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:21:36,275 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:21:36,275 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 98 to run in 10 seconds 2017-09-14 15:21:36,411 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:21:36,412 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:21:36,412 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:21:36,484 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 312 2017-09-14 15:21:36,484 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:36,485 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 313 to run in 1 seconds 2017-09-14 15:21:37,493 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 313 2017-09-14 15:21:37,493 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:37,493 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 314 to run in 1 seconds 2017-09-14 15:21:38,499 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 314 2017-09-14 15:21:38,499 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:38,500 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 315 to run in 1 seconds 2017-09-14 15:21:39,504 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 315 2017-09-14 15:21:39,504 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:39,504 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 316 to run in 1 seconds 2017-09-14 15:21:40,513 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 316 2017-09-14 15:21:40,513 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:40,513 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 317 to run in 1 seconds 2017-09-14 15:21:41,523 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 317 2017-09-14 15:21:41,523 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:41,523 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 318 to run in 1 seconds 2017-09-14 15:21:42,525 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 318 2017-09-14 15:21:42,525 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:42,525 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 319 to run in 1 seconds 2017-09-14 15:21:43,529 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 319 2017-09-14 15:21:43,529 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:43,529 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 320 to run in 1 seconds 2017-09-14 15:21:44,536 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 320 2017-09-14 15:21:44,536 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:44,537 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 321 to run in 1 seconds 2017-09-14 15:21:45,538 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 321 2017-09-14 15:21:45,538 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:45,539 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 322 to run in 1 seconds 2017-09-14 15:21:46,282 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 98 2017-09-14 15:21:46,282 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:21:46,282 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:21:46,282 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:21:46,282 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:21:46,283 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:21:46,283 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:21:46,283 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 99 to run in 10 seconds 2017-09-14 15:21:46,549 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 322 2017-09-14 15:21:46,550 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:46,550 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 323 to run in 1 seconds 2017-09-14 15:21:47,555 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 323 2017-09-14 15:21:47,555 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:47,555 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 324 to run in 1 seconds 2017-09-14 15:21:48,560 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 324 2017-09-14 15:21:48,561 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:48,561 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 325 to run in 1 seconds 2017-09-14 15:21:49,565 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 325 2017-09-14 15:21:49,566 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:49,566 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 326 to run in 1 seconds 2017-09-14 15:21:50,567 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 326 2017-09-14 15:21:50,567 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:50,567 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 327 to run in 1 seconds 2017-09-14 15:21:51,414 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:21:51,414 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:21:51,415 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:21:51,570 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 327 2017-09-14 15:21:51,571 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:51,571 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 328 to run in 1 seconds 2017-09-14 15:21:52,572 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 328 2017-09-14 15:21:52,572 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:52,573 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 329 to run in 1 seconds 2017-09-14 15:21:53,574 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 329 2017-09-14 15:21:53,575 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:53,575 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 330 to run in 1 seconds 2017-09-14 15:21:54,582 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 330 2017-09-14 15:21:54,582 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:54,582 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 331 to run in 1 seconds 2017-09-14 15:21:55,584 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 331 2017-09-14 15:21:55,584 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:55,585 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 332 to run in 1 seconds 2017-09-14 15:21:56,296 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 99 2017-09-14 15:21:56,296 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:21:56,297 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:21:56,297 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:21:56,297 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:21:56,297 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:21:56,297 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:21:56,297 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 100 to run in 10 seconds 2017-09-14 15:21:56,596 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 332 2017-09-14 15:21:56,596 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:56,596 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 333 to run in 1 seconds 2017-09-14 15:21:57,601 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 333 2017-09-14 15:21:57,601 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:57,601 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 334 to run in 1 seconds 2017-09-14 15:21:58,611 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 334 2017-09-14 15:21:58,612 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:58,612 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 335 to run in 1 seconds 2017-09-14 15:21:59,615 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 335 2017-09-14 15:21:59,616 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:21:59,616 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 336 to run in 1 seconds 2017-09-14 15:22:00,618 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 336 2017-09-14 15:22:00,618 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:00,618 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 337 to run in 1 seconds 2017-09-14 15:22:01,630 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 337 2017-09-14 15:22:01,630 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:01,631 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 338 to run in 1 seconds 2017-09-14 15:22:02,632 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 338 2017-09-14 15:22:02,632 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:02,632 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 339 to run in 1 seconds 2017-09-14 15:22:03,638 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 339 2017-09-14 15:22:03,638 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:03,638 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 340 to run in 1 seconds 2017-09-14 15:22:04,640 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 340 2017-09-14 15:22:04,640 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:04,640 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 341 to run in 1 seconds 2017-09-14 15:22:05,642 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 341 2017-09-14 15:22:05,642 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:05,642 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 342 to run in 1 seconds 2017-09-14 15:22:05,861 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 93 2017-09-14 15:22:05,861 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node2 checking its request amount 2017-09-14 15:22:05,861 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:22:05,861 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 101 to run in 60 seconds 2017-09-14 15:22:05,861 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 12 2017-09-14 15:22:05,862 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-09-14 15:22:05,862 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 13 to run in 60 seconds 2017-09-14 15:22:06,207 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action dump_json_file with id 94 2017-09-14 15:22:06,208 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action dump_json_file with id 102 to run in 60 seconds 2017-09-14 15:22:06,298 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 100 2017-09-14 15:22:06,298 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:22:06,298 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:22:06,298 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:22:06,298 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:22:06,299 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:22:06,299 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:22:06,299 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 103 to run in 10 seconds 2017-09-14 15:22:06,418 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:22:06,418 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:22:06,419 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:22:06,646 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 342 2017-09-14 15:22:06,647 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:06,647 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 343 to run in 1 seconds 2017-09-14 15:22:07,656 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 343 2017-09-14 15:22:07,657 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:07,657 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 344 to run in 1 seconds 2017-09-14 15:22:08,670 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 344 2017-09-14 15:22:08,670 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:08,670 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 345 to run in 1 seconds 2017-09-14 15:22:09,673 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 345 2017-09-14 15:22:09,673 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:09,673 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 346 to run in 1 seconds 2017-09-14 15:22:10,678 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 346 2017-09-14 15:22:10,678 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:10,679 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 347 to run in 1 seconds 2017-09-14 15:22:11,691 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 347 2017-09-14 15:22:11,692 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:11,692 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 348 to run in 1 seconds 2017-09-14 15:22:12,693 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 348 2017-09-14 15:22:12,693 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:12,694 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 349 to run in 1 seconds 2017-09-14 15:22:13,699 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 349 2017-09-14 15:22:13,700 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:13,700 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 350 to run in 1 seconds 2017-09-14 15:22:14,706 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 350 2017-09-14 15:22:14,706 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:14,706 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 351 to run in 1 seconds 2017-09-14 15:22:15,710 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 351 2017-09-14 15:22:15,711 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:15,711 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 352 to run in 1 seconds 2017-09-14 15:22:16,309 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 103 2017-09-14 15:22:16,309 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:22:16,310 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:22:16,310 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:22:16,310 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:22:16,310 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:22:16,310 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:22:16,310 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 104 to run in 10 seconds 2017-09-14 15:22:16,723 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 352 2017-09-14 15:22:16,723 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:16,723 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 353 to run in 1 seconds 2017-09-14 15:22:17,733 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 353 2017-09-14 15:22:17,733 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:17,734 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 354 to run in 1 seconds 2017-09-14 15:22:18,738 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 354 2017-09-14 15:22:18,739 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:18,739 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 355 to run in 1 seconds 2017-09-14 15:22:19,744 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 355 2017-09-14 15:22:19,744 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:19,744 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 356 to run in 1 seconds 2017-09-14 15:22:20,745 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 356 2017-09-14 15:22:20,746 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:20,746 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 357 to run in 1 seconds 2017-09-14 15:22:21,421 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:22:21,421 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:22:21,421 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:22:21,753 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 357 2017-09-14 15:22:21,753 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:21,753 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 358 to run in 1 seconds 2017-09-14 15:22:22,755 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 358 2017-09-14 15:22:22,755 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:22,755 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 359 to run in 1 seconds 2017-09-14 15:22:23,762 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 359 2017-09-14 15:22:23,762 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:23,762 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 360 to run in 1 seconds 2017-09-14 15:22:24,770 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 360 2017-09-14 15:22:24,771 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:24,771 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 361 to run in 1 seconds 2017-09-14 15:22:25,781 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 361 2017-09-14 15:22:25,781 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:25,781 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 362 to run in 1 seconds 2017-09-14 15:22:26,317 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 104 2017-09-14 15:22:26,317 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:22:26,317 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:22:26,317 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:22:26,317 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:22:26,317 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:22:26,317 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:22:26,318 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 105 to run in 10 seconds 2017-09-14 15:22:26,782 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 362 2017-09-14 15:22:26,783 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:26,783 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 363 to run in 1 seconds 2017-09-14 15:22:27,790 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 363 2017-09-14 15:22:27,790 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:27,791 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 364 to run in 1 seconds 2017-09-14 15:22:28,795 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 364 2017-09-14 15:22:28,795 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:28,795 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 365 to run in 1 seconds 2017-09-14 15:22:29,797 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 365 2017-09-14 15:22:29,798 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:29,798 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 366 to run in 1 seconds 2017-09-14 15:22:30,798 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 366 2017-09-14 15:22:30,799 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:30,799 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 367 to run in 1 seconds 2017-09-14 15:22:31,801 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 367 2017-09-14 15:22:31,802 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:31,802 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 368 to run in 1 seconds 2017-09-14 15:22:32,810 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 368 2017-09-14 15:22:32,810 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:32,810 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 369 to run in 1 seconds 2017-09-14 15:22:33,818 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 369 2017-09-14 15:22:33,819 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:33,819 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 370 to run in 1 seconds 2017-09-14 15:22:34,829 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 370 2017-09-14 15:22:34,829 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:34,829 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 371 to run in 1 seconds 2017-09-14 15:22:35,840 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 371 2017-09-14 15:22:35,841 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:35,841 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 372 to run in 1 seconds 2017-09-14 15:22:36,323 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 105 2017-09-14 15:22:36,323 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:22:36,323 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:22:36,323 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:22:36,323 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:22:36,324 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:22:36,324 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:22:36,324 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 106 to run in 10 seconds 2017-09-14 15:22:36,422 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:22:36,422 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:22:36,422 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:22:36,843 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 372 2017-09-14 15:22:36,843 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:36,844 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 373 to run in 1 seconds 2017-09-14 15:22:37,857 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 373 2017-09-14 15:22:37,857 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:37,857 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 374 to run in 1 seconds 2017-09-14 15:22:38,867 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 374 2017-09-14 15:22:38,867 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:38,867 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 375 to run in 1 seconds 2017-09-14 15:22:39,875 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 375 2017-09-14 15:22:39,876 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:39,876 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 376 to run in 1 seconds 2017-09-14 15:22:40,888 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 376 2017-09-14 15:22:40,888 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:40,888 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 377 to run in 1 seconds 2017-09-14 15:22:41,903 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 377 2017-09-14 15:22:41,903 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:41,904 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 378 to run in 1 seconds 2017-09-14 15:22:42,909 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 378 2017-09-14 15:22:42,909 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:42,909 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 379 to run in 1 seconds 2017-09-14 15:22:43,917 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 379 2017-09-14 15:22:43,918 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:43,918 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 380 to run in 1 seconds 2017-09-14 15:22:44,927 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 380 2017-09-14 15:22:44,927 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:44,928 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 381 to run in 1 seconds 2017-09-14 15:22:45,936 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 381 2017-09-14 15:22:45,936 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:45,936 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 382 to run in 1 seconds 2017-09-14 15:22:46,330 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 106 2017-09-14 15:22:46,330 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:22:46,330 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:22:46,330 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:22:46,331 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:22:46,331 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:22:46,331 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:22:46,331 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 107 to run in 10 seconds 2017-09-14 15:22:46,938 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 382 2017-09-14 15:22:46,938 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:46,939 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 383 to run in 1 seconds 2017-09-14 15:22:47,949 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 383 2017-09-14 15:22:47,950 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:47,950 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 384 to run in 1 seconds 2017-09-14 15:22:48,961 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 384 2017-09-14 15:22:48,962 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:48,962 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 385 to run in 1 seconds 2017-09-14 15:22:49,962 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 385 2017-09-14 15:22:49,963 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:49,963 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 386 to run in 1 seconds 2017-09-14 15:22:50,969 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 386 2017-09-14 15:22:50,969 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:50,969 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 387 to run in 1 seconds 2017-09-14 15:22:51,431 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:22:51,432 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:22:51,432 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:22:51,970 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 387 2017-09-14 15:22:51,970 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:51,970 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 388 to run in 1 seconds 2017-09-14 15:22:52,972 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 388 2017-09-14 15:22:52,973 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:52,973 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 389 to run in 1 seconds 2017-09-14 15:22:53,976 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 389 2017-09-14 15:22:53,976 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:53,976 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 390 to run in 1 seconds 2017-09-14 15:22:54,987 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 390 2017-09-14 15:22:54,987 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:54,988 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 391 to run in 1 seconds 2017-09-14 15:22:55,996 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 391 2017-09-14 15:22:55,996 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:55,996 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 392 to run in 1 seconds 2017-09-14 15:22:56,340 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 107 2017-09-14 15:22:56,340 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:22:56,340 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:22:56,340 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:22:56,340 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:22:56,341 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:22:56,341 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:22:56,341 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 108 to run in 10 seconds 2017-09-14 15:22:57,006 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 392 2017-09-14 15:22:57,006 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:57,006 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 393 to run in 1 seconds 2017-09-14 15:22:58,012 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 393 2017-09-14 15:22:58,012 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:58,012 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 394 to run in 1 seconds 2017-09-14 15:22:59,018 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 394 2017-09-14 15:22:59,018 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:22:59,019 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 395 to run in 1 seconds 2017-09-14 15:23:00,028 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 395 2017-09-14 15:23:00,028 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:23:00,028 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 396 to run in 1 seconds 2017-09-14 15:23:01,039 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 396 2017-09-14 15:23:01,040 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:23:01,040 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 397 to run in 1 seconds 2017-09-14 15:23:02,043 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 397 2017-09-14 15:23:02,043 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:23:02,043 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 398 to run in 1 seconds 2017-09-14 15:23:03,052 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 398 2017-09-14 15:23:03,052 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:23:03,052 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 399 to run in 1 seconds 2017-09-14 15:23:04,059 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 399 2017-09-14 15:23:04,059 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:23:04,059 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 400 to run in 1 seconds 2017-09-14 15:23:05,065 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 400 2017-09-14 15:23:05,065 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:23:05,065 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 401 to run in 1 seconds 2017-09-14 15:23:05,863 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 101 2017-09-14 15:23:05,864 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node2 checking its request amount 2017-09-14 15:23:05,864 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:23:05,864 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 109 to run in 60 seconds 2017-09-14 15:23:05,864 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 13 2017-09-14 15:23:05,865 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-09-14 15:23:05,865 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 14 to run in 60 seconds 2017-09-14 15:23:06,071 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 401 2017-09-14 15:23:06,071 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:23:06,072 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 402 to run in 1 seconds 2017-09-14 15:23:06,217 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action dump_json_file with id 102 2017-09-14 15:23:06,218 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action dump_json_file with id 110 to run in 60 seconds 2017-09-14 15:23:06,347 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 108 2017-09-14 15:23:06,347 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:23:06,347 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:23:06,347 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:23:06,347 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:23:06,347 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:23:06,348 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:23:06,348 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 111 to run in 10 seconds 2017-09-14 15:23:06,443 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:23:06,443 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:23:06,443 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:23:07,083 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 402 2017-09-14 15:23:07,083 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:23:07,083 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 403 to run in 1 seconds 2017-09-14 15:23:08,093 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 403 2017-09-14 15:23:08,094 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:23:08,094 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 404 to run in 1 seconds 2017-09-14 15:23:09,096 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 404 2017-09-14 15:23:09,096 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:23:09,096 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 405 to run in 1 seconds 2017-09-14 15:23:10,097 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 405 2017-09-14 15:23:10,097 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:23:10,097 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 406 to run in 1 seconds 2017-09-14 15:23:11,107 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 406 2017-09-14 15:23:11,107 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:23:11,107 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 407 to run in 1 seconds 2017-09-14 15:23:12,109 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 407 2017-09-14 15:23:12,109 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:23:12,109 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 408 to run in 1 seconds 2017-09-14 15:23:13,109 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 408 2017-09-14 15:23:13,110 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:23:13,110 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 409 to run in 1 seconds 2017-09-14 15:23:14,111 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 409 2017-09-14 15:23:14,111 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:23:14,112 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 410 to run in 1 seconds 2017-09-14 15:23:15,112 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 410 2017-09-14 15:23:15,112 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:23:15,113 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 411 to run in 1 seconds 2017-09-14 15:23:16,117 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 411 2017-09-14 15:23:16,118 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:23:16,118 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 412 to run in 1 seconds 2017-09-14 15:23:16,350 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 111 2017-09-14 15:23:16,351 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:23:16,351 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:23:16,351 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:23:16,351 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:23:16,351 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:23:16,351 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:23:16,351 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 112 to run in 10 seconds 2017-09-14 15:23:17,123 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 412 2017-09-14 15:23:17,124 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:23:17,124 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 413 to run in 1 seconds 2017-09-14 15:23:18,133 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 413 2017-09-14 15:23:18,134 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:23:18,134 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 414 to run in 1 seconds 2017-09-14 15:23:19,135 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 414 2017-09-14 15:23:19,136 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:23:19,136 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 415 to run in 1 seconds 2017-09-14 15:23:20,140 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 415 2017-09-14 15:23:20,141 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:23:20,141 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 416 to run in 1 seconds 2017-09-14 15:23:21,150 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 416 2017-09-14 15:23:21,151 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:23:21,151 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 417 to run in 1 seconds 2017-09-14 15:23:21,444 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-09-14 15:23:21,444 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:23:21,444 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-09-14 15:23:22,162 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 417 2017-09-14 15:23:22,162 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:23:22,162 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 418 to run in 1 seconds 2017-09-14 15:23:23,173 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 418 2017-09-14 15:23:23,174 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:23:23,174 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 419 to run in 1 seconds 2017-09-14 15:23:24,180 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 419 2017-09-14 15:23:24,181 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:23:24,181 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 420 to run in 1 seconds 2017-09-14 15:23:25,182 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 420 2017-09-14 15:23:25,182 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:23:25,182 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 421 to run in 1 seconds 2017-09-14 15:23:26,185 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 421 2017-09-14 15:23:26,186 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:23:26,186 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 422 to run in 1 seconds 2017-09-14 15:23:26,354 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 112 2017-09-14 15:23:26,354 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-09-14 15:23:26,354 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:23:26,354 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-09-14 15:23:26,354 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:23:26,354 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:23:26,354 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-09-14 15:23:26,355 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 113 to run in 10 seconds 2017-09-14 15:23:27,192 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 422 2017-09-14 15:23:27,192 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:23:27,192 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 423 to run in 1 seconds 2017-09-14 15:23:28,202 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 423 2017-09-14 15:23:28,202 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:23:28,203 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 424 to run in 1 seconds 2017-09-14 15:23:29,211 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2:0 running action process_stashed_out_of_order_commits with id 424 2017-09-14 15:23:29,212 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node2:0 trying to order from out of order commits. OrderedSet([(0, 1), (0, 2), (0, 3), (1, 1), (1, 2)]) {1: {4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 1}}} 2017-09-14 15:23:29,212 | TRACE | has_action_queue.py ( 36) | _schedule | Node2:0 scheduling action process_stashed_out_of_order_commits with id 425 to run in 1 seconds