2017-09-14 15:10:01,555 | DEBUG | node_runner.py ( 18) | run_node | You can find logs in /home/sovrin/.sovrin/Node1.log 2017-09-14 15:10:01,556 | DEBUG | node_runner.py ( 21) | run_node | Sovrin related env vars: [] 2017-09-14 15:10:03,387 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: git 2017-09-14 15:10:03,471 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: hg 2017-09-14 15:10:03,795 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: svn 2017-09-14 15:10:03,796 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: bzr 2017-09-14 15:10:04,991 | DEBUG | selector_events.py ( 53) | __init__ | Using selector: EpollSelector 2017-09-14 15:10:04,991 | DEBUG | looper.py ( 125) | __init__ | Setting handler for SIGINT 2017-09-14 15:10:05,094 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-09-14 15:10:05,131 | DEBUG | ledger.py ( 67) | recoverTree | Recovering tree from transaction log 2017-09-14 15:10:05,212 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.08130777500082331 seconds 2017-09-14 15:10:05,329 | DEBUG | idr_cache.py ( 25) | __init__ | Initializing identity cache Node1 2017-09-14 15:10:05,356 | INFO | node.py (2420) | initStateFromLedger | Node1 found state to be empty, recreating from ledger 2017-09-14 15:10:05,491 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-09-14 15:10:05,522 | DEBUG | ledger.py ( 67) | recoverTree | Recovering tree from transaction log 2017-09-14 15:10:05,604 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.08254399999896123 seconds 2017-09-14 15:10:05,607 | INFO | node.py (2420) | initStateFromLedger | Node1 found state to be empty, recreating from ledger 2017-09-14 15:10:05,726 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-09-14 15:10:05,726 | INFO | plugin_loader.py ( 117) | _load | plugin FirebaseStatsConsumer successfully loaded from module plugin_firebase_stats_consumer 2017-09-14 15:10:05,726 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-09-14 15:10:05,727 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: typing.Dict<~KT, ~VT>] because it does not have a 'pluginType' attribute 2017-09-14 15:10:05,727 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-09-14 15:10:05,727 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-09-14 15:10:05,727 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkPerformance will be repeating every 60 seconds 2017-09-14 15:10:05,727 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 1 to run in 60 seconds 2017-09-14 15:10:05,728 | DEBUG | replica.py ( 313) | h | Node1:0 set watermarks as 0 300 2017-09-14 15:10:05,728 | DISPLAY | replicas.py ( 36) | grow | Node1 added replica Node1:0 to instance 0 (master) 2017-09-14 15:10:05,728 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkPerformance will be repeating every 10 seconds 2017-09-14 15:10:05,728 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 1 to run in 10 seconds 2017-09-14 15:10:05,728 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkNodeRequestSpike will be repeating every 60 seconds 2017-09-14 15:10:05,729 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkNodeRequestSpike with id 2 to run in 60 seconds 2017-09-14 15:10:05,729 | DEBUG | plugin_helper.py ( 24) | loadPlugins | Plugin loading started to load plugins from basedir: /home/sovrin/.sovrin 2017-09-14 15:10:05,729 | DEBUG | plugin_helper.py ( 33) | loadPlugins | Plugin directory created at: /home/sovrin/.sovrin/plugins 2017-09-14 15:10:05,733 | DEBUG | plugin_helper.py ( 68) | loadPlugins | Total plugins loaded from basedir /home/sovrin/.sovrin are : 0 2017-09-14 15:10:05,733 | DEBUG | node.py ( 340) | __init__ | total plugins loaded in node: 0 2017-09-14 15:10:05,854 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-09-14 15:10:05,888 | DEBUG | ledger.py ( 67) | recoverTree | Recovering tree from transaction log 2017-09-14 15:10:05,988 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.1001437340000848 seconds 2017-09-14 15:10:06,017 | DEBUG | upgrader.py ( 118) | check_upgrade_succeeded | Node Node1 has no upgrade events 2017-09-14 15:10:06,017 | INFO | node.py (2420) | initStateFromLedger | Node1 found state to be empty, recreating from ledger 2017-09-14 15:10:06,017 | DEBUG | motor.py ( 34) | set_status | Node1 changing status from stopped to starting 2017-09-14 15:10:06,017 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-09-14 15:10:06,017 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-09-14 15:10:06,017 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-09-14 15:10:06,018 | DEBUG | zstack.py ( 319) | start | Node1 starting with restricted as True and reSetupAuth as True 2017-09-14 15:10:06,018 | DEBUG | authenticator.py ( 31) | start | Starting ZAP at inproc://zeromq.zap.1 2017-09-14 15:10:06,018 | DEBUG | base.py ( 72) | allow | Allowing 0.0.0.0 2017-09-14 15:10:06,018 | DEBUG | base.py ( 112) | configure_curve | Configure curve: *[/home/sovrin/.sovrin/Node1/public_keys] 2017-09-14 15:10:06,019 | DEBUG | zstack.py ( 347) | open | Node1 will bind its listener at 9701 2017-09-14 15:10:06,019 | INFO | stacks.py ( 84) | start | CONNECTION: Node1 listening for other nodes at 0.0.0.0:9701 2017-09-14 15:10:06,019 | DEBUG | zstack.py ( 319) | start | Node1C starting with restricted as False and reSetupAuth as True 2017-09-14 15:10:06,020 | DEBUG | authenticator.py ( 31) | start | Starting ZAP at inproc://zeromq.zap.2 2017-09-14 15:10:06,020 | DEBUG | base.py ( 72) | allow | Allowing 0.0.0.0 2017-09-14 15:10:06,020 | DEBUG | base.py ( 112) | configure_curve | Configure curve: *[*] 2017-09-14 15:10:06,020 | DEBUG | zstack.py ( 347) | open | Node1C will bind its listener at 9702 2017-09-14 15:10:06,020 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action propose_view_change with id 3 to run in 60 seconds 2017-09-14 15:10:06,020 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action dump_json_file with id 4 to run in 3 seconds 2017-09-14 15:10:06,020 | DEBUG | has_action_queue.py ( 79) | startRepeating | dump_json_file will be repeating every 60 seconds 2017-09-14 15:10:06,021 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action dump_json_file with id 5 to run in 60 seconds 2017-09-14 15:10:06,021 | INFO | node.py ( 621) | start | Node1 first time running... 2017-09-14 15:10:06,022 | DEBUG | kit_zstack.py ( 97) | connectToMissing | CONNECTION: Node1 found the following missing connections: Node2, Node3 2017-09-14 15:10:06,023 | TRACE | remote.py ( 86) | connect | connecting socket 78 53389504 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-09-14 15:10:06,023 | INFO | zstack.py ( 590) | connect | CONNECTION: Node1 looking for Node2 at 10.0.0.3:9703 2017-09-14 15:10:06,024 | DEBUG | zstack.py ( 645) | sendPingPong | Node1 pinged Node2 2017-09-14 15:10:06,024 | TRACE | remote.py ( 86) | connect | connecting socket 79 53411664 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-09-14 15:10:06,024 | INFO | zstack.py ( 590) | connect | CONNECTION: Node1 looking for Node3 at 10.0.0.4:9705 2017-09-14 15:10:06,025 | DEBUG | zstack.py ( 645) | sendPingPong | Node1 pinged Node3 2017-09-14 15:10:06,025 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:10:06,036 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'pi' to Node2 2017-09-14 15:10:06,036 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'pi' to Node2 2017-09-14 15:10:06,036 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'pi' to Node3 2017-09-14 15:10:06,036 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'pi' to Node3 2017-09-14 15:10:06,164 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 1 messages through listener 2017-09-14 15:10:06,164 | DEBUG | zstack.py ( 663) | handlePingPong | Node1 got ping from Node2 2017-09-14 15:10:06,164 | DEBUG | zstack.py ( 645) | sendPingPong | Node1 ponged Node2 2017-09-14 15:10:06,164 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'po' to Node2 2017-09-14 15:10:06,165 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'po' to Node2 2017-09-14 15:10:06,187 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 1 messages through listener 2017-09-14 15:10:06,187 | DEBUG | zstack.py ( 663) | handlePingPong | Node1 got ping from Node3 2017-09-14 15:10:06,187 | DEBUG | zstack.py ( 645) | sendPingPong | Node1 ponged Node3 2017-09-14 15:10:06,187 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'po' to Node3 2017-09-14 15:10:06,187 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'po' to Node3 2017-09-14 15:10:06,199 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 1 messages through listener 2017-09-14 15:10:06,200 | DEBUG | zstack.py ( 669) | handlePingPong | Node1 got pong from Node3 2017-09-14 15:10:06,200 | DEBUG | keep_in_touch.py ( 68) | conns | Node1's connections changed from set() to {'Node3'} 2017-09-14 15:10:06,200 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node1 now connected to Node3 2017-09-14 15:10:06,200 | DEBUG | motor.py ( 34) | set_status | Node1 changing status from starting to started_hungry 2017-09-14 15:10:06,201 | DEBUG | node.py ( 941) | checkInstances | Node1 choosing to start election on the basis of count 2 and nodes {'Node3'} 2017-09-14 15:10:06,201 | DEBUG | primary_selector.py ( 77) | get_msgs_for_lagged_nodes | Node1 has no ViewChangeDone message to send for view 0 2017-09-14 15:10:06,201 | DEBUG | node.py ( 902) | send_current_state_to_lagging_node | Node1 sending current state CURRENT_STATE{'primary': [], 'viewNo': 0} to lagged node Node3 2017-09-14 15:10:06,201 | DEBUG | node.py (2608) | send | Node1 sending message CURRENT_STATE{'primary': [], 'viewNo': 0} to 1 recipients: ['Node3'] 2017-09-14 15:10:06,201 | DEBUG | node.py (2608) | send | Node1 sending message LEDGER_STATUS{'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ledgerId': 0, 'txnSeqNo': 3, 'ppSeqNo': None, 'viewNo': None} to 1 recipients: ['Node3'] 2017-09-14 15:10:06,202 | DEBUG | batched.py ( 100) | flushOutBoxes | Node1 batching 2 msgs to Node3 into one transmission 2017-09-14 15:10:06,202 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"viewNo":0,"op":"CURRENT_STATE","primary":[]}', b'{"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","op":"LEDGER_STATUS","viewNo":null,"ledgerId":0,"txnSeqNo":3,"ppSeqNo":null}']) 2017-09-14 15:10:06,202 | TRACE | batched.py ( 110) | flushOutBoxes | Node1 sending payload to Node3: b'{"signature":null,"op":"BATCH","messages":["{\\"viewNo\\":0,\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[]}","{\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null,\\"ledgerId\\":0,\\"txnSeqNo\\":3,\\"ppSeqNo\\":null}"]}' 2017-09-14 15:10:06,202 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"signature":null,"op":"BATCH","messages":["{\\"viewNo\\":0,\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[]}","{\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null,\\"ledgerId\\":0,\\"txnSeqNo\\":3,\\"ppSeqNo\\":null}"]}' to Node3 2017-09-14 15:10:06,214 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 1 messages through listener 2017-09-14 15:10:06,215 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: BATCH{'signature': None, 'messages': ['{"primary":[],"op":"CURRENT_STATE","viewNo":0}', '{"op":"LEDGER_STATUS","txnSeqNo":3,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","viewNo":null,"ppSeqNo":null,"ledgerId":0}']} 2017-09-14 15:10:06,215 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'op': 'BATCH', 'messages': ['{"primary":[],"op":"CURRENT_STATE","viewNo":0}', '{"op":"LEDGER_STATUS","txnSeqNo":3,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","viewNo":null,"ppSeqNo":null,"ledgerId":0}']}, 'Node3') 2017-09-14 15:10:06,215 | DEBUG | node.py (1272) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"primary":[],"op":"CURRENT_STATE","viewNo":0}', '{"op":"LEDGER_STATUS","txnSeqNo":3,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","viewNo":null,"ppSeqNo":null,"ledgerId":0}']} 2017-09-14 15:10:06,215 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: CURRENT_STATE{'primary': [], 'viewNo': 0} 2017-09-14 15:10:06,215 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'viewNo': 0, 'op': 'CURRENT_STATE', 'primary': []}, 'Node3') 2017-09-14 15:10:06,216 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox CURRENT_STATE{'primary': [], 'viewNo': 0} 2017-09-14 15:10:06,216 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: LEDGER_STATUS{'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ledgerId': 0, 'txnSeqNo': 3, 'ppSeqNo': None, 'viewNo': None} 2017-09-14 15:10:06,216 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'op': 'LEDGER_STATUS', 'viewNo': None, 'ledgerId': 0, 'txnSeqNo': 3, 'ppSeqNo': None}, 'Node3') 2017-09-14 15:10:06,216 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox LEDGER_STATUS{'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ledgerId': 0, 'txnSeqNo': 3, 'ppSeqNo': None, 'viewNo': None} 2017-09-14 15:10:06,216 | DEBUG | node.py ( 907) | process_current_state_message | Node1 processing current state CURRENT_STATE{'primary': [], 'viewNo': 0} from Node3 2017-09-14 15:10:06,219 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ledgerId': 0, 'txnSeqNo': 3, 'ppSeqNo': None, 'viewNo': None} from Node3 2017-09-14 15:10:06,219 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 3 2017-09-14 15:10:06,219 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 3 2017-09-14 15:10:06,268 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 2 messages through listener 2017-09-14 15:10:06,269 | DEBUG | zstack.py ( 669) | handlePingPong | Node1 got pong from Node2 2017-09-14 15:10:06,269 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: BATCH{'signature': None, 'messages': ['{"op":"CURRENT_STATE","primary":[],"viewNo":0}', '{"ledgerId":0,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","txnSeqNo":3,"viewNo":null}']} 2017-09-14 15:10:06,269 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'op': 'BATCH', 'messages': ['{"op":"CURRENT_STATE","primary":[],"viewNo":0}', '{"ledgerId":0,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","txnSeqNo":3,"viewNo":null}']}, 'Node2') 2017-09-14 15:10:06,269 | DEBUG | node.py (1272) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"CURRENT_STATE","primary":[],"viewNo":0}', '{"ledgerId":0,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","txnSeqNo":3,"viewNo":null}']} 2017-09-14 15:10:06,269 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: CURRENT_STATE{'primary': [], 'viewNo': 0} 2017-09-14 15:10:06,269 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'viewNo': 0, 'op': 'CURRENT_STATE', 'primary': []}, 'Node2') 2017-09-14 15:10:06,269 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox CURRENT_STATE{'primary': [], 'viewNo': 0} 2017-09-14 15:10:06,270 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: LEDGER_STATUS{'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ledgerId': 0, 'txnSeqNo': 3, 'ppSeqNo': None, 'viewNo': None} 2017-09-14 15:10:06,270 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'op': 'LEDGER_STATUS', 'viewNo': None, 'ledgerId': 0, 'txnSeqNo': 3, 'ppSeqNo': None}, 'Node2') 2017-09-14 15:10:06,270 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox LEDGER_STATUS{'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ledgerId': 0, 'txnSeqNo': 3, 'ppSeqNo': None, 'viewNo': None} 2017-09-14 15:10:06,270 | DEBUG | node.py ( 907) | process_current_state_message | Node1 processing current state CURRENT_STATE{'primary': [], 'viewNo': 0} from Node2 2017-09-14 15:10:06,270 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ledgerId': 0, 'txnSeqNo': 3, 'ppSeqNo': None, 'viewNo': None} from Node2 2017-09-14 15:10:06,270 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 3 2017-09-14 15:10:06,270 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 3 2017-09-14 15:10:06,271 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node2', 'Node3'} that its ledger of type 0 is latest 2017-09-14 15:10:06,271 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node1 found from ledger status LEDGER_STATUS{'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ledgerId': 0, 'txnSeqNo': 3, 'ppSeqNo': None, 'viewNo': None} that it does not need catchup 2017-09-14 15:10:06,271 | DEBUG | node.py (1479) | preLedgerCatchUp | Node1 going to process any ordered requests before starting catchup. 2017-09-14 15:10:06,271 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node1:0 going to remove 0 Ordered messages from outbox 2017-09-14 15:10:06,271 | DEBUG | node.py (1888) | force_process_ordered | Node1 processed 0 Ordered batches for instance 0 before starting catch up 2017-09-14 15:10:06,271 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node1 processed 0 stashed ordered requests 2017-09-14 15:10:06,282 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-09-14 15:10:06,282 | INFO | node.py (1489) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 0 2017-09-14 15:10:06,282 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 3 2017-09-14 15:10:06,283 | DEBUG | node.py ( 941) | checkInstances | Node1 choosing to start election on the basis of count 2 and nodes {'Node3'} 2017-09-14 15:10:06,283 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to all recipients: ['Node2', 'Node3'] 2017-09-14 15:10:06,283 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node1 asking Node1 for ledger status of ledger 2 2017-09-14 15:10:06,283 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node2'] 2017-09-14 15:10:06,283 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node1 asking Node2 for ledger status of ledger 2 2017-09-14 15:10:06,283 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-09-14 15:10:06,283 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node1 asking Node3 for ledger status of ledger 2 2017-09-14 15:10:06,283 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node1 going to process 0 stashed ledger statuses for ledger 2 2017-09-14 15:10:06,284 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 0, caught up 0 in total 2017-09-14 15:10:06,284 | DEBUG | batched.py ( 100) | flushOutBoxes | Node1 batching 2 msgs to Node2 into one transmission 2017-09-14 15:10:06,284 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}']) 2017-09-14 15:10:06,284 | TRACE | batched.py ( 110) | flushOutBoxes | Node1 sending payload to Node2: b'{"signature":null,"op":"BATCH","messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}"]}' 2017-09-14 15:10:06,290 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"signature":null,"op":"BATCH","messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}"]}' to Node2 2017-09-14 15:10:06,291 | DEBUG | batched.py ( 100) | flushOutBoxes | Node1 batching 2 msgs to Node3 into one transmission 2017-09-14 15:10:06,291 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}']) 2017-09-14 15:10:06,291 | TRACE | batched.py ( 110) | flushOutBoxes | Node1 sending payload to Node3: b'{"signature":null,"op":"BATCH","messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}"]}' 2017-09-14 15:10:06,291 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"signature":null,"op":"BATCH","messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}"]}' to Node3 2017-09-14 15:10:06,291 | DEBUG | keep_in_touch.py ( 68) | conns | Node1's connections changed from {'Node3'} to {'Node2', 'Node3'} 2017-09-14 15:10:06,292 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node1 now connected to Node2 2017-09-14 15:10:06,292 | DEBUG | motor.py ( 34) | set_status | Node1 changing status from started_hungry to started 2017-09-14 15:10:06,292 | DEBUG | node.py ( 941) | checkInstances | Node1 choosing to start election on the basis of count 3 and nodes {'Node2', 'Node3'} 2017-09-14 15:10:06,293 | DEBUG | primary_selector.py ( 77) | get_msgs_for_lagged_nodes | Node1 has no ViewChangeDone message to send for view 0 2017-09-14 15:10:06,293 | DEBUG | node.py ( 902) | send_current_state_to_lagging_node | Node1 sending current state CURRENT_STATE{'primary': [], 'viewNo': 0} to lagged node Node2 2017-09-14 15:10:06,293 | DEBUG | node.py (2608) | send | Node1 sending message CURRENT_STATE{'primary': [], 'viewNo': 0} to 1 recipients: ['Node2'] 2017-09-14 15:10:06,294 | DEBUG | node.py (2608) | send | Node1 sending message LEDGER_STATUS{'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ledgerId': 0, 'txnSeqNo': 3, 'ppSeqNo': None, 'viewNo': None} to 1 recipients: ['Node2'] 2017-09-14 15:10:06,294 | DEBUG | node.py (2608) | send | Node1 sending message LEDGER_STATUS{'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'ledgerId': 1, 'txnSeqNo': 5, 'ppSeqNo': None, 'viewNo': None} to 1 recipients: ['Node2'] 2017-09-14 15:10:06,294 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 2 messages through listener 2017-09-14 15:10:06,295 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":2},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}']} 2017-09-14 15:10:06,295 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'op': 'BATCH', 'messages': ['{"params":{"ledgerId":2},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":2},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}']}, 'Node3') 2017-09-14 15:10:06,295 | DEBUG | node.py (1272) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":2},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}']} 2017-09-14 15:10:06,295 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:10:06,295 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'op': 'MESSAGE_REQUEST'}, 'Node3') 2017-09-14 15:10:06,295 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:10:06,295 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:10:06,296 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'op': 'MESSAGE_REQUEST'}, 'Node3') 2017-09-14 15:10:06,296 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:10:06,296 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']} 2017-09-14 15:10:06,296 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'op': 'BATCH', 'messages': ['{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']}, 'Node2') 2017-09-14 15:10:06,296 | DEBUG | node.py (1272) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']} 2017-09-14 15:10:06,296 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None}} 2017-09-14 15:10:06,297 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE'}, 'Node2') 2017-09-14 15:10:06,297 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None}} 2017-09-14 15:10:06,297 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None}} 2017-09-14 15:10:06,297 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE'}, 'Node2') 2017-09-14 15:10:06,297 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None}} 2017-09-14 15:10:06,297 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None}} to 1 recipients: ['Node3'] 2017-09-14 15:10:06,298 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None}} to 1 recipients: ['Node3'] 2017-09-14 15:10:06,298 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None} from Node2 2017-09-14 15:10:06,298 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:10:06,298 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:10:06,298 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None} from Node2 2017-09-14 15:10:06,299 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:10:06,299 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:10:06,299 | DEBUG | batched.py ( 100) | flushOutBoxes | Node1 batching 3 msgs to Node2 into one transmission 2017-09-14 15:10:06,299 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"viewNo":0,"op":"CURRENT_STATE","primary":[]}', b'{"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","op":"LEDGER_STATUS","viewNo":null,"ledgerId":0,"txnSeqNo":3,"ppSeqNo":null}', b'{"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","op":"LEDGER_STATUS","viewNo":null,"ledgerId":1,"txnSeqNo":5,"ppSeqNo":null}']) 2017-09-14 15:10:06,299 | TRACE | batched.py ( 110) | flushOutBoxes | Node1 sending payload to Node2: b'{"signature":null,"op":"BATCH","messages":["{\\"viewNo\\":0,\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[]}","{\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null,\\"ledgerId\\":0,\\"txnSeqNo\\":3,\\"ppSeqNo\\":null}","{\\"merkleRoot\\":\\"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj\\",\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null,\\"ledgerId\\":1,\\"txnSeqNo\\":5,\\"ppSeqNo\\":null}"]}' 2017-09-14 15:10:06,299 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"signature":null,"op":"BATCH","messages":["{\\"viewNo\\":0,\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[]}","{\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null,\\"ledgerId\\":0,\\"txnSeqNo\\":3,\\"ppSeqNo\\":null}","{\\"merkleRoot\\":\\"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj\\",\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null,\\"ledgerId\\":1,\\"txnSeqNo\\":5,\\"ppSeqNo\\":null}"]}' to Node2 2017-09-14 15:10:06,313 | DEBUG | batched.py ( 100) | flushOutBoxes | Node1 batching 2 msgs to Node3 into one transmission 2017-09-14 15:10:06,313 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}']) 2017-09-14 15:10:06,313 | TRACE | batched.py ( 110) | flushOutBoxes | Node1 sending payload to Node3: b'{"signature":null,"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}"]}' 2017-09-14 15:10:06,313 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"signature":null,"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}"]}' to Node3 2017-09-14 15:10:06,326 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 2 messages through listener 2017-09-14 15:10:06,326 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']} 2017-09-14 15:10:06,326 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'op': 'BATCH', 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']}, 'Node2') 2017-09-14 15:10:06,326 | DEBUG | node.py (1272) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}']} 2017-09-14 15:10:06,327 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:10:06,327 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'op': 'MESSAGE_REQUEST'}, 'Node2') 2017-09-14 15:10:06,327 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:10:06,327 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:10:06,327 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'op': 'MESSAGE_REQUEST'}, 'Node2') 2017-09-14 15:10:06,327 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:10:06,327 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']} 2017-09-14 15:10:06,328 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'op': 'BATCH', 'messages': ['{"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']}, 'Node3') 2017-09-14 15:10:06,328 | DEBUG | node.py (1272) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}']} 2017-09-14 15:10:06,328 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None}} 2017-09-14 15:10:06,328 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE'}, 'Node3') 2017-09-14 15:10:06,328 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None}} 2017-09-14 15:10:06,328 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None}} 2017-09-14 15:10:06,332 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE'}, 'Node3') 2017-09-14 15:10:06,333 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None}} 2017-09-14 15:10:06,333 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None}} to 1 recipients: ['Node2'] 2017-09-14 15:10:06,333 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None}} to 1 recipients: ['Node2'] 2017-09-14 15:10:06,334 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None} from Node3 2017-09-14 15:10:06,334 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:10:06,334 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:10:06,334 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node2', 'Node3'} that its ledger of type 2 is latest 2017-09-14 15:10:06,334 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node1 found from ledger status LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None} that it does not need catchup 2017-09-14 15:10:06,334 | DEBUG | node.py (1479) | preLedgerCatchUp | Node1 going to process any ordered requests before starting catchup. 2017-09-14 15:10:06,334 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node1:0 going to remove 0 Ordered messages from outbox 2017-09-14 15:10:06,334 | DEBUG | node.py (1888) | force_process_ordered | Node1 processed 0 Ordered batches for instance 0 before starting catch up 2017-09-14 15:10:06,335 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node1 processed 0 stashed ordered requests 2017-09-14 15:10:06,335 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-09-14 15:10:06,335 | INFO | node.py (1489) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 2 2017-09-14 15:10:06,335 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:10:06,335 | DEBUG | pool_config.py ( 35) | processLedger | processing config ledger for any POOL_CONFIGs 2017-09-14 15:10:06,335 | DEBUG | upgrader.py ( 208) | processLedger | Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv processing config ledger for any upgrades 2017-09-14 15:10:06,335 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to all recipients: ['Node2', 'Node3'] 2017-09-14 15:10:06,336 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node1 asking Node1 for ledger status of ledger 1 2017-09-14 15:10:06,336 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node2'] 2017-09-14 15:10:06,336 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node1 asking Node2 for ledger status of ledger 1 2017-09-14 15:10:06,336 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-09-14 15:10:06,336 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node1 asking Node3 for ledger status of ledger 1 2017-09-14 15:10:06,344 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node1 going to process 0 stashed ledger statuses for ledger 1 2017-09-14 15:10:06,344 | INFO | upgrader.py ( 150) | should_notify_about_upgrade_result | Node's 'Node1' last upgrade txn is None 2017-09-14 15:10:06,344 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 2, caught up 0 in total 2017-09-14 15:10:06,344 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None} from Node3 2017-09-14 15:10:06,344 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:10:06,345 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:10:06,345 | DEBUG | batched.py ( 100) | flushOutBoxes | Node1 batching 4 msgs to Node2 into one transmission 2017-09-14 15:10:06,345 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}']) 2017-09-14 15:10:06,345 | TRACE | batched.py ( 110) | flushOutBoxes | Node1 sending payload to Node2: b'{"signature":null,"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}"]}' 2017-09-14 15:10:06,345 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"signature":null,"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}"]}' to Node2 2017-09-14 15:10:06,350 | DEBUG | batched.py ( 100) | flushOutBoxes | Node1 batching 2 msgs to Node3 into one transmission 2017-09-14 15:10:06,350 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}']) 2017-09-14 15:10:06,350 | TRACE | batched.py ( 110) | flushOutBoxes | Node1 sending payload to Node3: b'{"signature":null,"op":"BATCH","messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}"]}' 2017-09-14 15:10:06,351 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"signature":null,"op":"BATCH","messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}"]}' to Node3 2017-09-14 15:10:06,351 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 1 messages through listener 2017-09-14 15:10:06,352 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":1},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":1},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}']} 2017-09-14 15:10:06,352 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'op': 'BATCH', 'messages': ['{"params":{"ledgerId":1},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":1},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}']}, 'Node3') 2017-09-14 15:10:06,352 | DEBUG | node.py (1272) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":1},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":1},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}']} 2017-09-14 15:10:06,352 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:10:06,352 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'op': 'MESSAGE_REQUEST'}, 'Node3') 2017-09-14 15:10:06,352 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:10:06,353 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:10:06,353 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'op': 'MESSAGE_REQUEST'}, 'Node3') 2017-09-14 15:10:06,353 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:10:06,353 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': LEDGER_STATUS{'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'ledgerId': 1, 'txnSeqNo': 5, 'ppSeqNo': None, 'viewNo': None}} to 1 recipients: ['Node3'] 2017-09-14 15:10:06,353 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': LEDGER_STATUS{'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'ledgerId': 1, 'txnSeqNo': 5, 'ppSeqNo': None, 'viewNo': None}} to 1 recipients: ['Node3'] 2017-09-14 15:10:06,354 | DEBUG | batched.py ( 100) | flushOutBoxes | Node1 batching 2 msgs to Node3 into one transmission 2017-09-14 15:10:06,354 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', b'{"msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}']) 2017-09-14 15:10:06,354 | TRACE | batched.py ( 110) | flushOutBoxes | Node1 sending payload to Node3: b'{"signature":null,"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":5,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":5,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}"]}' 2017-09-14 15:10:06,354 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"signature":null,"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":5,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":5,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}"]}' to Node3 2017-09-14 15:10:06,378 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 2 messages through listener 2017-09-14 15:10:06,378 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null}}']} 2017-09-14 15:10:06,378 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'op': 'BATCH', 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null}}']}, 'Node2') 2017-09-14 15:10:06,378 | DEBUG | node.py (1272) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null}}']} 2017-09-14 15:10:06,379 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:10:06,379 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'op': 'MESSAGE_REQUEST'}, 'Node2') 2017-09-14 15:10:06,379 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:10:06,379 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:10:06,379 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'op': 'MESSAGE_REQUEST'}, 'Node2') 2017-09-14 15:10:06,379 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:10:06,379 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'ledgerId': 1, 'txnSeqNo': 5, 'viewNo': None, 'ppSeqNo': None}} 2017-09-14 15:10:06,379 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg': {'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'ledgerId': 1, 'txnSeqNo': 5, 'viewNo': None, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE'}, 'Node2') 2017-09-14 15:10:06,380 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'ledgerId': 1, 'txnSeqNo': 5, 'viewNo': None, 'ppSeqNo': None}} 2017-09-14 15:10:06,380 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'ledgerId': 1, 'txnSeqNo': 5, 'viewNo': None, 'ppSeqNo': None}} 2017-09-14 15:10:06,380 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg': {'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'ledgerId': 1, 'txnSeqNo': 5, 'viewNo': None, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE'}, 'Node2') 2017-09-14 15:10:06,380 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'ledgerId': 1, 'txnSeqNo': 5, 'viewNo': None, 'ppSeqNo': None}} 2017-09-14 15:10:06,380 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null}}', '{"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null}}']} 2017-09-14 15:10:06,380 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'op': 'BATCH', 'messages': ['{"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null}}', '{"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null}}']}, 'Node3') 2017-09-14 15:10:06,380 | DEBUG | node.py (1272) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null}}', '{"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null}}']} 2017-09-14 15:10:06,381 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'ledgerId': 1, 'txnSeqNo': 5, 'viewNo': None, 'ppSeqNo': None}} 2017-09-14 15:10:06,381 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg': {'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'ledgerId': 1, 'txnSeqNo': 5, 'viewNo': None, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE'}, 'Node3') 2017-09-14 15:10:06,381 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'ledgerId': 1, 'txnSeqNo': 5, 'viewNo': None, 'ppSeqNo': None}} 2017-09-14 15:10:06,381 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'ledgerId': 1, 'txnSeqNo': 5, 'viewNo': None, 'ppSeqNo': None}} 2017-09-14 15:10:06,381 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg': {'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'ledgerId': 1, 'txnSeqNo': 5, 'viewNo': None, 'ppSeqNo': None}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE'}, 'Node3') 2017-09-14 15:10:06,381 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'ledgerId': 1, 'txnSeqNo': 5, 'viewNo': None, 'ppSeqNo': None}} 2017-09-14 15:10:06,381 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': LEDGER_STATUS{'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'ledgerId': 1, 'txnSeqNo': 5, 'ppSeqNo': None, 'viewNo': None}} to 1 recipients: ['Node2'] 2017-09-14 15:10:06,382 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': LEDGER_STATUS{'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'ledgerId': 1, 'txnSeqNo': 5, 'ppSeqNo': None, 'viewNo': None}} to 1 recipients: ['Node2'] 2017-09-14 15:10:06,382 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'ledgerId': 1, 'txnSeqNo': 5, 'ppSeqNo': None, 'viewNo': None} from Node2 2017-09-14 15:10:06,382 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 5 with 5 2017-09-14 15:10:06,382 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 5 with 5 2017-09-14 15:10:06,383 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'ledgerId': 1, 'txnSeqNo': 5, 'ppSeqNo': None, 'viewNo': None} from Node2 2017-09-14 15:10:06,383 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 5 with 5 2017-09-14 15:10:06,383 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 5 with 5 2017-09-14 15:10:06,383 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'ledgerId': 1, 'txnSeqNo': 5, 'ppSeqNo': None, 'viewNo': None} from Node3 2017-09-14 15:10:06,383 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 5 with 5 2017-09-14 15:10:06,383 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 5 with 5 2017-09-14 15:10:06,384 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node2', 'Node3'} that its ledger of type 1 is latest 2017-09-14 15:10:06,384 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node1 found from ledger status LEDGER_STATUS{'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'ledgerId': 1, 'txnSeqNo': 5, 'ppSeqNo': None, 'viewNo': None} that it does not need catchup 2017-09-14 15:10:06,384 | DEBUG | node.py (1479) | preLedgerCatchUp | Node1 going to process any ordered requests before starting catchup. 2017-09-14 15:10:06,384 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node1:0 going to remove 0 Ordered messages from outbox 2017-09-14 15:10:06,384 | DEBUG | node.py (1888) | force_process_ordered | Node1 processed 0 Ordered batches for instance 0 before starting catch up 2017-09-14 15:10:06,384 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node1 processed 0 stashed ordered requests 2017-09-14 15:10:06,384 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-09-14 15:10:06,384 | INFO | node.py (1489) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 1 2017-09-14 15:10:06,385 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 5 with 5 2017-09-14 15:10:06,385 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 1, caught up 0 in total 2017-09-14 15:10:06,385 | DEBUG | node.py (1609) | num_txns_caught_up_in_last_catchup | Node1 caught up to 0 txns in the last catchup 2017-09-14 15:10:06,385 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node1 processed 0 stashed ordered requests 2017-09-14 15:10:06,385 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-09-14 15:10:06,385 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node1 needs 1 ViewChangeDone messages 2017-09-14 15:10:06,385 | DEBUG | node.py (1568) | caught_up_for_current_view | Node1 does not have view change quorum for view 0 2017-09-14 15:10:06,385 | DEBUG | node.py (1552) | is_catchup_needed | Node1 is not caught up for the current view 0 2017-09-14 15:10:06,385 | DEBUG | node.py (1609) | num_txns_caught_up_in_last_catchup | Node1 caught up to 0 txns in the last catchup 2017-09-14 15:10:06,385 | DEBUG | node.py (1557) | is_catchup_needed | Node1 ordered till last prepared certificate 2017-09-14 15:10:06,386 | INFO | node.py (1537) | allLedgersCaughtUp | CATCH-UP: Node1 does not need any more catchups 2017-09-14 15:10:06,386 | DEBUG | primary_decider.py ( 134) | send | Node1's elector sending VIEW_CHANGE_DONE{'ledgerInfo': [(0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'name': 'Node1', 'viewNo': 0} 2017-09-14 15:10:06,386 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node1 got view change quorum (1 >= 1) 2017-09-14 15:10:06,386 | DEBUG | primary_selector.py ( 215) | has_view_change_from_primary | Node1 received ViewChangeDone from primary Node1 2017-09-14 15:10:06,387 | DEBUG | primary_selector.py ( 238) | has_sufficient_same_view_change_done_messages | Node1 found acceptable primary Node1 and ledger info ((0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')) 2017-09-14 15:10:06,397 | DEBUG | primary_selector.py ( 281) | _startSelection | Node1 starting selection 2017-09-14 15:10:06,397 | DISPLAY | primary_selector.py ( 291) | _startSelection | PRIMARY SELECTION: Node1:0 selected primary Node1:0 for instance 0 (view 0) 2017-09-14 15:10:06,397 | INFO | node.py ( 510) | start_participating | Node1 started participating 2017-09-14 15:10:06,397 | DEBUG | replica.py ( 408) | primaryName | Node1:0 setting primaryName for view no 0 to: Node1:0 2017-09-14 15:10:06,397 | DEBUG | replica.py (1694) | _gc | Node1:0 cleaning up till (0, 0) 2017-09-14 15:10:06,397 | DEBUG | replica.py (1709) | _gc | Node1:0 found 0 3-phase keys to clean 2017-09-14 15:10:06,397 | DEBUG | replica.py (1711) | _gc | Node1:0 found 0 request keys to clean 2017-09-14 15:10:06,397 | DEBUG | replica.py ( 313) | h | Node1:0 set watermarks as 0 300 2017-09-14 15:10:06,398 | DISPLAY | primary_selector.py ( 315) | _startSelection | VIEW CHANGE: Node1:0 declares view change 0 as completed for instance 0, new primary is Node1:0, ledger info is [(0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-09-14 15:10:06,398 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'merkleRoot': 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj', 'ledgerId': 1, 'txnSeqNo': 5, 'ppSeqNo': None, 'viewNo': None} from Node3 2017-09-14 15:10:06,398 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 5 with 5 2017-09-14 15:10:06,398 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 5 with 5 2017-09-14 15:10:06,399 | DEBUG | node.py (2608) | send | Node1 sending message VIEW_CHANGE_DONE{'ledgerInfo': [(0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'name': 'Node1', 'viewNo': 0} to all recipients: ['Node2', 'Node3'] 2017-09-14 15:10:06,399 | DEBUG | batched.py ( 100) | flushOutBoxes | Node1 batching 3 msgs to Node2 into one transmission 2017-09-14 15:10:06,399 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', b'{"msg":{"ledgerId":1,"merkleRoot":"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj","ppSeqNo":null,"txnSeqNo":5,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', b'{"ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,5,"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","op":"VIEW_CHANGE_DONE","viewNo":0}']) 2017-09-14 15:10:06,399 | TRACE | batched.py ( 110) | flushOutBoxes | Node1 sending payload to Node2: b'{"signature":null,"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":5,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":5,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"ledgerInfo\\":[[0,3,\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\"],[1,5,\\"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"name\\":\\"Node1\\",\\"op\\":\\"VIEW_CHANGE_DONE\\",\\"viewNo\\":0}"]}' 2017-09-14 15:10:06,399 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"signature":null,"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":5,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":5,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"ledgerInfo\\":[[0,3,\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\"],[1,5,\\"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"name\\":\\"Node1\\",\\"op\\":\\"VIEW_CHANGE_DONE\\",\\"viewNo\\":0}"]}' to Node2 2017-09-14 15:10:06,410 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,5,"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","op":"VIEW_CHANGE_DONE","viewNo":0}' to Node3 2017-09-14 15:10:06,410 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,5,"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","op":"VIEW_CHANGE_DONE","viewNo":0}' to Node3 2017-09-14 15:10:06,411 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 1 messages through listener 2017-09-14 15:10:06,411 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-09-14 15:10:06,412 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'op': 'VIEW_CHANGE_DONE', 'viewNo': 0}, 'Node3') 2017-09-14 15:10:06,412 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-09-14 15:10:06,412 | DEBUG | node.py (1205) | sendToElector | Node1 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0}, 'Node3') 2017-09-14 15:10:06,412 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node1's primary selector started processing of ViewChangeDone msg from Node3 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-09-14 15:10:06,412 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} because it already decided primary which is Node1:0 2017-09-14 15:10:06,439 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 1 messages through listener 2017-09-14 15:10:06,440 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-09-14 15:10:06,440 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'op': 'VIEW_CHANGE_DONE', 'viewNo': 0}, 'Node2') 2017-09-14 15:10:06,440 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-09-14 15:10:06,441 | DEBUG | node.py (1205) | sendToElector | Node1 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0}, 'Node2') 2017-09-14 15:10:06,441 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node1's primary selector started processing of ViewChangeDone msg from Node2 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-09-14 15:10:06,441 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} because it already decided primary which is Node1:0 2017-09-14 15:10:09,030 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action dump_json_file with id 4 2017-09-14 15:10:15,731 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 1 2017-09-14 15:10:15,732 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:10:15,732 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:10:15,732 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:10:15,732 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:10:15,732 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:10:15,732 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:10:15,732 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 6 to run in 10 seconds 2017-09-14 15:10:21,028 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:10:21,028 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:10:21,028 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:10:25,738 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 6 2017-09-14 15:10:25,738 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:10:25,738 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:10:25,738 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:10:25,738 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:10:25,738 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:10:25,738 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:10:25,739 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 7 to run in 10 seconds 2017-09-14 15:10:35,748 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 7 2017-09-14 15:10:35,749 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:10:35,749 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:10:35,749 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:10:35,749 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:10:35,749 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:10:35,749 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:10:35,749 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 8 to run in 10 seconds 2017-09-14 15:10:36,034 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:10:36,034 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:10:36,035 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:10:45,754 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 8 2017-09-14 15:10:45,754 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:10:45,754 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:10:45,754 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:10:45,755 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:10:45,755 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:10:45,755 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:10:45,755 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 9 to run in 10 seconds 2017-09-14 15:10:51,037 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:10:51,037 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:10:51,038 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:10:55,766 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 9 2017-09-14 15:10:55,766 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:10:55,767 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:10:55,767 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:10:55,767 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:10:55,767 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:10:55,767 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:10:55,767 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 10 to run in 10 seconds 2017-09-14 15:11:05,731 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkNodeRequestSpike with id 2 2017-09-14 15:11:05,731 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node1 checking its request amount 2017-09-14 15:11:05,732 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:11:05,732 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkNodeRequestSpike with id 11 to run in 60 seconds 2017-09-14 15:11:05,732 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 1 2017-09-14 15:11:05,732 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-09-14 15:11:05,732 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 2 to run in 60 seconds 2017-09-14 15:11:05,767 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 10 2017-09-14 15:11:05,767 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:11:05,768 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:11:05,768 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:11:05,768 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:11:05,768 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:11:05,768 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:11:05,768 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 12 to run in 10 seconds 2017-09-14 15:11:06,026 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action dump_json_file with id 5 2017-09-14 15:11:06,027 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action dump_json_file with id 13 to run in 60 seconds 2017-09-14 15:11:06,027 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action propose_view_change with id 3 2017-09-14 15:11:06,027 | DEBUG | throttler.py ( 31) | acquire | now: 13242.878861581, len(actionsLog): 0 2017-09-14 15:11:06,027 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 0 2017-09-14 15:11:06,027 | DEBUG | throttler.py ( 39) | acquire | len(actionsLog) was 0, after append, len(actionsLog): 1 2017-09-14 15:11:06,027 | INFO | node.py (2048) | sendInstanceChange | VIEW CHANGE: Node1 sending an instance change with view_no 1 since Primary of master protocol instance disconnected 2017-09-14 15:11:06,027 | INFO | node.py (2051) | sendInstanceChange | MONITORING: Node1 metrics for monitor: Node1 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [13182.579971101] ordered request counts: {0: 0} ordered request durations: {0: 0} master request latencies: {} client avg request latencies: [{}] throughput: {0: 0} master throughput: None total requests: 0 avg backup throughput: None master throughput ratio: None 2017-09-14 15:11:06,027 | DEBUG | node.py (2608) | send | Node1 sending message INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} to all recipients: ['Node2', 'Node3'] 2017-09-14 15:11:06,028 | DEBUG | node.py (1958) | do_view_change_if_possible | Node1 has no quorum for view 1 2017-09-14 15:11:06,028 | INFO | node.py (2108) | propose_view_change | Node1 sent view change since was disconnected from primary for too long 2017-09-14 15:11:06,028 | DEBUG | node.py (1958) | do_view_change_if_possible | Node1 has no quorum for view 1 2017-09-14 15:11:06,028 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"reason":26,"op":"INSTANCE_CHANGE","viewNo":1}' to Node2 2017-09-14 15:11:06,028 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"reason":26,"op":"INSTANCE_CHANGE","viewNo":1}' to Node2 2017-09-14 15:11:06,028 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"reason":26,"op":"INSTANCE_CHANGE","viewNo":1}' to Node3 2017-09-14 15:11:06,029 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"reason":26,"op":"INSTANCE_CHANGE","viewNo":1}' to Node3 2017-09-14 15:11:06,041 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:11:06,041 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:11:06,041 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:11:15,778 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 12 2017-09-14 15:11:15,778 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:11:15,778 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:11:15,779 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:11:15,779 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:11:15,779 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:11:15,779 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:11:15,779 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 14 to run in 10 seconds 2017-09-14 15:11:21,052 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:11:21,052 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:11:21,053 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:11:25,789 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 14 2017-09-14 15:11:25,789 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:11:25,789 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:11:25,789 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:11:25,790 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:11:25,790 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:11:25,790 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:11:25,790 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 15 to run in 10 seconds 2017-09-14 15:11:35,798 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 15 2017-09-14 15:11:35,798 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:11:35,800 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:11:35,800 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:11:35,800 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:11:35,800 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:11:35,801 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:11:35,801 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 16 to run in 10 seconds 2017-09-14 15:11:36,053 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:11:36,053 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:11:36,054 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:11:45,802 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 16 2017-09-14 15:11:45,802 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:11:45,802 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:11:45,802 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:11:45,802 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:11:45,802 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:11:45,803 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:11:45,803 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 17 to run in 10 seconds 2017-09-14 15:11:51,061 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:11:51,062 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:11:51,062 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:11:55,815 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 17 2017-09-14 15:11:55,815 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:11:55,815 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:11:55,815 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:11:55,815 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:11:55,816 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:11:55,816 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:11:55,816 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 18 to run in 10 seconds 2017-09-14 15:12:05,738 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkNodeRequestSpike with id 11 2017-09-14 15:12:05,738 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node1 checking its request amount 2017-09-14 15:12:05,738 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:12:05,738 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkNodeRequestSpike with id 19 to run in 60 seconds 2017-09-14 15:12:05,739 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 2 2017-09-14 15:12:05,739 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-09-14 15:12:05,739 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 3 to run in 60 seconds 2017-09-14 15:12:05,820 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 18 2017-09-14 15:12:05,820 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:12:05,821 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:12:05,821 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:12:05,821 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:12:05,821 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:12:05,821 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:12:05,821 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 20 to run in 10 seconds 2017-09-14 15:12:06,037 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action dump_json_file with id 13 2017-09-14 15:12:06,039 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action dump_json_file with id 21 to run in 60 seconds 2017-09-14 15:12:06,063 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:12:06,063 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:12:06,063 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:12:15,824 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 20 2017-09-14 15:12:15,825 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:12:15,825 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:12:15,825 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:12:15,825 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:12:15,825 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:12:15,825 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:12:15,825 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 22 to run in 10 seconds 2017-09-14 15:12:21,063 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:12:21,064 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:12:21,064 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:12:25,829 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 22 2017-09-14 15:12:25,830 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:12:25,830 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:12:25,830 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:12:25,830 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:12:25,830 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:12:25,830 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:12:25,830 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 23 to run in 10 seconds 2017-09-14 15:12:35,835 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 23 2017-09-14 15:12:35,835 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:12:35,835 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:12:35,835 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:12:35,835 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:12:35,835 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:12:35,835 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:12:35,836 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 24 to run in 10 seconds 2017-09-14 15:12:36,073 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:12:36,073 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:12:36,073 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:12:41,836 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1C got 1 messages through listener 2017-09-14 15:12:41,836 | DEBUG | zstack.py ( 663) | handlePingPong | Node1C got ping from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:12:41,836 | TRACE | zstack.py ( 754) | transmitThroughListener | Node1C transmitting b'po' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:12:41,837 | DEBUG | zstack.py ( 645) | sendPingPong | Node1C ponged b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:12:41,861 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1C got 1 messages through listener 2017-09-14 15:12:41,862 | TRACE | node.py (1388) | validateClientMsg | Node1C received CLIENT message: LEDGER_STATUS{'merkleRoot': 'CWoRZ62yMB12JyQEUAYU73CNBqJXWaKUafN9yiR9Dre2', 'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None} 2017-09-14 15:12:41,862 | DEBUG | node.py (1434) | processClientInBox | Node1C processing b'v#ZljK34WWooA6%RgESn.Ox' request LEDGER_STATUS{'merkleRoot': 'CWoRZ62yMB12JyQEUAYU73CNBqJXWaKUafN9yiR9Dre2', 'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None} 2017-09-14 15:12:41,862 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'merkleRoot': 'CWoRZ62yMB12JyQEUAYU73CNBqJXWaKUafN9yiR9Dre2', 'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None, 'viewNo': None} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:12:41,862 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 4 2017-09-14 15:12:41,862 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 4 2017-09-14 15:12:41,862 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 4 2017-09-14 15:12:41,863 | TRACE | zstack.py ( 754) | transmitThroughListener | Node1C transmitting b'{"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","op":"LEDGER_STATUS","viewNo":null,"ledgerId":0,"txnSeqNo":3,"ppSeqNo":null}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:12:45,840 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 24 2017-09-14 15:12:45,840 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:12:45,840 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:12:45,840 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:12:45,841 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:12:45,841 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:12:45,841 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:12:45,841 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 25 to run in 10 seconds 2017-09-14 15:12:51,075 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:12:51,075 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:12:51,076 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:12:55,849 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 25 2017-09-14 15:12:55,849 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:12:55,849 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:12:55,849 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:12:55,849 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:12:55,849 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:12:55,850 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:12:55,850 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 26 to run in 10 seconds 2017-09-14 15:13:05,747 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkNodeRequestSpike with id 19 2017-09-14 15:13:05,748 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node1 checking its request amount 2017-09-14 15:13:05,748 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:13:05,748 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkNodeRequestSpike with id 27 to run in 60 seconds 2017-09-14 15:13:05,748 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 3 2017-09-14 15:13:05,748 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-09-14 15:13:05,748 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 4 to run in 60 seconds 2017-09-14 15:13:05,852 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 26 2017-09-14 15:13:05,852 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:13:05,852 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:13:05,853 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:13:05,853 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:13:05,853 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:13:05,853 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:13:05,853 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 28 to run in 10 seconds 2017-09-14 15:13:06,042 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action dump_json_file with id 21 2017-09-14 15:13:06,043 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action dump_json_file with id 29 to run in 60 seconds 2017-09-14 15:13:06,080 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:13:06,080 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:13:06,081 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:13:15,861 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 28 2017-09-14 15:13:15,861 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:13:15,862 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:13:15,862 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:13:15,862 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:13:15,863 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:13:15,864 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:13:15,864 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 30 to run in 10 seconds 2017-09-14 15:13:21,086 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:13:21,086 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:13:21,087 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:13:25,873 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 30 2017-09-14 15:13:25,873 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:13:25,873 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:13:25,873 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:13:25,873 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:13:25,873 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:13:25,874 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:13:25,874 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 31 to run in 10 seconds 2017-09-14 15:13:35,876 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 31 2017-09-14 15:13:35,877 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:13:35,877 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:13:35,877 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:13:35,877 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:13:35,878 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:13:35,878 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:13:35,878 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 32 to run in 10 seconds 2017-09-14 15:13:36,094 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:13:36,094 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:13:36,094 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:13:36,730 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1C got 1 messages through listener 2017-09-14 15:13:36,737 | DEBUG | node.py (2236) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1505402016669500 2017-09-14 15:13:36,737 | TRACE | node.py (1388) | validateClientMsg | Node1C received CLIENT message: SafeRequest: {'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi', 'reqId': 1505402016669500, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} 2017-09-14 15:13:36,737 | DEBUG | node.py (1434) | processClientInBox | Node1C processing b'v#ZljK34WWooA6%RgESn.Ox' request SafeRequest: {'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi', 'reqId': 1505402016669500, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} 2017-09-14 15:13:36,737 | DEBUG | node.py (1708) | processRequest | Node1 received client request: SafeRequest: {'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi', 'reqId': 1505402016669500, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:13:36,737 | TRACE | propagator.py ( 165) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi', 'reqId': 1505402016669500, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} 2017-09-14 15:13:36,737 | INFO | propagator.py ( 148) | propagate | Node1 propagating request ('V4SGRU86Z58d6TV7PBUe6f', 1505402016669500) from client b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:13:36,738 | DEBUG | node.py (2608) | send | Node1 sending message PROPAGATE{'request': {'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi', 'reqId': 1505402016669500, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} to all recipients: ['Node2', 'Node3'] 2017-09-14 15:13:36,738 | DEBUG | propagator.py ( 214) | forward | Node1 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1505402016669500) to 0 replicas 2017-09-14 15:13:36,738 | TRACE | zstack.py ( 754) | transmitThroughListener | Node1C transmitting b'{"reqId":1505402016669500,"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f"}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:13:36,738 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"request":{"signature":"3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi","reqId":1505402016669500,"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe61"},"identifier":"V4SGRU86Z58d6TV7PBUe6f"},"op":"PROPAGATE","senderClient":"v#ZljK34WWooA6%RgESn.Ox"}' to Node2 2017-09-14 15:13:36,739 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"request":{"signature":"3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi","reqId":1505402016669500,"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe61"},"identifier":"V4SGRU86Z58d6TV7PBUe6f"},"op":"PROPAGATE","senderClient":"v#ZljK34WWooA6%RgESn.Ox"}' to Node2 2017-09-14 15:13:36,739 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"request":{"signature":"3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi","reqId":1505402016669500,"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe61"},"identifier":"V4SGRU86Z58d6TV7PBUe6f"},"op":"PROPAGATE","senderClient":"v#ZljK34WWooA6%RgESn.Ox"}' to Node3 2017-09-14 15:13:36,739 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"request":{"signature":"3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi","reqId":1505402016669500,"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe61"},"identifier":"V4SGRU86Z58d6TV7PBUe6f"},"op":"PROPAGATE","senderClient":"v#ZljK34WWooA6%RgESn.Ox"}' to Node3 2017-09-14 15:13:36,740 | DEBUG | replica.py ( 667) | create3PCBatch | Node1:0 creating batch 1 for ledger 1 with state root b'4\xa0"\x9b\xd7\xca\x02\x04/\xd6I@G\x9a\xb1\x81\x0c=m\xed\xcbA"$\x05\x06\xd5.\x8at_\x81' 2017-09-14 15:13:36,744 | DEBUG | replica.py ( 700) | create3PCBatch | Node1:0 created a PRE-PREPARE with 1 requests for ledger 1 2017-09-14 15:13:36,744 | DEBUG | replica.py (2193) | send | Node1:0 sending PrePrepare 2017-09-14 15:13:36,745 | TRACE | replica.py (2194) | send | Node1:0 sending PREPREPARE{'discarded': 1, 'ppTime': 1505402016, 'viewNo': 0, 'txnRootHash': 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE', 'digest': '62b4f2d2503fa0bc8be5767f4b54fed1f273375fa9d8cedbb60ff060ef786196', 'instId': 0, 'stateRootHash': '6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC', 'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402016669500)], 'ppSeqNo': 1} 2017-09-14 15:13:36,745 | DEBUG | replica.py ( 614) | trackBatches | Node1:0 tracking batch for PREPREPARE{'discarded': 1, 'ppTime': 1505402016, 'viewNo': 0, 'txnRootHash': 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE', 'digest': '62b4f2d2503fa0bc8be5767f4b54fed1f273375fa9d8cedbb60ff060ef786196', 'instId': 0, 'stateRootHash': '6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC', 'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402016669500)], 'ppSeqNo': 1} with state root b'4\xa0"\x9b\xd7\xca\x02\x04/\xd6I@G\x9a\xb1\x81\x0c=m\xed\xcbA"$\x05\x06\xd5.\x8at_\x81' 2017-09-14 15:13:36,745 | DEBUG | node.py (2608) | send | Node1 sending message PREPREPARE{'discarded': 1, 'ppTime': 1505402016, 'viewNo': 0, 'txnRootHash': 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE', 'digest': '62b4f2d2503fa0bc8be5767f4b54fed1f273375fa9d8cedbb60ff060ef786196', 'instId': 0, 'stateRootHash': '6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC', 'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402016669500)], 'ppSeqNo': 1} to all recipients: ['Node2', 'Node3'] 2017-09-14 15:13:36,746 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 2 messages through listener 2017-09-14 15:13:36,747 | DEBUG | node.py (2236) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1505402016669500 2017-09-14 15:13:36,748 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: PROPAGATE{'request': {'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi', 'reqId': 1505402016669500, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:13:36,748 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'request': {'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi', 'reqId': 1505402016669500, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}, 'op': 'PROPAGATE', 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'}, 'Node3') 2017-09-14 15:13:36,748 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'request': {'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi', 'reqId': 1505402016669500, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:13:36,749 | DEBUG | node.py (2236) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1505402016669500 2017-09-14 15:13:36,749 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: PROPAGATE{'request': {'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi', 'reqId': 1505402016669500, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:13:36,749 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'request': {'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi', 'reqId': 1505402016669500, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}, 'op': 'PROPAGATE', 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'}, 'Node2') 2017-09-14 15:13:36,749 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'request': {'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi', 'reqId': 1505402016669500, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:13:36,749 | DEBUG | node.py (1757) | processPropagate | Node Node1 received propagated request: PROPAGATE{'request': {'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi', 'reqId': 1505402016669500, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:13:36,750 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi', 'reqId': 1505402016669500, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} 2017-09-14 15:13:36,750 | DEBUG | propagator.py ( 246) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi', 'reqId': 1505402016669500, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} to its replicas since already forwarded 2017-09-14 15:13:36,750 | DEBUG | node.py (1757) | processPropagate | Node Node1 received propagated request: PROPAGATE{'request': {'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi', 'reqId': 1505402016669500, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:13:36,750 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi', 'reqId': 1505402016669500, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} 2017-09-14 15:13:36,750 | DEBUG | propagator.py ( 246) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': '3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi', 'reqId': 1505402016669500, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe61'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} to its replicas since already forwarded 2017-09-14 15:13:36,751 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"discarded":1,"ppTime":1505402016,"digest":"62b4f2d2503fa0bc8be5767f4b54fed1f273375fa9d8cedbb60ff060ef786196","instId":0,"op":"PREPREPARE","viewNo":0,"txnRootHash":"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE","stateRootHash":"6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC","ledgerId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1505402016669500]],"ppSeqNo":1}' to Node2 2017-09-14 15:13:36,751 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"discarded":1,"ppTime":1505402016,"digest":"62b4f2d2503fa0bc8be5767f4b54fed1f273375fa9d8cedbb60ff060ef786196","instId":0,"op":"PREPREPARE","viewNo":0,"txnRootHash":"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE","stateRootHash":"6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC","ledgerId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1505402016669500]],"ppSeqNo":1}' to Node2 2017-09-14 15:13:36,751 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"discarded":1,"ppTime":1505402016,"digest":"62b4f2d2503fa0bc8be5767f4b54fed1f273375fa9d8cedbb60ff060ef786196","instId":0,"op":"PREPREPARE","viewNo":0,"txnRootHash":"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE","stateRootHash":"6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC","ledgerId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1505402016669500]],"ppSeqNo":1}' to Node3 2017-09-14 15:13:36,751 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"discarded":1,"ppTime":1505402016,"digest":"62b4f2d2503fa0bc8be5767f4b54fed1f273375fa9d8cedbb60ff060ef786196","instId":0,"op":"PREPREPARE","viewNo":0,"txnRootHash":"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE","stateRootHash":"6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC","ledgerId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1505402016669500]],"ppSeqNo":1}' to Node3 2017-09-14 15:13:36,764 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 1 messages through listener 2017-09-14 15:13:36,764 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: PREPARE{'ppTime': 1505402016, 'viewNo': 0, 'txnRootHash': 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE', 'digest': '62b4f2d2503fa0bc8be5767f4b54fed1f273375fa9d8cedbb60ff060ef786196', 'instId': 0, 'stateRootHash': '6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC', 'ppSeqNo': 1} 2017-09-14 15:13:36,765 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'op': 'PREPARE', 'ppTime': 1505402016, 'instId': 0, 'txnRootHash': 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE', 'stateRootHash': '6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC', 'viewNo': 0, 'digest': '62b4f2d2503fa0bc8be5767f4b54fed1f273375fa9d8cedbb60ff060ef786196', 'ppSeqNo': 1}, 'Node2') 2017-09-14 15:13:36,765 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'ppTime': 1505402016, 'viewNo': 0, 'txnRootHash': 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE', 'digest': '62b4f2d2503fa0bc8be5767f4b54fed1f273375fa9d8cedbb60ff060ef786196', 'instId': 0, 'stateRootHash': '6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC', 'ppSeqNo': 1} 2017-09-14 15:13:36,765 | DEBUG | replica.py ( 917) | tryCommit | Node1:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'ppTime': 1505402016, 'viewNo': 0, 'txnRootHash': 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE', 'digest': '62b4f2d2503fa0bc8be5767f4b54fed1f273375fa9d8cedbb60ff060ef786196', 'instId': 0, 'stateRootHash': '6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC', 'ppSeqNo': 1} 2017-09-14 15:13:36,766 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (0, 1) 2017-09-14 15:13:36,782 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 3 messages through listener 2017-09-14 15:13:36,783 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: PREPARE{'ppTime': 1505402016, 'viewNo': 0, 'txnRootHash': 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE', 'digest': '62b4f2d2503fa0bc8be5767f4b54fed1f273375fa9d8cedbb60ff060ef786196', 'instId': 0, 'stateRootHash': '6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC', 'ppSeqNo': 1} 2017-09-14 15:13:36,783 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'ppTime': 1505402016, 'op': 'PREPARE', 'viewNo': 0, 'txnRootHash': 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE', 'digest': '62b4f2d2503fa0bc8be5767f4b54fed1f273375fa9d8cedbb60ff060ef786196', 'instId': 0, 'stateRootHash': '6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC', 'ppSeqNo': 1}, 'Node3') 2017-09-14 15:13:36,783 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'ppTime': 1505402016, 'viewNo': 0, 'txnRootHash': 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE', 'digest': '62b4f2d2503fa0bc8be5767f4b54fed1f273375fa9d8cedbb60ff060ef786196', 'instId': 0, 'stateRootHash': '6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC', 'ppSeqNo': 1} 2017-09-14 15:13:36,783 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: COMMIT{'viewNo': 0, 'ppSeqNo': 1, 'instId': 0} 2017-09-14 15:13:36,783 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'viewNo': 0, 'ppSeqNo': 1, 'op': 'COMMIT', 'instId': 0}, 'Node2') 2017-09-14 15:13:36,783 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'viewNo': 0, 'ppSeqNo': 1, 'instId': 0} 2017-09-14 15:13:36,784 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: COMMIT{'viewNo': 0, 'ppSeqNo': 1, 'instId': 0} 2017-09-14 15:13:36,784 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'instId': 0, 'ppSeqNo': 1, 'op': 'COMMIT', 'viewNo': 0}, 'Node3') 2017-09-14 15:13:36,784 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'viewNo': 0, 'ppSeqNo': 1, 'instId': 0} 2017-09-14 15:13:36,785 | DEBUG | replica.py ( 953) | doCommit | Node1:0 Sending COMMIT(0, 1) at 13393.636625341 2017-09-14 15:13:36,785 | DEBUG | replica.py (2193) | send | Node1:0 sending Commit 2017-09-14 15:13:36,785 | TRACE | replica.py (2194) | send | Node1:0 sending COMMIT{'viewNo': 0, 'ppSeqNo': 1, 'instId': 0} 2017-09-14 15:13:36,785 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'viewNo': 0, 'ppSeqNo': 1, 'instId': 0} commits where f is 0 2017-09-14 15:13:36,785 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (0, 1) 2017-09-14 15:13:36,785 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(0, 1) from Node2:0 2017-09-14 15:13:36,785 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'viewNo': 0, 'ppSeqNo': 1, 'instId': 0} commits where f is 0 2017-09-14 15:13:36,785 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(0, 1) 2017-09-14 15:13:36,786 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(0, 1) from Node3:0 2017-09-14 15:13:36,786 | TRACE | replica.py ( 925) | tryOrder | Node1:0 returning request to node 2017-09-14 15:13:36,786 | DEBUG | replica.py (1444) | doOrder | Node1:0 ordering COMMIT (0, 1) 2017-09-14 15:13:36,786 | DEBUG | replica.py ( 323) | last_ordered_3pc | Node1:0 set last ordered as (0, 1) 2017-09-14 15:13:36,786 | DEBUG | replica.py (2193) | send | Node1:0 sending Ordered 2017-09-14 15:13:36,786 | TRACE | replica.py (2194) | send | Node1:0 sending ORDERED{'ppTime': 1505402016, 'viewNo': 0, 'txnRootHash': 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE', 'ledgerId': 1, 'instId': 0, 'stateRootHash': '6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC', 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402016669500)], 'ppSeqNo': 1} 2017-09-14 15:13:36,786 | INFO | replica.py (1483) | order_3pc_key | Node1:0 ordered batch request, view no 0, ppSeqNo 1, ledger 1, state root 6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC, txn root Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1505402016669500)], discarded [] 2017-09-14 15:13:36,786 | DEBUG | replica.py (1556) | _newCheckpointState | Node1:0 adding new checkpoint state for (1, 100) 2017-09-14 15:13:36,787 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(0, 1) 2017-09-14 15:13:36,787 | DEBUG | node.py (2608) | send | Node1 sending message COMMIT{'viewNo': 0, 'ppSeqNo': 1, 'instId': 0} to all recipients: ['Node2', 'Node3'] 2017-09-14 15:13:36,787 | DEBUG | node.py (1847) | processOrdered | Node1 executing Ordered batch 0 1 of 1 requests 2017-09-14 15:13:36,787 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-09-14 15:13:36,788 | DEBUG | node.py (2386) | sendReplyToClient | Node1 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1505402016669500) to client 2017-09-14 15:13:36,788 | TRACE | zstack.py ( 754) | transmitThroughListener | Node1C transmitting b'{"op":"REPLY","result":{"signature":"3L68fDCfEQosmSQ7dirxcALDwbDC2UDB9UPUxCZNqQ1sFXF3eThsqErNG5AipHsKYaJqBrseZL7GHMcZh8PX8fhi","txnTime":1505402016,"seqNo":6,"reqId":1505402016669500,"identifier":"V4SGRU86Z58d6TV7PBUe6f","type":"1","rootHash":"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE","auditPath":["BSt4P6vp8UXJgSxC9MPouLUPB4VfTSyweJRx1pusrHzc","FA73SaKHCBxBXhf9gis5FxVgz3KxD5GJ1n12yyVYHhDj"],"dest":"V4SGRU86Z58d6TV7PBUe61"}}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:13:36,788 | INFO | node.py (2290) | executeBatch | Node1 committed batch request, view no 0, ppSeqNo 1, ledger 1, state root 6EzdTN9DsCzLdZphhX7yNyL2kBBd81DgsuiGQFPKJmTC, txn root Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE, requests: [('V4SGRU86Z58d6TV7PBUe6f', 1505402016669500)] 2017-09-14 15:13:36,788 | DEBUG | node.py (2305) | executeBatch | Node1 storing 3PC key (0, 1) for ledger 1 range (6, 6) 2017-09-14 15:13:36,789 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:13:36,789 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"instId":0,"ppSeqNo":1,"op":"COMMIT","viewNo":0}' to Node2 2017-09-14 15:13:36,789 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"instId":0,"ppSeqNo":1,"op":"COMMIT","viewNo":0}' to Node2 2017-09-14 15:13:36,790 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"instId":0,"ppSeqNo":1,"op":"COMMIT","viewNo":0}' to Node3 2017-09-14 15:13:36,790 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"instId":0,"ppSeqNo":1,"op":"COMMIT","viewNo":0}' to Node3 2017-09-14 15:13:45,885 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 32 2017-09-14 15:13:45,885 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:13:45,885 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:13:45,885 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:13:45,885 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:13:45,886 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:13:45,886 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:13:45,886 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 33 to run in 10 seconds 2017-09-14 15:13:51,103 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:13:51,105 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:13:51,106 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:13:55,888 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 33 2017-09-14 15:13:55,889 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:13:55,889 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:13:55,889 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:13:55,889 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:13:55,889 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:13:55,889 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:13:55,889 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 34 to run in 10 seconds 2017-09-14 15:14:05,757 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkNodeRequestSpike with id 27 2017-09-14 15:14:05,758 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node1 checking its request amount 2017-09-14 15:14:05,758 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:14:05,758 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkNodeRequestSpike with id 35 to run in 60 seconds 2017-09-14 15:14:05,758 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 4 2017-09-14 15:14:05,758 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-09-14 15:14:05,758 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 5 to run in 60 seconds 2017-09-14 15:14:05,895 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 34 2017-09-14 15:14:05,895 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:14:05,895 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:14:05,895 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:14:05,895 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:14:05,895 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:14:05,896 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:14:05,896 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 36 to run in 10 seconds 2017-09-14 15:14:06,043 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action dump_json_file with id 29 2017-09-14 15:14:06,044 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action dump_json_file with id 37 to run in 60 seconds 2017-09-14 15:14:06,104 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:14:06,104 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:14:06,106 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:14:06,256 | TRACE | remote.py ( 119) | hasLostConnection | Remote Node3:HA(host='10.0.0.4', port=9705) has monitor events: [512, 4] 2017-09-14 15:14:06,256 | DEBUG | remote.py ( 123) | hasLostConnection | Node3:HA(host='10.0.0.4', port=9705) found disconnected event on monitor 2017-09-14 15:14:06,256 | DEBUG | keep_in_touch.py ( 68) | conns | Node1's connections changed from {'Node2', 'Node3'} to {'Node2'} 2017-09-14 15:14:06,257 | INFO | keep_in_touch.py ( 93) | _connsChanged | CONNECTION: Node1 disconnected from Node3 2017-09-14 15:14:06,257 | DEBUG | motor.py ( 34) | set_status | Node1 changing status from started to started_hungry 2017-09-14 15:14:06,257 | DEBUG | node.py ( 941) | checkInstances | Node1 choosing to start election on the basis of count 2 and nodes {'Node2'} 2017-09-14 15:14:12,971 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1C got 1 messages through listener 2017-09-14 15:14:12,972 | DEBUG | node.py (2236) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1505402052949993 2017-09-14 15:14:12,972 | TRACE | node.py (1388) | validateClientMsg | Node1C received CLIENT message: SafeRequest: {'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk', 'reqId': 1505402052949993, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} 2017-09-14 15:14:12,972 | DEBUG | node.py (1434) | processClientInBox | Node1C processing b'v#ZljK34WWooA6%RgESn.Ox' request SafeRequest: {'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk', 'reqId': 1505402052949993, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} 2017-09-14 15:14:12,972 | DEBUG | node.py (1708) | processRequest | Node1 received client request: SafeRequest: {'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk', 'reqId': 1505402052949993, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:14:12,972 | TRACE | propagator.py ( 165) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk', 'reqId': 1505402052949993, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} 2017-09-14 15:14:12,972 | INFO | propagator.py ( 148) | propagate | Node1 propagating request ('V4SGRU86Z58d6TV7PBUe6f', 1505402052949993) from client b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:14:12,973 | DEBUG | node.py (2608) | send | Node1 sending message PROPAGATE{'request': {'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk', 'reqId': 1505402052949993, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} to all recipients: ['Node2', 'Node3'] 2017-09-14 15:14:12,973 | DEBUG | propagator.py ( 214) | forward | Node1 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1505402052949993) to 0 replicas 2017-09-14 15:14:12,973 | TRACE | zstack.py ( 754) | transmitThroughListener | Node1C transmitting b'{"reqId":1505402052949993,"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f"}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:14:12,973 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"request":{"signature":"48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk","reqId":1505402052949993,"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe62"},"identifier":"V4SGRU86Z58d6TV7PBUe6f"},"op":"PROPAGATE","senderClient":"v#ZljK34WWooA6%RgESn.Ox"}' to Node2 2017-09-14 15:14:12,974 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"request":{"signature":"48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk","reqId":1505402052949993,"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe62"},"identifier":"V4SGRU86Z58d6TV7PBUe6f"},"op":"PROPAGATE","senderClient":"v#ZljK34WWooA6%RgESn.Ox"}' to Node2 2017-09-14 15:14:12,974 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"request":{"signature":"48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk","reqId":1505402052949993,"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe62"},"identifier":"V4SGRU86Z58d6TV7PBUe6f"},"op":"PROPAGATE","senderClient":"v#ZljK34WWooA6%RgESn.Ox"}' to Node3 2017-09-14 15:14:12,974 | DEBUG | zstack.py ( 728) | transmit | Remote Node3 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-09-14 15:14:12,974 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"request":{"signature":"48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk","reqId":1505402052949993,"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe62"},"identifier":"V4SGRU86Z58d6TV7PBUe6f"},"op":"PROPAGATE","senderClient":"v#ZljK34WWooA6%RgESn.Ox"}' to Node3 2017-09-14 15:14:12,974 | DEBUG | replica.py ( 667) | create3PCBatch | Node1:0 creating batch 2 for ledger 1 with state root b'M\xe0%[\xda\xe2\xc9\x99\x12N\xc5\xa9\x16\xe1G\xd6\x13\x88&\xb7\xb8\x92\xd7N\xf33\xc3\x98\x12+\x9d\xa7' 2017-09-14 15:14:12,976 | DEBUG | replica.py ( 700) | create3PCBatch | Node1:0 created a PRE-PREPARE with 1 requests for ledger 1 2017-09-14 15:14:12,976 | DEBUG | replica.py (2193) | send | Node1:0 sending PrePrepare 2017-09-14 15:14:12,976 | TRACE | replica.py (2194) | send | Node1:0 sending PREPREPARE{'discarded': 1, 'ppTime': 1505402052, 'viewNo': 0, 'txnRootHash': 'C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby', 'digest': '13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755', 'instId': 0, 'stateRootHash': 'BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH', 'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402052949993)], 'ppSeqNo': 2} 2017-09-14 15:14:12,976 | DEBUG | replica.py ( 614) | trackBatches | Node1:0 tracking batch for PREPREPARE{'discarded': 1, 'ppTime': 1505402052, 'viewNo': 0, 'txnRootHash': 'C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby', 'digest': '13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755', 'instId': 0, 'stateRootHash': 'BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH', 'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402052949993)], 'ppSeqNo': 2} with state root b'M\xe0%[\xda\xe2\xc9\x99\x12N\xc5\xa9\x16\xe1G\xd6\x13\x88&\xb7\xb8\x92\xd7N\xf33\xc3\x98\x12+\x9d\xa7' 2017-09-14 15:14:12,976 | DEBUG | node.py (2608) | send | Node1 sending message PREPREPARE{'discarded': 1, 'ppTime': 1505402052, 'viewNo': 0, 'txnRootHash': 'C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby', 'digest': '13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755', 'instId': 0, 'stateRootHash': 'BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH', 'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402052949993)], 'ppSeqNo': 2} to all recipients: ['Node2', 'Node3'] 2017-09-14 15:14:12,977 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"discarded":1,"ppTime":1505402052,"digest":"13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755","instId":0,"op":"PREPREPARE","viewNo":0,"txnRootHash":"C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby","stateRootHash":"BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH","ledgerId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1505402052949993]],"ppSeqNo":2}' to Node2 2017-09-14 15:14:12,977 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"discarded":1,"ppTime":1505402052,"digest":"13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755","instId":0,"op":"PREPREPARE","viewNo":0,"txnRootHash":"C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby","stateRootHash":"BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH","ledgerId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1505402052949993]],"ppSeqNo":2}' to Node2 2017-09-14 15:14:12,977 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"discarded":1,"ppTime":1505402052,"digest":"13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755","instId":0,"op":"PREPREPARE","viewNo":0,"txnRootHash":"C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby","stateRootHash":"BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH","ledgerId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1505402052949993]],"ppSeqNo":2}' to Node3 2017-09-14 15:14:12,977 | DEBUG | zstack.py ( 728) | transmit | Remote Node3 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-09-14 15:14:12,978 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"discarded":1,"ppTime":1505402052,"digest":"13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755","instId":0,"op":"PREPREPARE","viewNo":0,"txnRootHash":"C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby","stateRootHash":"BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH","ledgerId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1505402052949993]],"ppSeqNo":2}' to Node3 2017-09-14 15:14:12,989 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 2 messages through listener 2017-09-14 15:14:12,990 | DEBUG | node.py (2236) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1505402052949993 2017-09-14 15:14:12,990 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: PROPAGATE{'request': {'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk', 'reqId': 1505402052949993, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:14:12,990 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'request': {'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk', 'reqId': 1505402052949993, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}, 'op': 'PROPAGATE', 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'}, 'Node2') 2017-09-14 15:14:12,991 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'request': {'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk', 'reqId': 1505402052949993, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:14:12,991 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: PREPARE{'ppTime': 1505402052, 'viewNo': 0, 'txnRootHash': 'C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby', 'digest': '13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755', 'instId': 0, 'stateRootHash': 'BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH', 'ppSeqNo': 2} 2017-09-14 15:14:12,991 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'op': 'PREPARE', 'ppTime': 1505402052, 'instId': 0, 'txnRootHash': 'C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby', 'stateRootHash': 'BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH', 'viewNo': 0, 'digest': '13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755', 'ppSeqNo': 2}, 'Node2') 2017-09-14 15:14:12,991 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'ppTime': 1505402052, 'viewNo': 0, 'txnRootHash': 'C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby', 'digest': '13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755', 'instId': 0, 'stateRootHash': 'BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH', 'ppSeqNo': 2} 2017-09-14 15:14:12,991 | DEBUG | node.py (1757) | processPropagate | Node Node1 received propagated request: PROPAGATE{'request': {'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk', 'reqId': 1505402052949993, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:14:12,991 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk', 'reqId': 1505402052949993, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} 2017-09-14 15:14:12,991 | DEBUG | propagator.py ( 246) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk', 'reqId': 1505402052949993, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} to its replicas since already forwarded 2017-09-14 15:14:12,992 | DEBUG | replica.py ( 917) | tryCommit | Node1:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'ppTime': 1505402052, 'viewNo': 0, 'txnRootHash': 'C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby', 'digest': '13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755', 'instId': 0, 'stateRootHash': 'BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH', 'ppSeqNo': 2} 2017-09-14 15:14:12,992 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (0, 2) 2017-09-14 15:14:15,903 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 36 2017-09-14 15:14:15,903 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:14:15,903 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:14:15,903 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:14:15,903 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:14:15,904 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:14:15,904 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:14:15,904 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 38 to run in 10 seconds 2017-09-14 15:14:21,107 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:14:21,107 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:14:21,108 | DEBUG | zstack.py ( 645) | sendPingPong | Node1 pinged Node3 2017-09-14 15:14:21,108 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:14:21,119 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'pi' to Node3 2017-09-14 15:14:21,120 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'pi' to Node3 2017-09-14 15:14:25,904 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 38 2017-09-14 15:14:25,905 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:14:25,905 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:14:25,905 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:14:25,905 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:14:25,905 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:14:25,905 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:14:25,905 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 39 to run in 10 seconds 2017-09-14 15:14:27,978 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1C got 1 messages through listener 2017-09-14 15:14:27,982 | DEBUG | node.py (2236) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1505402052949993 2017-09-14 15:14:27,982 | TRACE | node.py (1388) | validateClientMsg | Node1C received CLIENT message: SafeRequest: {'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk', 'reqId': 1505402052949993, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} 2017-09-14 15:14:27,983 | DEBUG | node.py (1434) | processClientInBox | Node1C processing b'v#ZljK34WWooA6%RgESn.Ox' request SafeRequest: {'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk', 'reqId': 1505402052949993, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} 2017-09-14 15:14:27,983 | DEBUG | node.py (1708) | processRequest | Node1 received client request: SafeRequest: {'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk', 'reqId': 1505402052949993, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:14:27,983 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk', 'reqId': 1505402052949993, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} 2017-09-14 15:14:27,983 | DEBUG | propagator.py ( 246) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk', 'reqId': 1505402052949993, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} to its replicas since already forwarded 2017-09-14 15:14:27,983 | TRACE | zstack.py ( 754) | transmitThroughListener | Node1C transmitting b'{"reqId":1505402052949993,"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f"}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:14:31,476 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 1 messages through listener 2017-09-14 15:14:31,476 | DEBUG | zstack.py ( 663) | handlePingPong | Node1 got ping from Node3 2017-09-14 15:14:31,477 | DEBUG | zstack.py ( 645) | sendPingPong | Node1 ponged Node3 2017-09-14 15:14:31,477 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'po' to Node3 2017-09-14 15:14:31,477 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'po' to Node3 2017-09-14 15:14:31,577 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 2 messages through listener 2017-09-14 15:14:31,577 | DEBUG | zstack.py ( 669) | handlePingPong | Node1 got pong from Node3 2017-09-14 15:14:31,578 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: BATCH{'signature': None, 'messages': ['{"request":{"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk","reqId":1505402052949993,"operation":{"dest":"V4SGRU86Z58d6TV7PBUe62","type":"1"}},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}']} 2017-09-14 15:14:31,578 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'op': 'BATCH', 'messages': ['{"request":{"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk","reqId":1505402052949993,"operation":{"dest":"V4SGRU86Z58d6TV7PBUe62","type":"1"}},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}']}, 'Node3') 2017-09-14 15:14:31,578 | DEBUG | node.py (1272) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"request":{"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk","reqId":1505402052949993,"operation":{"dest":"V4SGRU86Z58d6TV7PBUe62","type":"1"}},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}']} 2017-09-14 15:14:31,579 | DEBUG | node.py (2236) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1505402052949993 2017-09-14 15:14:31,579 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: PROPAGATE{'request': {'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk', 'reqId': 1505402052949993, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:14:31,579 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'request': {'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk', 'reqId': 1505402052949993, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}, 'op': 'PROPAGATE', 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'}, 'Node3') 2017-09-14 15:14:31,579 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'request': {'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk', 'reqId': 1505402052949993, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:14:31,579 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: BATCH{'signature': None, 'messages': ['{"op":"CURRENT_STATE","viewNo":0,"primary":[]}', '{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","txnSeqNo":3,"viewNo":null,"op":"LEDGER_STATUS","ppSeqNo":null}']} 2017-09-14 15:14:31,579 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'op': 'BATCH', 'messages': ['{"op":"CURRENT_STATE","viewNo":0,"primary":[]}', '{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","txnSeqNo":3,"viewNo":null,"op":"LEDGER_STATUS","ppSeqNo":null}']}, 'Node3') 2017-09-14 15:14:31,579 | DEBUG | node.py (1272) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"op":"CURRENT_STATE","viewNo":0,"primary":[]}', '{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","txnSeqNo":3,"viewNo":null,"op":"LEDGER_STATUS","ppSeqNo":null}']} 2017-09-14 15:14:31,579 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: CURRENT_STATE{'primary': [], 'viewNo': 0} 2017-09-14 15:14:31,579 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'primary': [], 'op': 'CURRENT_STATE', 'viewNo': 0}, 'Node3') 2017-09-14 15:14:31,580 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox CURRENT_STATE{'primary': [], 'viewNo': 0} 2017-09-14 15:14:31,580 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: LEDGER_STATUS{'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ledgerId': 0, 'txnSeqNo': 3, 'ppSeqNo': None, 'viewNo': None} 2017-09-14 15:14:31,580 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'op': 'LEDGER_STATUS', 'viewNo': None, 'ledgerId': 0, 'txnSeqNo': 3, 'ppSeqNo': None}, 'Node3') 2017-09-14 15:14:31,580 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox LEDGER_STATUS{'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ledgerId': 0, 'txnSeqNo': 3, 'ppSeqNo': None, 'viewNo': None} 2017-09-14 15:14:31,580 | DEBUG | node.py (1757) | processPropagate | Node Node1 received propagated request: PROPAGATE{'request': {'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk', 'reqId': 1505402052949993, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:14:31,580 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk', 'reqId': 1505402052949993, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} 2017-09-14 15:14:31,580 | DEBUG | propagator.py ( 246) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': '48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk', 'reqId': 1505402052949993, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe62'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} to its replicas since already forwarded 2017-09-14 15:14:31,581 | DEBUG | node.py ( 907) | process_current_state_message | Node1 processing current state CURRENT_STATE{'primary': [], 'viewNo': 0} from Node3 2017-09-14 15:14:31,581 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ledgerId': 0, 'txnSeqNo': 3, 'ppSeqNo': None, 'viewNo': None} from Node3 2017-09-14 15:14:31,581 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 3 2017-09-14 15:14:31,581 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 3 2017-09-14 15:14:31,584 | TRACE | remote.py ( 119) | hasLostConnection | Remote Node3:HA(host='10.0.0.4', port=9705) has monitor events: [2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 1] 2017-09-14 15:14:31,584 | DEBUG | remote.py ( 123) | hasLostConnection | Node3:HA(host='10.0.0.4', port=9705) found disconnected event on monitor 2017-09-14 15:14:31,584 | DEBUG | keep_in_touch.py ( 68) | conns | Node1's connections changed from {'Node2'} to {'Node2', 'Node3'} 2017-09-14 15:14:31,584 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node1 now connected to Node3 2017-09-14 15:14:31,584 | DEBUG | motor.py ( 34) | set_status | Node1 changing status from started_hungry to started 2017-09-14 15:14:31,586 | DEBUG | node.py ( 941) | checkInstances | Node1 choosing to start election on the basis of count 3 and nodes {'Node2', 'Node3'} 2017-09-14 15:14:31,586 | DEBUG | node.py ( 902) | send_current_state_to_lagging_node | Node1 sending current state CURRENT_STATE{'primary': [VIEW_CHANGE_DONE{'ledgerInfo': ((0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')), 'name': 'Node1', 'viewNo': 0}], 'viewNo': 0} to lagged node Node3 2017-09-14 15:14:31,586 | DEBUG | node.py (2608) | send | Node1 sending message CURRENT_STATE{'primary': [VIEW_CHANGE_DONE{'ledgerInfo': ((0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')), 'name': 'Node1', 'viewNo': 0}], 'viewNo': 0} to 1 recipients: ['Node3'] 2017-09-14 15:14:31,586 | DEBUG | node.py (2608) | send | Node1 sending message LEDGER_STATUS{'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ledgerId': 0, 'txnSeqNo': 3, 'ppSeqNo': None, 'viewNo': None} to 1 recipients: ['Node3'] 2017-09-14 15:14:31,587 | DEBUG | node.py (2608) | send | Node1 sending message LEDGER_STATUS{'merkleRoot': 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE', 'ledgerId': 1, 'txnSeqNo': 6, 'ppSeqNo': 1, 'viewNo': 0} to 1 recipients: ['Node3'] 2017-09-14 15:14:31,587 | DEBUG | node.py (2608) | send | Node1 sending message LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None} to 1 recipients: ['Node3'] 2017-09-14 15:14:31,589 | DEBUG | batched.py ( 100) | flushOutBoxes | Node1 batching 4 msgs to Node3 into one transmission 2017-09-14 15:14:31,589 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"viewNo":0,"op":"CURRENT_STATE","primary":[{"ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,5,"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0}]}', b'{"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","op":"LEDGER_STATUS","viewNo":null,"ledgerId":0,"txnSeqNo":3,"ppSeqNo":null}', b'{"merkleRoot":"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE","op":"LEDGER_STATUS","viewNo":0,"ledgerId":1,"txnSeqNo":6,"ppSeqNo":1}', b'{"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","op":"LEDGER_STATUS","viewNo":null,"ledgerId":2,"txnSeqNo":0,"ppSeqNo":null}']) 2017-09-14 15:14:31,589 | TRACE | batched.py ( 110) | flushOutBoxes | Node1 sending payload to Node3: b'{"signature":null,"op":"BATCH","messages":["{\\"viewNo\\":0,\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[{\\"ledgerInfo\\":[[0,3,\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\"],[1,5,\\"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"name\\":\\"Node1\\",\\"viewNo\\":0}]}","{\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null,\\"ledgerId\\":0,\\"txnSeqNo\\":3,\\"ppSeqNo\\":null}","{\\"merkleRoot\\":\\"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE\\",\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":0,\\"ledgerId\\":1,\\"txnSeqNo\\":6,\\"ppSeqNo\\":1}","{\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null,\\"ledgerId\\":2,\\"txnSeqNo\\":0,\\"ppSeqNo\\":null}"]}' 2017-09-14 15:14:31,589 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"signature":null,"op":"BATCH","messages":["{\\"viewNo\\":0,\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[{\\"ledgerInfo\\":[[0,3,\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\"],[1,5,\\"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"name\\":\\"Node1\\",\\"viewNo\\":0}]}","{\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null,\\"ledgerId\\":0,\\"txnSeqNo\\":3,\\"ppSeqNo\\":null}","{\\"merkleRoot\\":\\"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE\\",\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":0,\\"ledgerId\\":1,\\"txnSeqNo\\":6,\\"ppSeqNo\\":1}","{\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null,\\"ledgerId\\":2,\\"txnSeqNo\\":0,\\"ppSeqNo\\":null}"]}' to Node3 2017-09-14 15:14:31,683 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 2 messages through listener 2017-09-14 15:14:31,684 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"name":"Node1","op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,6,"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":0}']} 2017-09-14 15:14:31,684 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'op': 'BATCH', 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"name":"Node1","op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,6,"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":0}']}, 'Node3') 2017-09-14 15:14:31,684 | DEBUG | node.py (1272) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"name":"Node1","op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,6,"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":0}']} 2017-09-14 15:14:31,684 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:14:31,684 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'op': 'MESSAGE_REQUEST'}, 'Node3') 2017-09-14 15:14:31,684 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:14:31,685 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:14:31,685 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'op': 'MESSAGE_REQUEST'}, 'Node3') 2017-09-14 15:14:31,685 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:14:31,685 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:14:31,685 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'op': 'MESSAGE_REQUEST'}, 'Node3') 2017-09-14 15:14:31,685 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:14:31,685 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:14:31,685 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'op': 'MESSAGE_REQUEST'}, 'Node3') 2017-09-14 15:14:31,685 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:14:31,686 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 6, 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-09-14 15:14:31,686 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 6, 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'op': 'VIEW_CHANGE_DONE', 'viewNo': 0}, 'Node3') 2017-09-14 15:14:31,686 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 6, 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-09-14 15:14:31,686 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: BATCH{'signature': None, 'messages': ['{"digest":"13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755","ppTime":1505402052,"stateRootHash":"BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH","op":"PREPARE","ppSeqNo":2,"txnRootHash":"C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby","viewNo":0,"instId":0}', '{"op":"COMMIT","instId":0,"viewNo":0,"ppSeqNo":2}']} 2017-09-14 15:14:31,686 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'signature': None, 'op': 'BATCH', 'messages': ['{"digest":"13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755","ppTime":1505402052,"stateRootHash":"BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH","op":"PREPARE","ppSeqNo":2,"txnRootHash":"C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby","viewNo":0,"instId":0}', '{"op":"COMMIT","instId":0,"viewNo":0,"ppSeqNo":2}']}, 'Node3') 2017-09-14 15:14:31,688 | DEBUG | node.py (1272) | unpackNodeMsg | Node1 processing a batch BATCH{'signature': None, 'messages': ['{"digest":"13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755","ppTime":1505402052,"stateRootHash":"BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH","op":"PREPARE","ppSeqNo":2,"txnRootHash":"C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby","viewNo":0,"instId":0}', '{"op":"COMMIT","instId":0,"viewNo":0,"ppSeqNo":2}']} 2017-09-14 15:14:31,689 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: PREPARE{'ppTime': 1505402052, 'viewNo': 0, 'txnRootHash': 'C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby', 'digest': '13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755', 'instId': 0, 'stateRootHash': 'BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH', 'ppSeqNo': 2} 2017-09-14 15:14:31,689 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'ppTime': 1505402052, 'op': 'PREPARE', 'viewNo': 0, 'txnRootHash': 'C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby', 'digest': '13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755', 'instId': 0, 'stateRootHash': 'BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH', 'ppSeqNo': 2}, 'Node3') 2017-09-14 15:14:31,689 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'ppTime': 1505402052, 'viewNo': 0, 'txnRootHash': 'C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby', 'digest': '13d0bc6d06baeb9cf3345aa191b38448a1c50cdd2f51b0d701cce21302f34755', 'instId': 0, 'stateRootHash': 'BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH', 'ppSeqNo': 2} 2017-09-14 15:14:31,689 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: COMMIT{'viewNo': 0, 'ppSeqNo': 2, 'instId': 0} 2017-09-14 15:14:31,689 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'viewNo': 0, 'ppSeqNo': 2, 'op': 'COMMIT', 'instId': 0}, 'Node3') 2017-09-14 15:14:31,689 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'viewNo': 0, 'ppSeqNo': 2, 'instId': 0} 2017-09-14 15:14:31,689 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None}} to 1 recipients: ['Node3'] 2017-09-14 15:14:31,690 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'txnSeqNo': 0, 'ppSeqNo': None, 'viewNo': None}} to 1 recipients: ['Node3'] 2017-09-14 15:14:31,690 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': LEDGER_STATUS{'merkleRoot': 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE', 'ledgerId': 1, 'txnSeqNo': 6, 'ppSeqNo': 1, 'viewNo': 0}} to 1 recipients: ['Node3'] 2017-09-14 15:14:31,690 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': LEDGER_STATUS{'merkleRoot': 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE', 'ledgerId': 1, 'txnSeqNo': 6, 'ppSeqNo': 1, 'viewNo': 0}} to 1 recipients: ['Node3'] 2017-09-14 15:14:31,691 | DEBUG | node.py (1205) | sendToElector | Node1 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 6, 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0}, 'Node3') 2017-09-14 15:14:31,691 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node1's primary selector started processing of ViewChangeDone msg from Node3 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 6, 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-09-14 15:14:31,691 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 6, 'Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} because it already decided primary which is Node1:0 2017-09-14 15:14:31,691 | DEBUG | batched.py ( 100) | flushOutBoxes | Node1 batching 4 msgs to Node3 into one transmission 2017-09-14 15:14:31,691 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', b'{"msg":{"ledgerId":1,"merkleRoot":"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE","ppSeqNo":1,"txnSeqNo":6,"viewNo":0},"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', b'{"msg":{"ledgerId":1,"merkleRoot":"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE","ppSeqNo":1,"txnSeqNo":6,"viewNo":0},"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}']) 2017-09-14 15:14:31,692 | TRACE | batched.py ( 110) | flushOutBoxes | Node1 sending payload to Node3: b'{"signature":null,"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE\\",\\"ppSeqNo\\":1,\\"txnSeqNo\\":6,\\"viewNo\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE\\",\\"ppSeqNo\\":1,\\"txnSeqNo\\":6,\\"viewNo\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}"]}' 2017-09-14 15:14:31,692 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"signature":null,"op":"BATCH","messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE\\",\\"ppSeqNo\\":1,\\"txnSeqNo\\":6,\\"viewNo\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"Gaxb4vvzCLfhbozxq1V3DPs7vHRXHZdhB1DfWFKxxjQE\\",\\"ppSeqNo\\":1,\\"txnSeqNo\\":6,\\"viewNo\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}"]}' to Node3 2017-09-14 15:14:31,692 | DEBUG | replica.py ( 953) | doCommit | Node1:0 Sending COMMIT(0, 2) at 13448.544374191 2017-09-14 15:14:31,693 | DEBUG | replica.py (2193) | send | Node1:0 sending Commit 2017-09-14 15:14:31,693 | TRACE | replica.py (2194) | send | Node1:0 sending COMMIT{'viewNo': 0, 'ppSeqNo': 2, 'instId': 0} 2017-09-14 15:14:31,693 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'viewNo': 0, 'ppSeqNo': 2, 'instId': 0} commits where f is 0 2017-09-14 15:14:31,693 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (0, 2) 2017-09-14 15:14:31,693 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(0, 2) from Node3:0 2017-09-14 15:14:31,693 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'viewNo': 0, 'ppSeqNo': 2, 'instId': 0} commits where f is 0 2017-09-14 15:14:31,693 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(0, 2) 2017-09-14 15:14:31,693 | DEBUG | node.py (2608) | send | Node1 sending message COMMIT{'viewNo': 0, 'ppSeqNo': 2, 'instId': 0} to all recipients: ['Node2', 'Node3'] 2017-09-14 15:14:31,694 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"instId":0,"ppSeqNo":2,"op":"COMMIT","viewNo":0}' to Node2 2017-09-14 15:14:31,694 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"instId":0,"ppSeqNo":2,"op":"COMMIT","viewNo":0}' to Node2 2017-09-14 15:14:31,694 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"instId":0,"ppSeqNo":2,"op":"COMMIT","viewNo":0}' to Node3 2017-09-14 15:14:31,695 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"instId":0,"ppSeqNo":2,"op":"COMMIT","viewNo":0}' to Node3 2017-09-14 15:14:31,706 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 1 messages through listener 2017-09-14 15:14:31,707 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: COMMIT{'viewNo': 0, 'ppSeqNo': 2, 'instId': 0} 2017-09-14 15:14:31,707 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'viewNo': 0, 'ppSeqNo': 2, 'op': 'COMMIT', 'instId': 0}, 'Node2') 2017-09-14 15:14:31,707 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'viewNo': 0, 'ppSeqNo': 2, 'instId': 0} 2017-09-14 15:14:31,708 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(0, 2) from Node2:0 2017-09-14 15:14:31,708 | TRACE | replica.py ( 925) | tryOrder | Node1:0 returning request to node 2017-09-14 15:14:31,708 | DEBUG | replica.py (1444) | doOrder | Node1:0 ordering COMMIT (0, 2) 2017-09-14 15:14:31,708 | DEBUG | replica.py ( 323) | last_ordered_3pc | Node1:0 set last ordered as (0, 2) 2017-09-14 15:14:31,708 | DEBUG | replica.py (2193) | send | Node1:0 sending Ordered 2017-09-14 15:14:31,709 | TRACE | replica.py (2194) | send | Node1:0 sending ORDERED{'ppTime': 1505402052, 'viewNo': 0, 'txnRootHash': 'C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby', 'ledgerId': 1, 'instId': 0, 'stateRootHash': 'BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH', 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402052949993)], 'ppSeqNo': 2} 2017-09-14 15:14:31,709 | INFO | replica.py (1483) | order_3pc_key | Node1:0 ordered batch request, view no 0, ppSeqNo 2, ledger 1, state root BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH, txn root C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1505402052949993)], discarded [] 2017-09-14 15:14:31,709 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(0, 2) 2017-09-14 15:14:31,709 | DEBUG | node.py (1847) | processOrdered | Node1 executing Ordered batch 0 2 of 1 requests 2017-09-14 15:14:31,709 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-09-14 15:14:31,710 | DEBUG | node.py (2386) | sendReplyToClient | Node1 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1505402052949993) to client 2017-09-14 15:14:31,710 | TRACE | zstack.py ( 754) | transmitThroughListener | Node1C transmitting b'{"op":"REPLY","result":{"signature":"48dbAqo8qbwK82kwYBRJv2Zxu5VmqpKRizg5xdMcw71WjzZj5px4Z5xnRCbzS6VmN6WsT1T3UTmm3mCZ6y95gsmk","txnTime":1505402052,"seqNo":7,"reqId":1505402052949993,"identifier":"V4SGRU86Z58d6TV7PBUe6f","type":"1","rootHash":"C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby","auditPath":["Dvt18TQs3z1emxZcnmW3jpcjdsycqSQwPFm63UUJzJbs","FA73SaKHCBxBXhf9gis5FxVgz3KxD5GJ1n12yyVYHhDj"],"dest":"V4SGRU86Z58d6TV7PBUe62"}}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:14:31,710 | INFO | node.py (2290) | executeBatch | Node1 committed batch request, view no 0, ppSeqNo 2, ledger 1, state root BFXNSfjBBvfFymjxRypUgbr7ecYhj94uAJmE2HmtEEBH, txn root C7hhTtCLhnHyWzGkvop4c7Jr6GfBoUbig1DrN8QgCcby, requests: [('V4SGRU86Z58d6TV7PBUe6f', 1505402052949993)] 2017-09-14 15:14:31,710 | DEBUG | node.py (2305) | executeBatch | Node1 storing 3PC key (0, 2) for ledger 1 range (7, 7) 2017-09-14 15:14:31,711 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:14:35,909 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 39 2017-09-14 15:14:35,909 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:14:35,909 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:14:35,910 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:14:35,910 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:14:35,910 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:14:35,910 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:14:35,910 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 40 to run in 10 seconds 2017-09-14 15:14:36,121 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:14:36,121 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:14:36,121 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:14:37,048 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 1 messages through listener 2017-09-14 15:14:37,049 | DEBUG | node.py (2236) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1505402077029452 2017-09-14 15:14:37,049 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: PROPAGATE{'request': {'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2', 'reqId': 1505402077029452, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe63'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:14:37,050 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'request': {'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2', 'reqId': 1505402077029452, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe63'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}, 'op': 'PROPAGATE', 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'}, 'Node2') 2017-09-14 15:14:37,050 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'request': {'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2', 'reqId': 1505402077029452, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe63'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:14:37,050 | DEBUG | node.py (1757) | processPropagate | Node Node1 received propagated request: PROPAGATE{'request': {'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2', 'reqId': 1505402077029452, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe63'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:14:37,050 | TRACE | propagator.py ( 165) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2', 'reqId': 1505402077029452, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe63'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} 2017-09-14 15:14:37,050 | INFO | propagator.py ( 148) | propagate | Node1 propagating request ('V4SGRU86Z58d6TV7PBUe6f', 1505402077029452) from client v#ZljK34WWooA6%RgESn.Ox 2017-09-14 15:14:37,050 | DEBUG | node.py (2608) | send | Node1 sending message PROPAGATE{'request': {'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2', 'reqId': 1505402077029452, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe63'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} to all recipients: ['Node2', 'Node3'] 2017-09-14 15:14:37,051 | DEBUG | propagator.py ( 214) | forward | Node1 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1505402077029452) to 0 replicas 2017-09-14 15:14:37,051 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1C got 1 messages through listener 2017-09-14 15:14:37,052 | DEBUG | node.py (2236) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1505402077029452 2017-09-14 15:14:37,052 | TRACE | node.py (1388) | validateClientMsg | Node1C received CLIENT message: SafeRequest: {'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2', 'reqId': 1505402077029452, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe63'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} 2017-09-14 15:14:37,052 | DEBUG | node.py (1434) | processClientInBox | Node1C processing b'v#ZljK34WWooA6%RgESn.Ox' request SafeRequest: {'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2', 'reqId': 1505402077029452, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe63'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} 2017-09-14 15:14:37,052 | DEBUG | node.py (1708) | processRequest | Node1 received client request: SafeRequest: {'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2', 'reqId': 1505402077029452, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe63'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:14:37,052 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2', 'reqId': 1505402077029452, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe63'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} 2017-09-14 15:14:37,052 | DEBUG | propagator.py ( 246) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2', 'reqId': 1505402077029452, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe63'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} to its replicas since already forwarded 2017-09-14 15:14:37,052 | TRACE | zstack.py ( 754) | transmitThroughListener | Node1C transmitting b'{"reqId":1505402077029452,"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f"}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:14:37,053 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"request":{"signature":"vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2","reqId":1505402077029452,"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe63"},"identifier":"V4SGRU86Z58d6TV7PBUe6f"},"op":"PROPAGATE","senderClient":"v#ZljK34WWooA6%RgESn.Ox"}' to Node2 2017-09-14 15:14:37,053 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"request":{"signature":"vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2","reqId":1505402077029452,"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe63"},"identifier":"V4SGRU86Z58d6TV7PBUe6f"},"op":"PROPAGATE","senderClient":"v#ZljK34WWooA6%RgESn.Ox"}' to Node2 2017-09-14 15:14:37,053 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"request":{"signature":"vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2","reqId":1505402077029452,"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe63"},"identifier":"V4SGRU86Z58d6TV7PBUe6f"},"op":"PROPAGATE","senderClient":"v#ZljK34WWooA6%RgESn.Ox"}' to Node3 2017-09-14 15:14:37,053 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"request":{"signature":"vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2","reqId":1505402077029452,"operation":{"type":"1","dest":"V4SGRU86Z58d6TV7PBUe63"},"identifier":"V4SGRU86Z58d6TV7PBUe6f"},"op":"PROPAGATE","senderClient":"v#ZljK34WWooA6%RgESn.Ox"}' to Node3 2017-09-14 15:14:37,054 | DEBUG | replica.py ( 667) | create3PCBatch | Node1:0 creating batch 3 for ledger 1 with state root b'\x98MV\xf7\xcf\xb2W\xefE\xef\t*Q\xe7\xf9\xa6tK\xed\xcd\xc4\xc7\xf9\x85\xf7\xcea\xf0\xce\xc8\xb2`' 2017-09-14 15:14:37,055 | DEBUG | replica.py ( 700) | create3PCBatch | Node1:0 created a PRE-PREPARE with 1 requests for ledger 1 2017-09-14 15:14:37,055 | DEBUG | replica.py (2193) | send | Node1:0 sending PrePrepare 2017-09-14 15:14:37,055 | TRACE | replica.py (2194) | send | Node1:0 sending PREPREPARE{'discarded': 1, 'ppTime': 1505402077, 'viewNo': 0, 'txnRootHash': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'digest': 'defbbfd4c1fa5e018f6bd533c9ead325d16bf5ff804d0c53735b190f68f6dbd2', 'instId': 0, 'stateRootHash': 'DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8', 'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402077029452)], 'ppSeqNo': 3} 2017-09-14 15:14:37,056 | DEBUG | replica.py ( 614) | trackBatches | Node1:0 tracking batch for PREPREPARE{'discarded': 1, 'ppTime': 1505402077, 'viewNo': 0, 'txnRootHash': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'digest': 'defbbfd4c1fa5e018f6bd533c9ead325d16bf5ff804d0c53735b190f68f6dbd2', 'instId': 0, 'stateRootHash': 'DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8', 'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402077029452)], 'ppSeqNo': 3} with state root b'\x98MV\xf7\xcf\xb2W\xefE\xef\t*Q\xe7\xf9\xa6tK\xed\xcd\xc4\xc7\xf9\x85\xf7\xcea\xf0\xce\xc8\xb2`' 2017-09-14 15:14:37,056 | DEBUG | node.py (2608) | send | Node1 sending message PREPREPARE{'discarded': 1, 'ppTime': 1505402077, 'viewNo': 0, 'txnRootHash': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'digest': 'defbbfd4c1fa5e018f6bd533c9ead325d16bf5ff804d0c53735b190f68f6dbd2', 'instId': 0, 'stateRootHash': 'DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8', 'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402077029452)], 'ppSeqNo': 3} to all recipients: ['Node2', 'Node3'] 2017-09-14 15:14:37,056 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 1 messages through listener 2017-09-14 15:14:37,057 | DEBUG | node.py (2236) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1505402077029452 2017-09-14 15:14:37,057 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: PROPAGATE{'request': {'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2', 'reqId': 1505402077029452, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe63'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:14:37,064 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'request': {'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2', 'reqId': 1505402077029452, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe63'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}, 'op': 'PROPAGATE', 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'}, 'Node3') 2017-09-14 15:14:37,065 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'request': {'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2', 'reqId': 1505402077029452, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe63'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:14:37,065 | DEBUG | node.py (1757) | processPropagate | Node Node1 received propagated request: PROPAGATE{'request': {'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2', 'reqId': 1505402077029452, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe63'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:14:37,065 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2', 'reqId': 1505402077029452, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe63'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} 2017-09-14 15:14:37,065 | DEBUG | propagator.py ( 246) | tryForwarding | Node1 not forwarding request SafeRequest: {'signature': 'vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2', 'reqId': 1505402077029452, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe63'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} to its replicas since already forwarded 2017-09-14 15:14:37,066 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"discarded":1,"ppTime":1505402077,"digest":"defbbfd4c1fa5e018f6bd533c9ead325d16bf5ff804d0c53735b190f68f6dbd2","instId":0,"op":"PREPREPARE","viewNo":0,"txnRootHash":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","stateRootHash":"DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8","ledgerId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1505402077029452]],"ppSeqNo":3}' to Node2 2017-09-14 15:14:37,066 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"discarded":1,"ppTime":1505402077,"digest":"defbbfd4c1fa5e018f6bd533c9ead325d16bf5ff804d0c53735b190f68f6dbd2","instId":0,"op":"PREPREPARE","viewNo":0,"txnRootHash":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","stateRootHash":"DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8","ledgerId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1505402077029452]],"ppSeqNo":3}' to Node2 2017-09-14 15:14:37,066 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"discarded":1,"ppTime":1505402077,"digest":"defbbfd4c1fa5e018f6bd533c9ead325d16bf5ff804d0c53735b190f68f6dbd2","instId":0,"op":"PREPREPARE","viewNo":0,"txnRootHash":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","stateRootHash":"DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8","ledgerId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1505402077029452]],"ppSeqNo":3}' to Node3 2017-09-14 15:14:37,066 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"discarded":1,"ppTime":1505402077,"digest":"defbbfd4c1fa5e018f6bd533c9ead325d16bf5ff804d0c53735b190f68f6dbd2","instId":0,"op":"PREPREPARE","viewNo":0,"txnRootHash":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","stateRootHash":"DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8","ledgerId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1505402077029452]],"ppSeqNo":3}' to Node3 2017-09-14 15:14:37,081 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 1 messages through listener 2017-09-14 15:14:37,081 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: PREPARE{'ppTime': 1505402077, 'viewNo': 0, 'txnRootHash': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'digest': 'defbbfd4c1fa5e018f6bd533c9ead325d16bf5ff804d0c53735b190f68f6dbd2', 'instId': 0, 'stateRootHash': 'DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8', 'ppSeqNo': 3} 2017-09-14 15:14:37,081 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'op': 'PREPARE', 'ppTime': 1505402077, 'instId': 0, 'txnRootHash': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'stateRootHash': 'DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8', 'viewNo': 0, 'digest': 'defbbfd4c1fa5e018f6bd533c9ead325d16bf5ff804d0c53735b190f68f6dbd2', 'ppSeqNo': 3}, 'Node2') 2017-09-14 15:14:37,082 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'ppTime': 1505402077, 'viewNo': 0, 'txnRootHash': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'digest': 'defbbfd4c1fa5e018f6bd533c9ead325d16bf5ff804d0c53735b190f68f6dbd2', 'instId': 0, 'stateRootHash': 'DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8', 'ppSeqNo': 3} 2017-09-14 15:14:37,082 | DEBUG | replica.py ( 917) | tryCommit | Node1:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'ppTime': 1505402077, 'viewNo': 0, 'txnRootHash': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'digest': 'defbbfd4c1fa5e018f6bd533c9ead325d16bf5ff804d0c53735b190f68f6dbd2', 'instId': 0, 'stateRootHash': 'DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8', 'ppSeqNo': 3} 2017-09-14 15:14:37,082 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (0, 3) 2017-09-14 15:14:37,095 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 3 messages through listener 2017-09-14 15:14:37,096 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: PREPARE{'ppTime': 1505402077, 'viewNo': 0, 'txnRootHash': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'digest': 'defbbfd4c1fa5e018f6bd533c9ead325d16bf5ff804d0c53735b190f68f6dbd2', 'instId': 0, 'stateRootHash': 'DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8', 'ppSeqNo': 3} 2017-09-14 15:14:37,096 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'ppTime': 1505402077, 'op': 'PREPARE', 'viewNo': 0, 'txnRootHash': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'digest': 'defbbfd4c1fa5e018f6bd533c9ead325d16bf5ff804d0c53735b190f68f6dbd2', 'instId': 0, 'stateRootHash': 'DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8', 'ppSeqNo': 3}, 'Node3') 2017-09-14 15:14:37,096 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'ppTime': 1505402077, 'viewNo': 0, 'txnRootHash': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'digest': 'defbbfd4c1fa5e018f6bd533c9ead325d16bf5ff804d0c53735b190f68f6dbd2', 'instId': 0, 'stateRootHash': 'DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8', 'ppSeqNo': 3} 2017-09-14 15:14:37,096 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: COMMIT{'viewNo': 0, 'ppSeqNo': 3, 'instId': 0} 2017-09-14 15:14:37,096 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'viewNo': 0, 'ppSeqNo': 3, 'op': 'COMMIT', 'instId': 0}, 'Node2') 2017-09-14 15:14:37,096 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'viewNo': 0, 'ppSeqNo': 3, 'instId': 0} 2017-09-14 15:14:37,097 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: COMMIT{'viewNo': 0, 'ppSeqNo': 3, 'instId': 0} 2017-09-14 15:14:37,097 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'viewNo': 0, 'ppSeqNo': 3, 'op': 'COMMIT', 'instId': 0}, 'Node3') 2017-09-14 15:14:37,097 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'viewNo': 0, 'ppSeqNo': 3, 'instId': 0} 2017-09-14 15:14:37,098 | DEBUG | replica.py ( 953) | doCommit | Node1:0 Sending COMMIT(0, 3) at 13453.949663044 2017-09-14 15:14:37,098 | DEBUG | replica.py (2193) | send | Node1:0 sending Commit 2017-09-14 15:14:37,098 | TRACE | replica.py (2194) | send | Node1:0 sending COMMIT{'viewNo': 0, 'ppSeqNo': 3, 'instId': 0} 2017-09-14 15:14:37,098 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'viewNo': 0, 'ppSeqNo': 3, 'instId': 0} commits where f is 0 2017-09-14 15:14:37,098 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (0, 3) 2017-09-14 15:14:37,098 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(0, 3) from Node2:0 2017-09-14 15:14:37,098 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'viewNo': 0, 'ppSeqNo': 3, 'instId': 0} commits where f is 0 2017-09-14 15:14:37,099 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(0, 3) 2017-09-14 15:14:37,099 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(0, 3) from Node3:0 2017-09-14 15:14:37,099 | TRACE | replica.py ( 925) | tryOrder | Node1:0 returning request to node 2017-09-14 15:14:37,099 | DEBUG | replica.py (1444) | doOrder | Node1:0 ordering COMMIT (0, 3) 2017-09-14 15:14:37,099 | DEBUG | replica.py ( 323) | last_ordered_3pc | Node1:0 set last ordered as (0, 3) 2017-09-14 15:14:37,099 | DEBUG | replica.py (2193) | send | Node1:0 sending Ordered 2017-09-14 15:14:37,099 | TRACE | replica.py (2194) | send | Node1:0 sending ORDERED{'ppTime': 1505402077, 'viewNo': 0, 'txnRootHash': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ledgerId': 1, 'instId': 0, 'stateRootHash': 'DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8', 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402077029452)], 'ppSeqNo': 3} 2017-09-14 15:14:37,099 | INFO | replica.py (1483) | order_3pc_key | Node1:0 ordered batch request, view no 0, ppSeqNo 3, ledger 1, state root DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8, txn root 4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1505402077029452)], discarded [] 2017-09-14 15:14:37,100 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(0, 3) 2017-09-14 15:14:37,100 | DEBUG | node.py (2608) | send | Node1 sending message COMMIT{'viewNo': 0, 'ppSeqNo': 3, 'instId': 0} to all recipients: ['Node2', 'Node3'] 2017-09-14 15:14:37,100 | DEBUG | node.py (1847) | processOrdered | Node1 executing Ordered batch 0 3 of 1 requests 2017-09-14 15:14:37,100 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-09-14 15:14:37,101 | DEBUG | node.py (2386) | sendReplyToClient | Node1 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1505402077029452) to client 2017-09-14 15:14:37,101 | TRACE | zstack.py ( 754) | transmitThroughListener | Node1C transmitting b'{"op":"REPLY","result":{"signature":"vQSGwr2GmUmVztNhKaf9Jfw7kkkmkNqn1DwzWY1bxpXZBJfVuRLTpTQ2hCMhE5TrBD6aiGNthg9YiGNS6Bjt1o2","txnTime":1505402077,"seqNo":8,"reqId":1505402077029452,"identifier":"V4SGRU86Z58d6TV7PBUe6f","type":"1","rootHash":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","auditPath":["BvyY1duMNWfjxw3JuAzptvrLmpLbN3W6vL68oL5CX86f","Dvt18TQs3z1emxZcnmW3jpcjdsycqSQwPFm63UUJzJbs","FA73SaKHCBxBXhf9gis5FxVgz3KxD5GJ1n12yyVYHhDj"],"dest":"V4SGRU86Z58d6TV7PBUe63"}}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:14:37,101 | INFO | node.py (2290) | executeBatch | Node1 committed batch request, view no 0, ppSeqNo 3, ledger 1, state root DZhECFJ3hshLYdKYHhCf5Xd1RzC5fVcXrSiiYbGqiWR8, txn root 4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja, requests: [('V4SGRU86Z58d6TV7PBUe6f', 1505402077029452)] 2017-09-14 15:14:37,101 | DEBUG | node.py (2305) | executeBatch | Node1 storing 3PC key (0, 3) for ledger 1 range (8, 8) 2017-09-14 15:14:37,101 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:14:37,102 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"instId":0,"ppSeqNo":3,"op":"COMMIT","viewNo":0}' to Node2 2017-09-14 15:14:37,102 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"instId":0,"ppSeqNo":3,"op":"COMMIT","viewNo":0}' to Node2 2017-09-14 15:14:37,102 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"instId":0,"ppSeqNo":3,"op":"COMMIT","viewNo":0}' to Node3 2017-09-14 15:14:37,103 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"instId":0,"ppSeqNo":3,"op":"COMMIT","viewNo":0}' to Node3 2017-09-14 15:14:45,912 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 40 2017-09-14 15:14:45,912 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:14:45,912 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:14:45,912 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:14:45,913 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:14:45,913 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:14:45,913 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:14:45,913 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 41 to run in 10 seconds 2017-09-14 15:14:51,125 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:14:51,126 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:14:51,126 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:14:55,920 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 41 2017-09-14 15:14:55,921 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:14:55,921 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:14:55,921 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:14:55,921 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:14:55,921 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:14:55,921 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:14:55,921 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 42 to run in 10 seconds 2017-09-14 15:15:22,393 | DEBUG | node_runner.py ( 18) | run_node | You can find logs in /home/sovrin/.sovrin/Node1.log 2017-09-14 15:15:22,393 | DEBUG | node_runner.py ( 21) | run_node | Sovrin related env vars: [] 2017-09-14 15:15:23,297 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: git 2017-09-14 15:15:23,311 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: hg 2017-09-14 15:15:23,343 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: svn 2017-09-14 15:15:23,344 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: bzr 2017-09-14 15:15:23,928 | DEBUG | selector_events.py ( 53) | __init__ | Using selector: EpollSelector 2017-09-14 15:15:23,929 | DEBUG | looper.py ( 125) | __init__ | Setting handler for SIGINT 2017-09-14 15:15:23,963 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-09-14 15:15:23,991 | DEBUG | ledger.py ( 72) | recoverTree | Recovering tree from hash store of size 8 2017-09-14 15:15:23,992 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.0005262359991320409 seconds 2017-09-14 15:15:24,050 | DEBUG | idr_cache.py ( 25) | __init__ | Initializing identity cache Node1 2017-09-14 15:15:24,160 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-09-14 15:15:24,193 | DEBUG | ledger.py ( 72) | recoverTree | Recovering tree from hash store of size 3 2017-09-14 15:15:24,193 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.00044606499977817293 seconds 2017-09-14 15:15:24,238 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-09-14 15:15:24,238 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: typing.Dict<~KT, ~VT>] because it does not have a 'pluginType' attribute 2017-09-14 15:15:24,238 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-09-14 15:15:24,238 | INFO | plugin_loader.py ( 117) | _load | plugin FirebaseStatsConsumer successfully loaded from module plugin_firebase_stats_consumer 2017-09-14 15:15:24,238 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-09-14 15:15:24,238 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-09-14 15:15:24,239 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkPerformance will be repeating every 60 seconds 2017-09-14 15:15:24,239 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 1 to run in 60 seconds 2017-09-14 15:15:24,239 | DEBUG | replica.py ( 313) | h | Node1:0 set watermarks as 0 300 2017-09-14 15:15:24,239 | DISPLAY | replicas.py ( 36) | grow | Node1 added replica Node1:0 to instance 0 (master) 2017-09-14 15:15:24,239 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkPerformance will be repeating every 10 seconds 2017-09-14 15:15:24,240 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 1 to run in 10 seconds 2017-09-14 15:15:24,240 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkNodeRequestSpike will be repeating every 60 seconds 2017-09-14 15:15:24,240 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkNodeRequestSpike with id 2 to run in 60 seconds 2017-09-14 15:15:24,240 | DEBUG | plugin_helper.py ( 24) | loadPlugins | Plugin loading started to load plugins from basedir: /home/sovrin/.sovrin 2017-09-14 15:15:24,240 | DEBUG | plugin_helper.py ( 68) | loadPlugins | Total plugins loaded from basedir /home/sovrin/.sovrin are : 0 2017-09-14 15:15:24,240 | DEBUG | node.py ( 340) | __init__ | total plugins loaded in node: 0 2017-09-14 15:15:24,309 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-09-14 15:15:24,330 | DEBUG | ledger.py ( 67) | recoverTree | Recovering tree from transaction log 2017-09-14 15:15:24,393 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.06353094199948828 seconds 2017-09-14 15:15:24,419 | DEBUG | upgrader.py ( 118) | check_upgrade_succeeded | Node Node1 has no upgrade events 2017-09-14 15:15:24,419 | INFO | node.py (2420) | initStateFromLedger | Node1 found state to be empty, recreating from ledger 2017-09-14 15:15:24,419 | DEBUG | motor.py ( 34) | set_status | Node1 changing status from stopped to starting 2017-09-14 15:15:24,419 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-09-14 15:15:24,420 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-09-14 15:15:24,420 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-09-14 15:15:24,420 | DEBUG | zstack.py ( 319) | start | Node1 starting with restricted as True and reSetupAuth as True 2017-09-14 15:15:24,420 | DEBUG | authenticator.py ( 31) | start | Starting ZAP at inproc://zeromq.zap.1 2017-09-14 15:15:24,420 | DEBUG | base.py ( 72) | allow | Allowing 0.0.0.0 2017-09-14 15:15:24,420 | DEBUG | base.py ( 112) | configure_curve | Configure curve: *[/home/sovrin/.sovrin/Node1/public_keys] 2017-09-14 15:15:24,421 | DEBUG | zstack.py ( 347) | open | Node1 will bind its listener at 9701 2017-09-14 15:15:24,421 | INFO | stacks.py ( 84) | start | CONNECTION: Node1 listening for other nodes at 0.0.0.0:9701 2017-09-14 15:15:24,421 | DEBUG | zstack.py ( 319) | start | Node1C starting with restricted as False and reSetupAuth as True 2017-09-14 15:15:24,422 | DEBUG | authenticator.py ( 31) | start | Starting ZAP at inproc://zeromq.zap.2 2017-09-14 15:15:24,422 | DEBUG | base.py ( 72) | allow | Allowing 0.0.0.0 2017-09-14 15:15:24,422 | DEBUG | base.py ( 112) | configure_curve | Configure curve: *[*] 2017-09-14 15:15:24,422 | DEBUG | zstack.py ( 347) | open | Node1C will bind its listener at 9702 2017-09-14 15:15:24,423 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action propose_view_change with id 3 to run in 60 seconds 2017-09-14 15:15:24,423 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action dump_json_file with id 4 to run in 3 seconds 2017-09-14 15:15:24,423 | DEBUG | has_action_queue.py ( 79) | startRepeating | dump_json_file will be repeating every 60 seconds 2017-09-14 15:15:24,423 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action dump_json_file with id 5 to run in 60 seconds 2017-09-14 15:15:24,423 | INFO | node.py ( 621) | start | Node1 first time running... 2017-09-14 15:15:24,424 | DEBUG | kit_zstack.py ( 97) | connectToMissing | CONNECTION: Node1 found the following missing connections: Node3, Node2 2017-09-14 15:15:24,424 | TRACE | remote.py ( 86) | connect | connecting socket 78 28835968 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-09-14 15:15:24,424 | INFO | zstack.py ( 590) | connect | CONNECTION: Node1 looking for Node3 at 10.0.0.4:9705 2017-09-14 15:15:24,428 | DEBUG | zstack.py ( 645) | sendPingPong | Node1 pinged Node3 2017-09-14 15:15:24,431 | TRACE | remote.py ( 86) | connect | connecting socket 81 28858128 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-09-14 15:15:24,431 | INFO | zstack.py ( 590) | connect | CONNECTION: Node1 looking for Node2 at 10.0.0.3:9703 2017-09-14 15:15:24,431 | DEBUG | zstack.py ( 645) | sendPingPong | Node1 pinged Node2 2017-09-14 15:15:24,431 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:15:24,442 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'pi' to Node3 2017-09-14 15:15:24,443 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'pi' to Node3 2017-09-14 15:15:24,443 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'pi' to Node2 2017-09-14 15:15:24,443 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'pi' to Node2 2017-09-14 15:15:24,480 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 2 messages through listener 2017-09-14 15:15:24,480 | DEBUG | zstack.py ( 663) | handlePingPong | Node1 got ping from Node2 2017-09-14 15:15:24,480 | DEBUG | zstack.py ( 645) | sendPingPong | Node1 ponged Node2 2017-09-14 15:15:24,481 | DEBUG | zstack.py ( 669) | handlePingPong | Node1 got pong from Node2 2017-09-14 15:15:24,481 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'po' to Node2 2017-09-14 15:15:24,481 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'po' to Node2 2017-09-14 15:15:24,481 | DEBUG | keep_in_touch.py ( 68) | conns | Node1's connections changed from set() to {'Node2'} 2017-09-14 15:15:24,481 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node1 now connected to Node2 2017-09-14 15:15:24,482 | DEBUG | motor.py ( 34) | set_status | Node1 changing status from starting to started_hungry 2017-09-14 15:15:24,482 | DEBUG | node.py ( 941) | checkInstances | Node1 choosing to start election on the basis of count 2 and nodes {'Node2'} 2017-09-14 15:15:24,483 | DEBUG | primary_selector.py ( 77) | get_msgs_for_lagged_nodes | Node1 has no ViewChangeDone message to send for view 0 2017-09-14 15:15:24,483 | DEBUG | node.py ( 902) | send_current_state_to_lagging_node | Node1 sending current state CURRENT_STATE{'primary': [], 'viewNo': 0} to lagged node Node2 2017-09-14 15:15:24,483 | DEBUG | node.py (2608) | send | Node1 sending message CURRENT_STATE{'primary': [], 'viewNo': 0} to 1 recipients: ['Node2'] 2017-09-14 15:15:24,483 | DEBUG | node.py (2608) | send | Node1 sending message LEDGER_STATUS{'txnSeqNo': 3, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None} to 1 recipients: ['Node2'] 2017-09-14 15:15:24,484 | DEBUG | batched.py ( 100) | flushOutBoxes | Node1 batching 2 msgs to Node2 into one transmission 2017-09-14 15:15:24,484 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"primary":[],"op":"CURRENT_STATE","viewNo":0}', b'{"txnSeqNo":3,"ppSeqNo":null,"op":"LEDGER_STATUS","viewNo":null,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ledgerId":0}']) 2017-09-14 15:15:24,484 | TRACE | batched.py ( 110) | flushOutBoxes | Node1 sending payload to Node2: b'{"messages":["{\\"primary\\":[],\\"op\\":\\"CURRENT_STATE\\",\\"viewNo\\":0}","{\\"txnSeqNo\\":3,\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null,\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"ledgerId\\":0}"],"op":"BATCH","signature":null}' 2017-09-14 15:15:24,484 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"messages":["{\\"primary\\":[],\\"op\\":\\"CURRENT_STATE\\",\\"viewNo\\":0}","{\\"txnSeqNo\\":3,\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null,\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"ledgerId\\":0}"],"op":"BATCH","signature":null}' to Node2 2017-09-14 15:15:24,496 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 4 messages through listener 2017-09-14 15:15:24,498 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} 2017-09-14 15:15:24,498 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'reason': 26, 'op': 'INSTANCE_CHANGE', 'viewNo': 1}, 'Node3') 2017-09-14 15:15:24,498 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} 2017-09-14 15:15:24,499 | DEBUG | node.py (2236) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1505402110572745 2017-09-14 15:15:24,499 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: PROPAGATE{'senderClient': 'v#ZljK34WWooA6%RgESn.Ox', 'request': {'signature': 'JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF', 'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe64', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402110572745}} 2017-09-14 15:15:24,499 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'senderClient': 'v#ZljK34WWooA6%RgESn.Ox', 'op': 'PROPAGATE', 'request': {'signature': 'JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF', 'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe64', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402110572745}}, 'Node3') 2017-09-14 15:15:24,499 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': 'v#ZljK34WWooA6%RgESn.Ox', 'request': {'signature': 'JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF', 'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe64', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402110572745}} 2017-09-14 15:15:24,500 | DEBUG | zstack.py ( 663) | handlePingPong | Node1 got ping from Node3 2017-09-14 15:15:24,500 | DEBUG | zstack.py ( 645) | sendPingPong | Node1 ponged Node3 2017-09-14 15:15:24,500 | DEBUG | zstack.py ( 669) | handlePingPong | Node1 got pong from Node3 2017-09-14 15:15:24,500 | DEBUG | node.py (1915) | processInstanceChange | Node1 received instance change request: INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} from Node3 2017-09-14 15:15:24,500 | DEBUG | node.py (1958) | do_view_change_if_possible | Node1 has no quorum for view 1 2017-09-14 15:15:24,500 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:15:24,500 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:15:24,500 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:15:24,500 | DEBUG | node.py (1946) | processInstanceChange | Node1 received instance change message INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} but did not find the master to be slow or has already sent an instance change message 2017-09-14 15:15:24,501 | DEBUG | node.py (1757) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': 'v#ZljK34WWooA6%RgESn.Ox', 'request': {'signature': 'JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF', 'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe64', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402110572745}} 2017-09-14 15:15:24,501 | TRACE | propagator.py ( 165) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'reqId': 1505402110572745, 'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe64', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF'} 2017-09-14 15:15:24,501 | INFO | propagator.py ( 148) | propagate | Node1 propagating request ('V4SGRU86Z58d6TV7PBUe6f', 1505402110572745) from client v#ZljK34WWooA6%RgESn.Ox 2017-09-14 15:15:24,501 | DEBUG | node.py (2608) | send | Node1 sending message PROPAGATE{'senderClient': 'v#ZljK34WWooA6%RgESn.Ox', 'request': {'reqId': 1505402110572745, 'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe64', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF'}} to all recipients: ['Node3', 'Node2'] 2017-09-14 15:15:24,501 | DEBUG | propagator.py ( 214) | forward | Node1 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1505402110572745) to 0 replicas 2017-09-14 15:15:24,502 | DEBUG | batched.py ( 100) | flushOutBoxes | Node1 batching 2 msgs to Node3 into one transmission 2017-09-14 15:15:24,502 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'po', b'{"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE","request":{"reqId":1505402110572745,"operation":{"dest":"V4SGRU86Z58d6TV7PBUe64","type":"1"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF"}}']) 2017-09-14 15:15:24,502 | TRACE | batched.py ( 110) | flushOutBoxes | Node1 sending payload to Node3: b'{"messages":["po","{\\"senderClient\\":\\"v#ZljK34WWooA6%RgESn.Ox\\",\\"op\\":\\"PROPAGATE\\",\\"request\\":{\\"reqId\\":1505402110572745,\\"operation\\":{\\"dest\\":\\"V4SGRU86Z58d6TV7PBUe64\\",\\"type\\":\\"1\\"},\\"identifier\\":\\"V4SGRU86Z58d6TV7PBUe6f\\",\\"signature\\":\\"JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF\\"}}"],"op":"BATCH","signature":null}' 2017-09-14 15:15:24,502 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"messages":["po","{\\"senderClient\\":\\"v#ZljK34WWooA6%RgESn.Ox\\",\\"op\\":\\"PROPAGATE\\",\\"request\\":{\\"reqId\\":1505402110572745,\\"operation\\":{\\"dest\\":\\"V4SGRU86Z58d6TV7PBUe64\\",\\"type\\":\\"1\\"},\\"identifier\\":\\"V4SGRU86Z58d6TV7PBUe6f\\",\\"signature\\":\\"JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF\\"}}"],"op":"BATCH","signature":null}' to Node3 2017-09-14 15:15:24,503 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE","request":{"reqId":1505402110572745,"operation":{"dest":"V4SGRU86Z58d6TV7PBUe64","type":"1"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF"}}' to Node2 2017-09-14 15:15:24,503 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE","request":{"reqId":1505402110572745,"operation":{"dest":"V4SGRU86Z58d6TV7PBUe64","type":"1"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF"}}' to Node2 2017-09-14 15:15:24,503 | DEBUG | keep_in_touch.py ( 68) | conns | Node1's connections changed from {'Node2'} to {'Node3', 'Node2'} 2017-09-14 15:15:24,503 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node1 now connected to Node3 2017-09-14 15:15:24,503 | DEBUG | motor.py ( 34) | set_status | Node1 changing status from started_hungry to started 2017-09-14 15:15:24,503 | DEBUG | node.py ( 941) | checkInstances | Node1 choosing to start election on the basis of count 3 and nodes {'Node3', 'Node2'} 2017-09-14 15:15:24,503 | DEBUG | primary_selector.py ( 77) | get_msgs_for_lagged_nodes | Node1 has no ViewChangeDone message to send for view 0 2017-09-14 15:15:24,504 | DEBUG | node.py ( 902) | send_current_state_to_lagging_node | Node1 sending current state CURRENT_STATE{'primary': [], 'viewNo': 0} to lagged node Node3 2017-09-14 15:15:24,504 | DEBUG | node.py (2608) | send | Node1 sending message CURRENT_STATE{'primary': [], 'viewNo': 0} to 1 recipients: ['Node3'] 2017-09-14 15:15:24,504 | DEBUG | node.py (2608) | send | Node1 sending message LEDGER_STATUS{'txnSeqNo': 3, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None} to 1 recipients: ['Node3'] 2017-09-14 15:15:24,505 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 1 messages through listener 2017-09-14 15:15:24,506 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None} 2017-09-14 15:15:24,506 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'op': 'BATCH', 'signature': None}, 'Node2') 2017-09-14 15:15:24,506 | DEBUG | node.py (1272) | unpackNodeMsg | Node1 processing a batch BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":2}}'], 'signature': None} 2017-09-14 15:15:24,506 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,506 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'op': 'MESSAGE_REQUEST'}, 'Node2') 2017-09-14 15:15:24,507 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,507 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,507 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'op': 'MESSAGE_REQUEST'}, 'Node2') 2017-09-14 15:15:24,507 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,507 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None}, 'params': {'ledgerId': 2}} to 1 recipients: ['Node2'] 2017-09-14 15:15:24,508 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None}, 'params': {'ledgerId': 2}} to 1 recipients: ['Node2'] 2017-09-14 15:15:24,513 | DEBUG | batched.py ( 100) | flushOutBoxes | Node1 batching 2 msgs to Node3 into one transmission 2017-09-14 15:15:24,513 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"primary":[],"op":"CURRENT_STATE","viewNo":0}', b'{"txnSeqNo":3,"ppSeqNo":null,"op":"LEDGER_STATUS","viewNo":null,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ledgerId":0}']) 2017-09-14 15:15:24,513 | TRACE | batched.py ( 110) | flushOutBoxes | Node1 sending payload to Node3: b'{"messages":["{\\"primary\\":[],\\"op\\":\\"CURRENT_STATE\\",\\"viewNo\\":0}","{\\"txnSeqNo\\":3,\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null,\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"ledgerId\\":0}"],"op":"BATCH","signature":null}' 2017-09-14 15:15:24,513 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"messages":["{\\"primary\\":[],\\"op\\":\\"CURRENT_STATE\\",\\"viewNo\\":0}","{\\"txnSeqNo\\":3,\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null,\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"ledgerId\\":0}"],"op":"BATCH","signature":null}' to Node3 2017-09-14 15:15:24,513 | DEBUG | batched.py ( 100) | flushOutBoxes | Node1 batching 2 msgs to Node2 into one transmission 2017-09-14 15:15:24,513 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', b'{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}']) 2017-09-14 15:15:24,514 | TRACE | batched.py ( 110) | flushOutBoxes | Node1 sending payload to Node2: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}"],"op":"BATCH","signature":null}' 2017-09-14 15:15:24,514 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}"],"op":"BATCH","signature":null}' to Node2 2017-09-14 15:15:24,514 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 1 messages through listener 2017-09-14 15:15:24,515 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: BATCH{'messages': ['{"op":"CURRENT_STATE","primary":[],"viewNo":1}', '{"ledgerId":0,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","txnSeqNo":3,"viewNo":null}', '{"ledgerId":1,"op":"LEDGER_STATUS","ppSeqNo":3,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","txnSeqNo":8,"viewNo":0}'], 'signature': None} 2017-09-14 15:15:24,515 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'messages': ['{"op":"CURRENT_STATE","primary":[],"viewNo":1}', '{"ledgerId":0,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","txnSeqNo":3,"viewNo":null}', '{"ledgerId":1,"op":"LEDGER_STATUS","ppSeqNo":3,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","txnSeqNo":8,"viewNo":0}'], 'op': 'BATCH', 'signature': None}, 'Node2') 2017-09-14 15:15:24,515 | DEBUG | node.py (1272) | unpackNodeMsg | Node1 processing a batch BATCH{'messages': ['{"op":"CURRENT_STATE","primary":[],"viewNo":1}', '{"ledgerId":0,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","txnSeqNo":3,"viewNo":null}', '{"ledgerId":1,"op":"LEDGER_STATUS","ppSeqNo":3,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","txnSeqNo":8,"viewNo":0}'], 'signature': None} 2017-09-14 15:15:24,515 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: CURRENT_STATE{'primary': [], 'viewNo': 1} 2017-09-14 15:15:24,515 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'primary': [], 'op': 'CURRENT_STATE', 'viewNo': 1}, 'Node2') 2017-09-14 15:15:24,516 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox CURRENT_STATE{'primary': [], 'viewNo': 1} 2017-09-14 15:15:24,516 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: LEDGER_STATUS{'txnSeqNo': 3, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None} 2017-09-14 15:15:24,516 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'txnSeqNo': 3, 'ledgerId': 0, 'op': 'LEDGER_STATUS', 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'viewNo': None, 'ppSeqNo': None}, 'Node2') 2017-09-14 15:15:24,516 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox LEDGER_STATUS{'txnSeqNo': 3, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None} 2017-09-14 15:15:24,517 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ledgerId': 1, 'viewNo': 0} 2017-09-14 15:15:24,517 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'txnSeqNo': 8, 'ledgerId': 1, 'op': 'LEDGER_STATUS', 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'viewNo': 0, 'ppSeqNo': 3}, 'Node2') 2017-09-14 15:15:24,517 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ledgerId': 1, 'viewNo': 0} 2017-09-14 15:15:24,517 | DEBUG | node.py ( 907) | process_current_state_message | Node1 processing current state CURRENT_STATE{'primary': [], 'viewNo': 1} from Node2 2017-09-14 15:15:24,517 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 3, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None} from Node2 2017-09-14 15:15:24,517 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 3 2017-09-14 15:15:24,518 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 3 2017-09-14 15:15:24,518 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ledgerId': 1, 'viewNo': 0} from Node2 2017-09-14 15:15:24,518 | DEBUG | ledger_manager.py (1016) | stashLedgerStatus | Node1 stashing ledger status LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ledgerId': 1, 'viewNo': 0} from Node2 2017-09-14 15:15:24,530 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 2 messages through listener 2017-09-14 15:15:24,530 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: BATCH{'messages': ['{"op":"CURRENT_STATE","viewNo":0,"primary":[{"ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,5,"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0}]}', '{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","txnSeqNo":3,"viewNo":null,"op":"LEDGER_STATUS","ppSeqNo":null}', '{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","txnSeqNo":8,"viewNo":0,"op":"LEDGER_STATUS","ppSeqNo":3}', '{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","txnSeqNo":0,"viewNo":null,"op":"LEDGER_STATUS","ppSeqNo":null}'], 'signature': None} 2017-09-14 15:15:24,530 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'messages': ['{"op":"CURRENT_STATE","viewNo":0,"primary":[{"ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,5,"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0}]}', '{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","txnSeqNo":3,"viewNo":null,"op":"LEDGER_STATUS","ppSeqNo":null}', '{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","txnSeqNo":8,"viewNo":0,"op":"LEDGER_STATUS","ppSeqNo":3}', '{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","txnSeqNo":0,"viewNo":null,"op":"LEDGER_STATUS","ppSeqNo":null}'], 'op': 'BATCH', 'signature': None}, 'Node3') 2017-09-14 15:15:24,530 | DEBUG | node.py (1272) | unpackNodeMsg | Node1 processing a batch BATCH{'messages': ['{"op":"CURRENT_STATE","viewNo":0,"primary":[{"ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,5,"ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0}]}', '{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","txnSeqNo":3,"viewNo":null,"op":"LEDGER_STATUS","ppSeqNo":null}', '{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","txnSeqNo":8,"viewNo":0,"op":"LEDGER_STATUS","ppSeqNo":3}', '{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","txnSeqNo":0,"viewNo":null,"op":"LEDGER_STATUS","ppSeqNo":null}'], 'signature': None} 2017-09-14 15:15:24,530 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: CURRENT_STATE{'primary': [{'name': 'Node1', 'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0}], 'viewNo': 0} 2017-09-14 15:15:24,531 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'primary': [{'name': 'Node1', 'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0}], 'op': 'CURRENT_STATE', 'viewNo': 0}, 'Node3') 2017-09-14 15:15:24,531 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox CURRENT_STATE{'primary': [{'name': 'Node1', 'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0}], 'viewNo': 0} 2017-09-14 15:15:24,531 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: LEDGER_STATUS{'txnSeqNo': 3, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None} 2017-09-14 15:15:24,531 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'txnSeqNo': 3, 'ppSeqNo': None, 'op': 'LEDGER_STATUS', 'viewNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ledgerId': 0}, 'Node3') 2017-09-14 15:15:24,531 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox LEDGER_STATUS{'txnSeqNo': 3, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None} 2017-09-14 15:15:24,532 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ledgerId': 1, 'viewNo': 0} 2017-09-14 15:15:24,532 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'txnSeqNo': 8, 'ppSeqNo': 3, 'op': 'LEDGER_STATUS', 'viewNo': 0, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ledgerId': 1}, 'Node3') 2017-09-14 15:15:24,532 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ledgerId': 1, 'viewNo': 0} 2017-09-14 15:15:24,532 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: LEDGER_STATUS{'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None} 2017-09-14 15:15:24,532 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'txnSeqNo': 0, 'ppSeqNo': None, 'op': 'LEDGER_STATUS', 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2}, 'Node3') 2017-09-14 15:15:24,532 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox LEDGER_STATUS{'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None} 2017-09-14 15:15:24,532 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None} 2017-09-14 15:15:24,532 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'op': 'BATCH', 'signature': None}, 'Node2') 2017-09-14 15:15:24,533 | DEBUG | node.py (1272) | unpackNodeMsg | Node1 processing a batch BATCH{'messages': ['{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}', '{"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS","params":{"ledgerId":1}}'], 'signature': None} 2017-09-14 15:15:24,533 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,533 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'op': 'MESSAGE_REQUEST'}, 'Node2') 2017-09-14 15:15:24,533 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,533 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,533 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'op': 'MESSAGE_REQUEST'}, 'Node2') 2017-09-14 15:15:24,533 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,533 | DEBUG | node.py ( 907) | process_current_state_message | Node1 processing current state CURRENT_STATE{'primary': [{'name': 'Node1', 'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0}], 'viewNo': 0} from Node3 2017-09-14 15:15:24,534 | DEBUG | node.py (1205) | sendToElector | Node1 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0}, 'Node3') 2017-09-14 15:15:24,534 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 3, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None} from Node3 2017-09-14 15:15:24,534 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 3 2017-09-14 15:15:24,534 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 3 2017-09-14 15:15:24,534 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node3', 'Node2'} that its ledger of type 0 is latest 2017-09-14 15:15:24,534 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node1 found from ledger status LEDGER_STATUS{'txnSeqNo': 3, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None} that it does not need catchup 2017-09-14 15:15:24,535 | DEBUG | node.py (1479) | preLedgerCatchUp | Node1 going to process any ordered requests before starting catchup. 2017-09-14 15:15:24,535 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node1:0 going to remove 0 Ordered messages from outbox 2017-09-14 15:15:24,535 | DEBUG | node.py (1888) | force_process_ordered | Node1 processed 0 Ordered batches for instance 0 before starting catch up 2017-09-14 15:15:24,535 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node1 processed 0 stashed ordered requests 2017-09-14 15:15:24,535 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-09-14 15:15:24,535 | INFO | node.py (1489) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 0 2017-09-14 15:15:24,535 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 3 2017-09-14 15:15:24,535 | DEBUG | node.py ( 941) | checkInstances | Node1 choosing to start election on the basis of count 3 and nodes {'Node3', 'Node2'} 2017-09-14 15:15:24,536 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to all recipients: ['Node3', 'Node2'] 2017-09-14 15:15:24,536 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node1 asking Node1 for ledger status of ledger 2 2017-09-14 15:15:24,536 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node2'] 2017-09-14 15:15:24,536 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node1 asking Node2 for ledger status of ledger 2 2017-09-14 15:15:24,536 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-09-14 15:15:24,536 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node1 asking Node3 for ledger status of ledger 2 2017-09-14 15:15:24,536 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node1 going to process 0 stashed ledger statuses for ledger 2 2017-09-14 15:15:24,536 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 0, caught up 0 in total 2017-09-14 15:15:24,537 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ledgerId': 1, 'viewNo': 0} from Node3 2017-09-14 15:15:24,537 | DEBUG | ledger_manager.py (1016) | stashLedgerStatus | Node1 stashing ledger status LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ledgerId': 1, 'viewNo': 0} from Node3 2017-09-14 15:15:24,537 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None} from Node3 2017-09-14 15:15:24,537 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:15:24,537 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:15:24,538 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None}, 'params': {'ledgerId': 1}} to 1 recipients: ['Node2'] 2017-09-14 15:15:24,538 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None}, 'params': {'ledgerId': 1}} to 1 recipients: ['Node2'] 2017-09-14 15:15:24,538 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node1's primary selector started processing of ViewChangeDone msg from Node3 : VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0} 2017-09-14 15:15:24,538 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node1 got view change quorum (1 >= 1) 2017-09-14 15:15:24,538 | DEBUG | primary_selector.py ( 209) | has_view_change_from_primary | Node1 has not received ViewChangeDone from the next primary Node1 (viewNo: 0, totalNodes: 3) 2017-09-14 15:15:24,539 | DEBUG | primary_selector.py ( 266) | _startSelection | Node1 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-09-14 15:15:24,539 | DEBUG | batched.py ( 100) | flushOutBoxes | Node1 batching 2 msgs to Node3 into one transmission 2017-09-14 15:15:24,539 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}']) 2017-09-14 15:15:24,539 | TRACE | batched.py ( 110) | flushOutBoxes | Node1 sending payload to Node3: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}"],"op":"BATCH","signature":null}' 2017-09-14 15:15:24,539 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}"],"op":"BATCH","signature":null}' to Node3 2017-09-14 15:15:24,544 | DEBUG | batched.py ( 100) | flushOutBoxes | Node1 batching 4 msgs to Node2 into one transmission 2017-09-14 15:15:24,544 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', b'{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":null,"txnSeqNo":8,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', b'{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":null,"txnSeqNo":8,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}']) 2017-09-14 15:15:24,544 | TRACE | batched.py ( 110) | flushOutBoxes | Node1 sending payload to Node2: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":8,\\"viewNo\\":null},\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":8,\\"viewNo\\":null},\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}"],"op":"BATCH","signature":null}' 2017-09-14 15:15:24,544 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":8,\\"viewNo\\":null},\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":8,\\"viewNo\\":null},\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}"],"op":"BATCH","signature":null}' to Node2 2017-09-14 15:15:24,557 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 1 messages through listener 2017-09-14 15:15:24,557 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}'], 'signature': None} 2017-09-14 15:15:24,558 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}'], 'op': 'BATCH', 'signature': None}, 'Node3') 2017-09-14 15:15:24,558 | DEBUG | node.py (1272) | unpackNodeMsg | Node1 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}'], 'signature': None} 2017-09-14 15:15:24,558 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,558 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'params': {'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE'}, 'Node3') 2017-09-14 15:15:24,558 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,558 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,559 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'params': {'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE'}, 'Node3') 2017-09-14 15:15:24,559 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,559 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None} from Node3 2017-09-14 15:15:24,559 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:15:24,559 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:15:24,559 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None} from Node3 2017-09-14 15:15:24,560 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:15:24,560 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:15:24,560 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1C got 1 messages through listener 2017-09-14 15:15:24,560 | DEBUG | zstack.py ( 663) | handlePingPong | Node1C got ping from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:15:24,560 | TRACE | zstack.py ( 754) | transmitThroughListener | Node1C transmitting b'po' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:15:24,560 | DEBUG | zstack.py ( 645) | sendPingPong | Node1C ponged b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:15:24,567 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1C got 1 messages through listener 2017-09-14 15:15:24,568 | TRACE | node.py (1388) | validateClientMsg | Node1C received CLIENT message: LEDGER_STATUS{'txnSeqNo': 4, 'merkleRoot': 'CWoRZ62yMB12JyQEUAYU73CNBqJXWaKUafN9yiR9Dre2', 'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None} 2017-09-14 15:15:24,568 | DEBUG | node.py (1434) | processClientInBox | Node1C processing b'v#ZljK34WWooA6%RgESn.Ox' request LEDGER_STATUS{'txnSeqNo': 4, 'merkleRoot': 'CWoRZ62yMB12JyQEUAYU73CNBqJXWaKUafN9yiR9Dre2', 'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None} 2017-09-14 15:15:24,568 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 4, 'merkleRoot': 'CWoRZ62yMB12JyQEUAYU73CNBqJXWaKUafN9yiR9Dre2', 'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:15:24,568 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 4 2017-09-14 15:15:24,568 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 4 2017-09-14 15:15:24,568 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 4 2017-09-14 15:15:24,569 | TRACE | zstack.py ( 754) | transmitThroughListener | Node1C transmitting b'{"txnSeqNo":3,"ppSeqNo":null,"op":"LEDGER_STATUS","viewNo":null,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ledgerId":0}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:15:24,580 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 1 messages through listener 2017-09-14 15:15:24,581 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: BATCH{'messages': ['{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,8,"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node2","viewNo":1}'], 'signature': None} 2017-09-14 15:15:24,581 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'messages': ['{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,8,"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node2","viewNo":1}'], 'op': 'BATCH', 'signature': None}, 'Node2') 2017-09-14 15:15:24,581 | DEBUG | node.py (1272) | unpackNodeMsg | Node1 processing a batch BATCH{'messages': ['{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,8,"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node2","viewNo":1}'], 'signature': None} 2017-09-14 15:15:24,581 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,582 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'params': {'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE'}, 'Node2') 2017-09-14 15:15:24,582 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,582 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,582 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'params': {'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE'}, 'Node2') 2017-09-14 15:15:24,582 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,582 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 1} 2017-09-14 15:15:24,583 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'name': 'Node2', 'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'op': 'VIEW_CHANGE_DONE', 'viewNo': 1}, 'Node2') 2017-09-14 15:15:24,583 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 1} 2017-09-14 15:15:24,583 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None} from Node2 2017-09-14 15:15:24,583 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:15:24,583 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:15:24,583 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node3', 'Node2'} that its ledger of type 2 is latest 2017-09-14 15:15:24,583 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node1 found from ledger status LEDGER_STATUS{'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None} that it does not need catchup 2017-09-14 15:15:24,583 | DEBUG | node.py (1479) | preLedgerCatchUp | Node1 going to process any ordered requests before starting catchup. 2017-09-14 15:15:24,584 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node1:0 going to remove 0 Ordered messages from outbox 2017-09-14 15:15:24,584 | DEBUG | node.py (1888) | force_process_ordered | Node1 processed 0 Ordered batches for instance 0 before starting catch up 2017-09-14 15:15:24,584 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node1 processed 0 stashed ordered requests 2017-09-14 15:15:24,584 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-09-14 15:15:24,584 | INFO | node.py (1489) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 2 2017-09-14 15:15:24,584 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:15:24,585 | DEBUG | pool_config.py ( 35) | processLedger | processing config ledger for any POOL_CONFIGs 2017-09-14 15:15:24,585 | DEBUG | upgrader.py ( 208) | processLedger | Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv processing config ledger for any upgrades 2017-09-14 15:15:24,585 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to all recipients: ['Node3', 'Node2'] 2017-09-14 15:15:24,585 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node1 asking Node1 for ledger status of ledger 1 2017-09-14 15:15:24,585 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node2'] 2017-09-14 15:15:24,585 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node1 asking Node2 for ledger status of ledger 1 2017-09-14 15:15:24,585 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-09-14 15:15:24,585 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node1 asking Node3 for ledger status of ledger 1 2017-09-14 15:15:24,585 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node1 going to process 2 stashed ledger statuses for ledger 1 2017-09-14 15:15:24,586 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ledgerId': 1, 'viewNo': 0} from Node2 2017-09-14 15:15:24,586 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 8 with 8 2017-09-14 15:15:24,586 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 8 with 8 2017-09-14 15:15:24,586 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ledgerId': 1, 'viewNo': 0} from Node3 2017-09-14 15:15:24,586 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 8 with 8 2017-09-14 15:15:24,586 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 8 with 8 2017-09-14 15:15:24,586 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node3', 'Node2'} that its ledger of type 1 is latest 2017-09-14 15:15:24,586 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node1 found from ledger status LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ledgerId': 1, 'viewNo': 0} that it does not need catchup 2017-09-14 15:15:24,587 | DEBUG | node.py (1479) | preLedgerCatchUp | Node1 going to process any ordered requests before starting catchup. 2017-09-14 15:15:24,587 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node1:0 going to remove 0 Ordered messages from outbox 2017-09-14 15:15:24,587 | DEBUG | node.py (1888) | force_process_ordered | Node1 processed 0 Ordered batches for instance 0 before starting catch up 2017-09-14 15:15:24,587 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node1 processed 0 stashed ordered requests 2017-09-14 15:15:24,587 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-09-14 15:15:24,587 | INFO | node.py (1489) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 1 2017-09-14 15:15:24,587 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 8 with 8 2017-09-14 15:15:24,587 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 1, caught up 0 in total 2017-09-14 15:15:24,587 | DEBUG | node.py (1609) | num_txns_caught_up_in_last_catchup | Node1 caught up to 0 txns in the last catchup 2017-09-14 15:15:24,587 | DEBUG | replica.py ( 323) | last_ordered_3pc | Node1:0 set last ordered as (0, 3) 2017-09-14 15:15:24,588 | DEBUG | replica.py (2234) | _remove_till_caught_up_3pc | Node1:0 going to remove messages for 0 3PC keys 2017-09-14 15:15:24,588 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node1:0 going to remove 0 Ordered messages from outbox 2017-09-14 15:15:24,588 | INFO | node.py (1525) | allLedgersCaughtUp | CATCH-UP: Node1 caught up till (0, 3) 2017-09-14 15:15:24,588 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node1 processed 0 stashed ordered requests 2017-09-14 15:15:24,588 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-09-14 15:15:24,588 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node1 got view change quorum (1 >= 1) 2017-09-14 15:15:24,589 | DEBUG | primary_selector.py ( 238) | has_sufficient_same_view_change_done_messages | Node1 found acceptable primary Node1 and ledger info ((0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 5, 'ArKCamA7sdNuerVaLV9DP3CUEsun5goNeN8sDcEksGfj'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')) 2017-09-14 15:15:24,590 | DEBUG | node.py (1549) | is_catchup_needed | Node1 is caught up for the current view 0 2017-09-14 15:15:24,590 | INFO | node.py (1537) | allLedgersCaughtUp | CATCH-UP: Node1 does not need any more catchups 2017-09-14 15:15:24,590 | DEBUG | primary_decider.py ( 134) | send | Node1's elector sending VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [(0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'viewNo': 0} 2017-09-14 15:15:24,590 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node1 got view change quorum (2 >= 1) 2017-09-14 15:15:24,590 | DEBUG | primary_selector.py ( 215) | has_view_change_from_primary | Node1 received ViewChangeDone from primary Node1 2017-09-14 15:15:24,591 | DEBUG | primary_selector.py ( 281) | _startSelection | Node1 starting selection 2017-09-14 15:15:24,591 | DISPLAY | primary_selector.py ( 291) | _startSelection | PRIMARY SELECTION: Node1:0 selected primary Node1:0 for instance 0 (view 0) 2017-09-14 15:15:24,592 | INFO | node.py ( 510) | start_participating | Node1 started participating 2017-09-14 15:15:24,592 | DEBUG | replica.py ( 408) | primaryName | Node1:0 setting primaryName for view no 0 to: Node1:0 2017-09-14 15:15:24,592 | DEBUG | replica.py (1694) | _gc | Node1:0 cleaning up till (0, 3) 2017-09-14 15:15:24,592 | DEBUG | replica.py (1709) | _gc | Node1:0 found 0 3-phase keys to clean 2017-09-14 15:15:24,592 | DEBUG | replica.py (1711) | _gc | Node1:0 found 0 request keys to clean 2017-09-14 15:15:24,592 | DEBUG | replica.py ( 313) | h | Node1:0 set watermarks as 0 300 2017-09-14 15:15:24,593 | DISPLAY | primary_selector.py ( 315) | _startSelection | VIEW CHANGE: Node1:0 declares view change 0 as completed for instance 0, new primary is Node1:0, ledger info is [(0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-09-14 15:15:24,593 | INFO | upgrader.py ( 150) | should_notify_about_upgrade_result | Node's 'Node1' last upgrade txn is None 2017-09-14 15:15:24,593 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 2, caught up 0 in total 2017-09-14 15:15:24,593 | DEBUG | node.py (1609) | num_txns_caught_up_in_last_catchup | Node1 caught up to 0 txns in the last catchup 2017-09-14 15:15:24,593 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node1 processed 0 stashed ordered requests 2017-09-14 15:15:24,593 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-09-14 15:15:24,593 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node1 got view change quorum (2 >= 1) 2017-09-14 15:15:24,594 | DEBUG | node.py (1549) | is_catchup_needed | Node1 is caught up for the current view 0 2017-09-14 15:15:24,594 | INFO | node.py (1537) | allLedgersCaughtUp | CATCH-UP: Node1 does not need any more catchups 2017-09-14 15:15:24,594 | DEBUG | primary_selector.py ( 281) | _startSelection | Node1 starting selection 2017-09-14 15:15:24,594 | DEBUG | primary_selector.py ( 284) | _startSelection | Node1:0 already has a primary 2017-09-14 15:15:24,594 | INFO | node.py (1624) | no_more_catchups_needed | Node1 starting to participate since catchup is done, primaries are selected but mode was not set to participating 2017-09-14 15:15:24,594 | INFO | node.py ( 510) | start_participating | Node1 started participating 2017-09-14 15:15:24,595 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None} from Node2 2017-09-14 15:15:24,595 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:15:24,595 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:15:24,596 | DEBUG | node.py (1170) | msgHasAcceptableViewNo | Node1 stashing a message for a future view: VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 1} 2017-09-14 15:15:24,596 | INFO | node.py (1967) | _start_view_change_if_possible | VIEW CHANGE: Node1 starting view change for 1 after 1 view change indications from other nodes 2017-09-14 15:15:24,596 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action _check_view_change_completed with id 6 to run in 60 seconds 2017-09-14 15:15:24,596 | DEBUG | replica.py ( 455) | on_view_change_start | Node1:0 setting last prepared for master to None 2017-09-14 15:15:24,596 | DEBUG | node.py (2146) | startViewChange | Node1 resetting monitor stats after view change 2017-09-14 15:15:24,596 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-09-14 15:15:24,597 | DEBUG | node.py (1205) | sendToElector | Node1 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 1}, 'Node2') 2017-09-14 15:15:24,597 | DEBUG | node.py ( 996) | processStashedMsgsForView | Node1 processed 1 stashed msgs for view no 1 2017-09-14 15:15:24,597 | DEBUG | replica.py ( 408) | primaryName | Node1:0 setting primaryName for view no 1 to: None 2017-09-14 15:15:24,597 | INFO | node.py (2159) | startViewChange | VIEW CHANGE: Node1 changed to view 1, will start catchup now 2017-09-14 15:15:24,598 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node1:0 going to remove 0 Ordered messages from outbox 2017-09-14 15:15:24,598 | DEBUG | node.py (1888) | force_process_ordered | Node1 processed 0 Ordered batches for instance 0 before starting catch up 2017-09-14 15:15:24,598 | DEBUG | node.py (2192) | start_catchup | Node1 reverted 0 batches before starting catch up 2017-09-14 15:15:24,598 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to all recipients: ['Node3', 'Node2'] 2017-09-14 15:15:24,598 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node1 asking Node1 for ledger status of ledger 0 2017-09-14 15:15:24,598 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node2'] 2017-09-14 15:15:24,599 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node1 asking Node2 for ledger status of ledger 0 2017-09-14 15:15:24,599 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node3'] 2017-09-14 15:15:24,599 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node1 asking Node3 for ledger status of ledger 0 2017-09-14 15:15:24,599 | DEBUG | node.py (2608) | send | Node1 sending message VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [(0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'viewNo': 0} to all recipients: ['Node3', 'Node2'] 2017-09-14 15:15:24,599 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node1's primary selector started processing of ViewChangeDone msg from Node2 : VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 1} 2017-09-14 15:15:24,600 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node1 got view change quorum (1 >= 1) 2017-09-14 15:15:24,601 | DEBUG | primary_selector.py ( 215) | has_view_change_from_primary | Node1 received ViewChangeDone from primary Node2 2017-09-14 15:15:24,601 | DEBUG | primary_selector.py ( 238) | has_sufficient_same_view_change_done_messages | Node1 found acceptable primary Node2 and ledger info ((0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')) 2017-09-14 15:15:24,601 | DEBUG | primary_selector.py ( 271) | _startSelection | Node1 cannot start primary selection since mode is 100 2017-09-14 15:15:24,601 | DEBUG | batched.py ( 100) | flushOutBoxes | Node1 batching 5 msgs to Node3 into one transmission 2017-09-14 15:15:24,601 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"op":"MESSAGE_REQUEST"}', b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"op":"MESSAGE_REQUEST"}', b'{"name":"Node1","ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,8,"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","viewNo":0}']) 2017-09-14 15:15:24,601 | TRACE | batched.py ( 110) | flushOutBoxes | Node1 sending payload to Node3: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"name\\":\\"Node1\\",\\"ledgerInfo\\":[[0,3,\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\"],[1,8,\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"op\\":\\"VIEW_CHANGE_DONE\\",\\"viewNo\\":0}"],"op":"BATCH","signature":null}' 2017-09-14 15:15:24,601 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"name\\":\\"Node1\\",\\"ledgerInfo\\":[[0,3,\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\"],[1,8,\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"op\\":\\"VIEW_CHANGE_DONE\\",\\"viewNo\\":0}"],"op":"BATCH","signature":null}' to Node3 2017-09-14 15:15:24,602 | DEBUG | batched.py ( 100) | flushOutBoxes | Node1 batching 5 msgs to Node2 into one transmission 2017-09-14 15:15:24,602 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"op":"MESSAGE_REQUEST"}', b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"op":"MESSAGE_REQUEST"}', b'{"name":"Node1","ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,8,"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","viewNo":0}']) 2017-09-14 15:15:24,602 | TRACE | batched.py ( 110) | flushOutBoxes | Node1 sending payload to Node2: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"name\\":\\"Node1\\",\\"ledgerInfo\\":[[0,3,\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\"],[1,8,\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"op\\":\\"VIEW_CHANGE_DONE\\",\\"viewNo\\":0}"],"op":"BATCH","signature":null}' 2017-09-14 15:15:24,602 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":0},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"name\\":\\"Node1\\",\\"ledgerInfo\\":[[0,3,\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\"],[1,8,\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"op\\":\\"VIEW_CHANGE_DONE\\",\\"viewNo\\":0}"],"op":"BATCH","signature":null}' to Node2 2017-09-14 15:15:24,603 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 1 messages through listener 2017-09-14 15:15:24,603 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":0}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":0}}'], 'signature': None} 2017-09-14 15:15:24,603 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":0}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":0}}'], 'op': 'BATCH', 'signature': None}, 'Node3') 2017-09-14 15:15:24,603 | DEBUG | node.py (1272) | unpackNodeMsg | Node1 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":0}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":0}}'], 'signature': None} 2017-09-14 15:15:24,604 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-09-14 15:15:24,604 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}, 'op': 'MESSAGE_REQUEST'}, 'Node3') 2017-09-14 15:15:24,604 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-09-14 15:15:24,604 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-09-14 15:15:24,604 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}, 'op': 'MESSAGE_REQUEST'}, 'Node3') 2017-09-14 15:15:24,604 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-09-14 15:15:24,604 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'txnSeqNo': 3, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None}, 'params': {'ledgerId': 0}} to 1 recipients: ['Node3'] 2017-09-14 15:15:24,605 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'txnSeqNo': 3, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None}, 'params': {'ledgerId': 0}} to 1 recipients: ['Node3'] 2017-09-14 15:15:24,605 | DEBUG | batched.py ( 100) | flushOutBoxes | Node1 batching 2 msgs to Node3 into one transmission 2017-09-14 15:15:24,605 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ppSeqNo":null,"txnSeqNo":3,"viewNo":null},"params":{"ledgerId":0},"op":"MESSAGE_RESPONSE"}', b'{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ppSeqNo":null,"txnSeqNo":3,"viewNo":null},"params":{"ledgerId":0},"op":"MESSAGE_RESPONSE"}']) 2017-09-14 15:15:24,605 | TRACE | batched.py ( 110) | flushOutBoxes | Node1 sending payload to Node3: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":3,\\"viewNo\\":null},\\"params\\":{\\"ledgerId\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":3,\\"viewNo\\":null},\\"params\\":{\\"ledgerId\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\"}"],"op":"BATCH","signature":null}' 2017-09-14 15:15:24,606 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":3,\\"viewNo\\":null},\\"params\\":{\\"ledgerId\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":3,\\"viewNo\\":null},\\"params\\":{\\"ledgerId\\":0},\\"op\\":\\"MESSAGE_RESPONSE\\"}"],"op":"BATCH","signature":null}' to Node3 2017-09-14 15:15:24,619 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 1 messages through listener 2017-09-14 15:15:24,620 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: BATCH{'messages': ['{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ppSeqNo":null,"txnSeqNo":3,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ppSeqNo":null,"txnSeqNo":3,"viewNo":null}}'], 'signature': None} 2017-09-14 15:15:24,620 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'messages': ['{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ppSeqNo":null,"txnSeqNo":3,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ppSeqNo":null,"txnSeqNo":3,"viewNo":null}}'], 'op': 'BATCH', 'signature': None}, 'Node2') 2017-09-14 15:15:24,620 | DEBUG | node.py (1272) | unpackNodeMsg | Node1 processing a batch BATCH{'messages': ['{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ppSeqNo":null,"txnSeqNo":3,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ppSeqNo":null,"txnSeqNo":3,"viewNo":null}}'], 'signature': None} 2017-09-14 15:15:24,620 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 8, 'viewNo': 0, 'ppSeqNo': 3, 'ledgerId': 1, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'}, 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,620 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 8, 'viewNo': 0, 'ppSeqNo': 3, 'ledgerId': 1, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'}, 'params': {'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE'}, 'Node2') 2017-09-14 15:15:24,620 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 8, 'viewNo': 0, 'ppSeqNo': 3, 'ledgerId': 1, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'}, 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,621 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 8, 'viewNo': 0, 'ppSeqNo': 3, 'ledgerId': 1, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'}, 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,621 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 8, 'viewNo': 0, 'ppSeqNo': 3, 'ledgerId': 1, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'}, 'params': {'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE'}, 'Node2') 2017-09-14 15:15:24,621 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 8, 'viewNo': 0, 'ppSeqNo': 3, 'ledgerId': 1, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'}, 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,621 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 3, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 0, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'}, 'params': {'ledgerId': 0}} 2017-09-14 15:15:24,621 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 3, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 0, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'}, 'params': {'ledgerId': 0}, 'op': 'MESSAGE_RESPONSE'}, 'Node2') 2017-09-14 15:15:24,621 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 3, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 0, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'}, 'params': {'ledgerId': 0}} 2017-09-14 15:15:24,621 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 3, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 0, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'}, 'params': {'ledgerId': 0}} 2017-09-14 15:15:24,621 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 3, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 0, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'}, 'params': {'ledgerId': 0}, 'op': 'MESSAGE_RESPONSE'}, 'Node2') 2017-09-14 15:15:24,622 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 3, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 0, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'}, 'params': {'ledgerId': 0}} 2017-09-14 15:15:24,622 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ledgerId': 1, 'viewNo': 0} from Node2 2017-09-14 15:15:24,622 | DEBUG | ledger_manager.py (1016) | stashLedgerStatus | Node1 stashing ledger status LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ledgerId': 1, 'viewNo': 0} from Node2 2017-09-14 15:15:24,622 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ledgerId': 1, 'viewNo': 0} from Node2 2017-09-14 15:15:24,622 | DEBUG | ledger_manager.py (1016) | stashLedgerStatus | Node1 stashing ledger status LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ledgerId': 1, 'viewNo': 0} from Node2 2017-09-14 15:15:24,623 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 3, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None} from Node2 2017-09-14 15:15:24,623 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 3 2017-09-14 15:15:24,623 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 3 2017-09-14 15:15:24,623 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 3, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None} from Node2 2017-09-14 15:15:24,623 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 3 2017-09-14 15:15:24,624 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 3 2017-09-14 15:15:24,624 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 2 messages through listener 2017-09-14 15:15:24,625 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ppSeqNo":null,"txnSeqNo":3,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ppSeqNo":null,"txnSeqNo":3,"viewNo":null}}'], 'signature': None} 2017-09-14 15:15:24,625 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ppSeqNo":null,"txnSeqNo":3,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ppSeqNo":null,"txnSeqNo":3,"viewNo":null}}'], 'op': 'BATCH', 'signature': None}, 'Node3') 2017-09-14 15:15:24,625 | DEBUG | node.py (1272) | unpackNodeMsg | Node1 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ppSeqNo":null,"txnSeqNo":3,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ppSeqNo":null,"txnSeqNo":3,"viewNo":null}}'], 'signature': None} 2017-09-14 15:15:24,625 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 8, 'viewNo': 0, 'ppSeqNo': 3, 'ledgerId': 1, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'}, 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,625 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 8, 'viewNo': 0, 'ppSeqNo': 3, 'ledgerId': 1, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'}, 'params': {'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE'}, 'Node3') 2017-09-14 15:15:24,626 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 8, 'viewNo': 0, 'ppSeqNo': 3, 'ledgerId': 1, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'}, 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,626 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 8, 'viewNo': 0, 'ppSeqNo': 3, 'ledgerId': 1, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'}, 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,626 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 8, 'viewNo': 0, 'ppSeqNo': 3, 'ledgerId': 1, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'}, 'params': {'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE'}, 'Node3') 2017-09-14 15:15:24,626 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 8, 'viewNo': 0, 'ppSeqNo': 3, 'ledgerId': 1, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'}, 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,626 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 3, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 0, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'}, 'params': {'ledgerId': 0}} 2017-09-14 15:15:24,626 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 3, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 0, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'}, 'params': {'ledgerId': 0}, 'op': 'MESSAGE_RESPONSE'}, 'Node3') 2017-09-14 15:15:24,627 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 3, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 0, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'}, 'params': {'ledgerId': 0}} 2017-09-14 15:15:24,627 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 3, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 0, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'}, 'params': {'ledgerId': 0}} 2017-09-14 15:15:24,627 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 3, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 0, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'}, 'params': {'ledgerId': 0}, 'op': 'MESSAGE_RESPONSE'}, 'Node3') 2017-09-14 15:15:24,627 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 3, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 0, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'}, 'params': {'ledgerId': 0}} 2017-09-14 15:15:24,627 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}'], 'signature': None} 2017-09-14 15:15:24,627 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}'], 'op': 'BATCH', 'signature': None}, 'Node3') 2017-09-14 15:15:24,627 | DEBUG | node.py (1272) | unpackNodeMsg | Node1 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}'], 'signature': None} 2017-09-14 15:15:24,628 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,628 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'op': 'MESSAGE_REQUEST'}, 'Node3') 2017-09-14 15:15:24,628 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,628 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,628 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'op': 'MESSAGE_REQUEST'}, 'Node3') 2017-09-14 15:15:24,628 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,628 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ledgerId': 1, 'viewNo': 0} from Node3 2017-09-14 15:15:24,629 | DEBUG | ledger_manager.py (1016) | stashLedgerStatus | Node1 stashing ledger status LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ledgerId': 1, 'viewNo': 0} from Node3 2017-09-14 15:15:24,629 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ledgerId': 1, 'viewNo': 0} from Node3 2017-09-14 15:15:24,629 | DEBUG | ledger_manager.py (1016) | stashLedgerStatus | Node1 stashing ledger status LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ledgerId': 1, 'viewNo': 0} from Node3 2017-09-14 15:15:24,629 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 3, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None} from Node3 2017-09-14 15:15:24,630 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 3 2017-09-14 15:15:24,630 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 3 2017-09-14 15:15:24,630 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node3', 'Node2'} that its ledger of type 0 is latest 2017-09-14 15:15:24,630 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node1 found from ledger status LEDGER_STATUS{'txnSeqNo': 3, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None} that it does not need catchup 2017-09-14 15:15:24,630 | DEBUG | node.py (1479) | preLedgerCatchUp | Node1 going to process any ordered requests before starting catchup. 2017-09-14 15:15:24,630 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node1:0 going to remove 0 Ordered messages from outbox 2017-09-14 15:15:24,630 | DEBUG | node.py (1888) | force_process_ordered | Node1 processed 0 Ordered batches for instance 0 before starting catch up 2017-09-14 15:15:24,630 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node1 processed 0 stashed ordered requests 2017-09-14 15:15:24,630 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-09-14 15:15:24,631 | INFO | node.py (1489) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 0 2017-09-14 15:15:24,631 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 3 2017-09-14 15:15:24,631 | DEBUG | node.py ( 941) | checkInstances | Node1 choosing to start election on the basis of count 3 and nodes {'Node3', 'Node2'} 2017-09-14 15:15:24,631 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to all recipients: ['Node3', 'Node2'] 2017-09-14 15:15:24,631 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node1 asking Node1 for ledger status of ledger 2 2017-09-14 15:15:24,631 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node2'] 2017-09-14 15:15:24,631 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node1 asking Node2 for ledger status of ledger 2 2017-09-14 15:15:24,632 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-09-14 15:15:24,632 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node1 asking Node3 for ledger status of ledger 2 2017-09-14 15:15:24,632 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node1 going to process 0 stashed ledger statuses for ledger 2 2017-09-14 15:15:24,632 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 0, caught up 0 in total 2017-09-14 15:15:24,632 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 3, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None} from Node3 2017-09-14 15:15:24,632 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 3 2017-09-14 15:15:24,632 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 3 2017-09-14 15:15:24,633 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None}, 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-09-14 15:15:24,633 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None}, 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-09-14 15:15:24,633 | DEBUG | batched.py ( 100) | flushOutBoxes | Node1 batching 4 msgs to Node3 into one transmission 2017-09-14 15:15:24,633 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', b'{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', b'{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}']) 2017-09-14 15:15:24,634 | TRACE | batched.py ( 110) | flushOutBoxes | Node1 sending payload to Node3: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}"],"op":"BATCH","signature":null}' 2017-09-14 15:15:24,634 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}"],"op":"BATCH","signature":null}' to Node3 2017-09-14 15:15:24,634 | DEBUG | batched.py ( 100) | flushOutBoxes | Node1 batching 2 msgs to Node2 into one transmission 2017-09-14 15:15:24,634 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}']) 2017-09-14 15:15:24,634 | TRACE | batched.py ( 110) | flushOutBoxes | Node1 sending payload to Node2: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}"],"op":"BATCH","signature":null}' 2017-09-14 15:15:24,634 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}"],"op":"BATCH","signature":null}' to Node2 2017-09-14 15:15:24,646 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 1 messages through listener 2017-09-14 15:15:24,647 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: BATCH{'messages': ['{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}'], 'signature': None} 2017-09-14 15:15:24,647 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'messages': ['{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}'], 'op': 'BATCH', 'signature': None}, 'Node2') 2017-09-14 15:15:24,647 | DEBUG | node.py (1272) | unpackNodeMsg | Node1 processing a batch BATCH{'messages': ['{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}'], 'signature': None} 2017-09-14 15:15:24,647 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,647 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'params': {'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE'}, 'Node2') 2017-09-14 15:15:24,647 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,647 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,648 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'params': {'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE'}, 'Node2') 2017-09-14 15:15:24,648 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,648 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None} from Node2 2017-09-14 15:15:24,648 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:15:24,648 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:15:24,648 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None} from Node2 2017-09-14 15:15:24,649 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:15:24,649 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:15:24,649 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 1 messages through listener 2017-09-14 15:15:24,650 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}'], 'signature': None} 2017-09-14 15:15:24,650 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}'], 'op': 'BATCH', 'signature': None}, 'Node3') 2017-09-14 15:15:24,650 | DEBUG | node.py (1272) | unpackNodeMsg | Node1 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}'], 'signature': None} 2017-09-14 15:15:24,650 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,650 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'params': {'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE'}, 'Node3') 2017-09-14 15:15:24,650 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,651 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,653 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'params': {'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE'}, 'Node3') 2017-09-14 15:15:24,653 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'params': {'ledgerId': 2}} 2017-09-14 15:15:24,653 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,653 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'op': 'MESSAGE_REQUEST'}, 'Node3') 2017-09-14 15:15:24,653 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,653 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,654 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'op': 'MESSAGE_REQUEST'}, 'Node3') 2017-09-14 15:15:24,654 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,654 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None} from Node3 2017-09-14 15:15:24,654 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:15:24,654 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:15:24,654 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node3', 'Node2'} that its ledger of type 2 is latest 2017-09-14 15:15:24,654 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node1 found from ledger status LEDGER_STATUS{'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None} that it does not need catchup 2017-09-14 15:15:24,655 | DEBUG | node.py (1479) | preLedgerCatchUp | Node1 going to process any ordered requests before starting catchup. 2017-09-14 15:15:24,655 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node1:0 going to remove 0 Ordered messages from outbox 2017-09-14 15:15:24,655 | DEBUG | node.py (1888) | force_process_ordered | Node1 processed 0 Ordered batches for instance 0 before starting catch up 2017-09-14 15:15:24,655 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node1 processed 0 stashed ordered requests 2017-09-14 15:15:24,655 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-09-14 15:15:24,655 | INFO | node.py (1489) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 2 2017-09-14 15:15:24,655 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:15:24,655 | DEBUG | pool_config.py ( 35) | processLedger | processing config ledger for any POOL_CONFIGs 2017-09-14 15:15:24,655 | DEBUG | upgrader.py ( 208) | processLedger | Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv processing config ledger for any upgrades 2017-09-14 15:15:24,656 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to all recipients: ['Node3', 'Node2'] 2017-09-14 15:15:24,656 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node1 asking Node1 for ledger status of ledger 1 2017-09-14 15:15:24,656 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node2'] 2017-09-14 15:15:24,656 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node1 asking Node2 for ledger status of ledger 1 2017-09-14 15:15:24,656 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-09-14 15:15:24,656 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node1 asking Node3 for ledger status of ledger 1 2017-09-14 15:15:24,656 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node1 going to process 4 stashed ledger statuses for ledger 1 2017-09-14 15:15:24,658 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ledgerId': 1, 'viewNo': 0} from Node2 2017-09-14 15:15:24,658 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 8 with 8 2017-09-14 15:15:24,658 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 8 with 8 2017-09-14 15:15:24,658 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ledgerId': 1, 'viewNo': 0} from Node2 2017-09-14 15:15:24,658 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 8 with 8 2017-09-14 15:15:24,658 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 8 with 8 2017-09-14 15:15:24,658 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ledgerId': 1, 'viewNo': 0} from Node3 2017-09-14 15:15:24,659 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 8 with 8 2017-09-14 15:15:24,659 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 8 with 8 2017-09-14 15:15:24,659 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node3', 'Node2'} that its ledger of type 1 is latest 2017-09-14 15:15:24,659 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node1 found from ledger status LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ledgerId': 1, 'viewNo': 0} that it does not need catchup 2017-09-14 15:15:24,659 | DEBUG | node.py (1479) | preLedgerCatchUp | Node1 going to process any ordered requests before starting catchup. 2017-09-14 15:15:24,659 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node1:0 going to remove 0 Ordered messages from outbox 2017-09-14 15:15:24,659 | DEBUG | node.py (1888) | force_process_ordered | Node1 processed 0 Ordered batches for instance 0 before starting catch up 2017-09-14 15:15:24,659 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node1 processed 0 stashed ordered requests 2017-09-14 15:15:24,659 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-09-14 15:15:24,661 | INFO | node.py (1489) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 1 2017-09-14 15:15:24,661 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 8 with 8 2017-09-14 15:15:24,661 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 1, caught up 0 in total 2017-09-14 15:15:24,661 | DEBUG | node.py (1609) | num_txns_caught_up_in_last_catchup | Node1 caught up to 0 txns in the last catchup 2017-09-14 15:15:24,661 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node1 processed 0 stashed ordered requests 2017-09-14 15:15:24,661 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-09-14 15:15:24,661 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node1 got view change quorum (1 >= 1) 2017-09-14 15:15:24,662 | DEBUG | node.py (1549) | is_catchup_needed | Node1 is caught up for the current view 1 2017-09-14 15:15:24,662 | INFO | node.py (1537) | allLedgersCaughtUp | CATCH-UP: Node1 does not need any more catchups 2017-09-14 15:15:24,662 | DEBUG | primary_decider.py ( 134) | send | Node1's elector sending VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [(0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'viewNo': 1} 2017-09-14 15:15:24,662 | DEBUG | primary_selector.py ( 281) | _startSelection | Node1 starting selection 2017-09-14 15:15:24,662 | DISPLAY | primary_selector.py ( 291) | _startSelection | PRIMARY SELECTION: Node1:0 selected primary Node2:0 for instance 0 (view 1) 2017-09-14 15:15:24,662 | INFO | node.py ( 510) | start_participating | Node1 started participating 2017-09-14 15:15:24,663 | DEBUG | replica.py ( 408) | primaryName | Node1:0 setting primaryName for view no 1 to: Node2:0 2017-09-14 15:15:24,663 | DEBUG | replica.py (1694) | _gc | Node1:0 cleaning up till (0, 3) 2017-09-14 15:15:24,663 | DEBUG | replica.py (1709) | _gc | Node1:0 found 0 3-phase keys to clean 2017-09-14 15:15:24,663 | DEBUG | replica.py (1711) | _gc | Node1:0 found 0 request keys to clean 2017-09-14 15:15:24,663 | DEBUG | replica.py ( 313) | h | Node1:0 set watermarks as 0 300 2017-09-14 15:15:24,663 | DISPLAY | primary_selector.py ( 315) | _startSelection | VIEW CHANGE: Node1:0 declares view change 1 as completed for instance 0, new primary is Node2:0, ledger info is [(0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-09-14 15:15:24,664 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ledgerId': 1, 'viewNo': 0} from Node3 2017-09-14 15:15:24,664 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 8 with 8 2017-09-14 15:15:24,664 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 8 with 8 2017-09-14 15:15:24,664 | INFO | upgrader.py ( 150) | should_notify_about_upgrade_result | Node's 'Node1' last upgrade txn is None 2017-09-14 15:15:24,664 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 2, caught up 0 in total 2017-09-14 15:15:24,664 | DEBUG | node.py (1609) | num_txns_caught_up_in_last_catchup | Node1 caught up to 0 txns in the last catchup 2017-09-14 15:15:24,664 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node1 processed 0 stashed ordered requests 2017-09-14 15:15:24,664 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-09-14 15:15:24,665 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node1 got view change quorum (2 >= 1) 2017-09-14 15:15:24,665 | DEBUG | node.py (1549) | is_catchup_needed | Node1 is caught up for the current view 1 2017-09-14 15:15:24,665 | INFO | node.py (1537) | allLedgersCaughtUp | CATCH-UP: Node1 does not need any more catchups 2017-09-14 15:15:24,665 | DEBUG | primary_selector.py ( 281) | _startSelection | Node1 starting selection 2017-09-14 15:15:24,665 | DEBUG | primary_selector.py ( 284) | _startSelection | Node1:0 already has a primary 2017-09-14 15:15:24,665 | INFO | node.py (1624) | no_more_catchups_needed | Node1 starting to participate since catchup is done, primaries are selected but mode was not set to participating 2017-09-14 15:15:24,665 | INFO | node.py ( 510) | start_participating | Node1 started participating 2017-09-14 15:15:24,665 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None} from Node3 2017-09-14 15:15:24,666 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:15:24,666 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:15:24,666 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None}, 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-09-14 15:15:24,666 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None}, 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-09-14 15:15:24,667 | DEBUG | node.py (2608) | send | Node1 sending message VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [(0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'viewNo': 1} to all recipients: ['Node3', 'Node2'] 2017-09-14 15:15:24,667 | DEBUG | batched.py ( 100) | flushOutBoxes | Node1 batching 5 msgs to Node3 into one transmission 2017-09-14 15:15:24,667 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', b'{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":null,"txnSeqNo":8,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', b'{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":null,"txnSeqNo":8,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', b'{"name":"Node2","ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,8,"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","viewNo":1}']) 2017-09-14 15:15:24,667 | TRACE | batched.py ( 110) | flushOutBoxes | Node1 sending payload to Node3: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":8,\\"viewNo\\":null},\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":8,\\"viewNo\\":null},\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"name\\":\\"Node2\\",\\"ledgerInfo\\":[[0,3,\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\"],[1,8,\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"op\\":\\"VIEW_CHANGE_DONE\\",\\"viewNo\\":1}"],"op":"BATCH","signature":null}' 2017-09-14 15:15:24,667 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":8,\\"viewNo\\":null},\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":8,\\"viewNo\\":null},\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"name\\":\\"Node2\\",\\"ledgerInfo\\":[[0,3,\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\"],[1,8,\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"op\\":\\"VIEW_CHANGE_DONE\\",\\"viewNo\\":1}"],"op":"BATCH","signature":null}' to Node3 2017-09-14 15:15:24,668 | DEBUG | batched.py ( 100) | flushOutBoxes | Node1 batching 3 msgs to Node2 into one transmission 2017-09-14 15:15:24,668 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', b'{"name":"Node2","ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,8,"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","viewNo":1}']) 2017-09-14 15:15:24,668 | TRACE | batched.py ( 110) | flushOutBoxes | Node1 sending payload to Node2: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"name\\":\\"Node2\\",\\"ledgerInfo\\":[[0,3,\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\"],[1,8,\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"op\\":\\"VIEW_CHANGE_DONE\\",\\"viewNo\\":1}"],"op":"BATCH","signature":null}' 2017-09-14 15:15:24,668 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"name\\":\\"Node2\\",\\"ledgerInfo\\":[[0,3,\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\"],[1,8,\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"op\\":\\"VIEW_CHANGE_DONE\\",\\"viewNo\\":1}"],"op":"BATCH","signature":null}' to Node2 2017-09-14 15:15:24,692 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 2 messages through listener 2017-09-14 15:15:24,692 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: BATCH{'messages': ['{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}'], 'signature': None} 2017-09-14 15:15:24,692 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'messages': ['{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}'], 'op': 'BATCH', 'signature': None}, 'Node2') 2017-09-14 15:15:24,693 | DEBUG | node.py (1272) | unpackNodeMsg | Node1 processing a batch BATCH{'messages': ['{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}'], 'signature': None} 2017-09-14 15:15:24,693 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 8, 'viewNo': 0, 'ppSeqNo': 3, 'ledgerId': 1, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'}, 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,693 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 8, 'viewNo': 0, 'ppSeqNo': 3, 'ledgerId': 1, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'}, 'params': {'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE'}, 'Node2') 2017-09-14 15:15:24,693 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 8, 'viewNo': 0, 'ppSeqNo': 3, 'ledgerId': 1, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'}, 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,693 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 8, 'viewNo': 0, 'ppSeqNo': 3, 'ledgerId': 1, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'}, 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,693 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 8, 'viewNo': 0, 'ppSeqNo': 3, 'ledgerId': 1, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'}, 'params': {'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE'}, 'Node2') 2017-09-14 15:15:24,693 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 8, 'viewNo': 0, 'ppSeqNo': 3, 'ledgerId': 1, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'}, 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,694 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}', '{"name":"Node2","op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,8,"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":1}'], 'signature': None} 2017-09-14 15:15:24,694 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}', '{"name":"Node2","op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,8,"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":1}'], 'op': 'BATCH', 'signature': None}, 'Node3') 2017-09-14 15:15:24,694 | DEBUG | node.py (1272) | unpackNodeMsg | Node1 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja","ppSeqNo":3,"txnSeqNo":8,"viewNo":0}}', '{"name":"Node2","op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,8,"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":1}'], 'signature': None} 2017-09-14 15:15:24,694 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 8, 'viewNo': 0, 'ppSeqNo': 3, 'ledgerId': 1, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'}, 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,694 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 8, 'viewNo': 0, 'ppSeqNo': 3, 'ledgerId': 1, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'}, 'params': {'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE'}, 'Node3') 2017-09-14 15:15:24,694 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 8, 'viewNo': 0, 'ppSeqNo': 3, 'ledgerId': 1, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'}, 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,694 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 8, 'viewNo': 0, 'ppSeqNo': 3, 'ledgerId': 1, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'}, 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,694 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 8, 'viewNo': 0, 'ppSeqNo': 3, 'ledgerId': 1, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'}, 'params': {'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE'}, 'Node3') 2017-09-14 15:15:24,694 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 8, 'viewNo': 0, 'ppSeqNo': 3, 'ledgerId': 1, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'}, 'params': {'ledgerId': 1}} 2017-09-14 15:15:24,695 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 1} 2017-09-14 15:15:24,695 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'name': 'Node2', 'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'op': 'VIEW_CHANGE_DONE', 'viewNo': 1}, 'Node3') 2017-09-14 15:15:24,695 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 1} 2017-09-14 15:15:24,695 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ledgerId': 1, 'viewNo': 0} from Node2 2017-09-14 15:15:24,695 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 8 with 8 2017-09-14 15:15:24,696 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 8 with 8 2017-09-14 15:15:24,696 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node3', 'Node2'} that its ledger of type 1 is latest 2017-09-14 15:15:24,696 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ledgerId': 1, 'viewNo': 0} from Node2 2017-09-14 15:15:24,696 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 8 with 8 2017-09-14 15:15:24,696 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 8 with 8 2017-09-14 15:15:24,696 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node3', 'Node2'} that its ledger of type 1 is latest 2017-09-14 15:15:24,696 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ledgerId': 1, 'viewNo': 0} from Node3 2017-09-14 15:15:24,697 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 8 with 8 2017-09-14 15:15:24,697 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 8 with 8 2017-09-14 15:15:24,697 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node3', 'Node2'} that its ledger of type 1 is latest 2017-09-14 15:15:24,697 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'txnSeqNo': 8, 'merkleRoot': '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja', 'ppSeqNo': 3, 'ledgerId': 1, 'viewNo': 0} from Node3 2017-09-14 15:15:24,697 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 8 with 8 2017-09-14 15:15:24,697 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 8 with 8 2017-09-14 15:15:24,697 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node3', 'Node2'} that its ledger of type 1 is latest 2017-09-14 15:15:24,698 | DEBUG | node.py (1205) | sendToElector | Node1 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 1}, 'Node3') 2017-09-14 15:15:24,698 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node1's primary selector started processing of ViewChangeDone msg from Node3 : VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 1} 2017-09-14 15:15:24,698 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 1} because it already decided primary which is Node2:0 2017-09-14 15:15:24,710 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 1 messages through listener 2017-09-14 15:15:24,711 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: PREPREPARE{'digest': '22c727caf41ca2e8f715bcfcf11f8730a7dc29d6ccc2553d64dc9a1e97f34261', 'stateRootHash': '5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq', 'txnRootHash': '8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm', 'ledgerId': 1, 'viewNo': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1505402110572745]], 'instId': 0, 'ppTime': 1505402124, 'ppSeqNo': 1, 'discarded': 1} 2017-09-14 15:15:24,711 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'digest': '22c727caf41ca2e8f715bcfcf11f8730a7dc29d6ccc2553d64dc9a1e97f34261', 'stateRootHash': '5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq', 'ppSeqNo': 1, 'op': 'PREPREPARE', 'viewNo': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1505402110572745]], 'ledgerId': 1, 'instId': 0, 'discarded': 1, 'txnRootHash': '8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm', 'ppTime': 1505402124}, 'Node2') 2017-09-14 15:15:24,711 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox PREPREPARE{'digest': '22c727caf41ca2e8f715bcfcf11f8730a7dc29d6ccc2553d64dc9a1e97f34261', 'stateRootHash': '5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq', 'txnRootHash': '8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm', 'ledgerId': 1, 'viewNo': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1505402110572745]], 'instId': 0, 'ppTime': 1505402124, 'ppSeqNo': 1, 'discarded': 1} 2017-09-14 15:15:24,712 | DEBUG | replica.py ( 823) | processPrePrepare | Node1:0 received PRE-PREPARE(1, 1) from Node2:0 at 13501.56385812 2017-09-14 15:15:24,713 | DEBUG | replica.py (1034) | validate_pre_prepare | Node1:0 state root before processing PREPREPARE{'digest': '22c727caf41ca2e8f715bcfcf11f8730a7dc29d6ccc2553d64dc9a1e97f34261', 'stateRootHash': '5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq', 'txnRootHash': '8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm', 'ledgerId': 1, 'viewNo': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402110572745)], 'instId': 0, 'ppTime': 1505402124, 'ppSeqNo': 1, 'discarded': 1} is b'\xba\xacPz1\x97K\xccr\xbe\xdc\x96\x95\x0b\xdb\x85y2\x03\xb9\xeeI\xda\xbd\x87\x8e\xf8:Gc\xa3S', 4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja 2017-09-14 15:15:24,714 | DEBUG | replica.py ( 934) | doPrepare | Node1:0 Sending PREPARE(1, 1) at 13501.566190549 2017-09-14 15:15:24,715 | DEBUG | replica.py (2193) | send | Node1:0 sending Prepare 2017-09-14 15:15:24,715 | TRACE | replica.py (2194) | send | Node1:0 sending PREPARE{'digest': '22c727caf41ca2e8f715bcfcf11f8730a7dc29d6ccc2553d64dc9a1e97f34261', 'stateRootHash': '5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq', 'txnRootHash': '8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm', 'viewNo': 1, 'instId': 0, 'ppTime': 1505402124, 'ppSeqNo': 1} 2017-09-14 15:15:24,715 | DEBUG | replica.py ( 917) | tryCommit | Node1:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': '22c727caf41ca2e8f715bcfcf11f8730a7dc29d6ccc2553d64dc9a1e97f34261', 'stateRootHash': '5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq', 'txnRootHash': '8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm', 'viewNo': 1, 'instId': 0, 'ppTime': 1505402124, 'ppSeqNo': 1} 2017-09-14 15:15:24,715 | DEBUG | replica.py ( 614) | trackBatches | Node1:0 tracking batch for PREPREPARE{'digest': '22c727caf41ca2e8f715bcfcf11f8730a7dc29d6ccc2553d64dc9a1e97f34261', 'stateRootHash': '5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq', 'txnRootHash': '8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm', 'ledgerId': 1, 'viewNo': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402110572745)], 'instId': 0, 'ppTime': 1505402124, 'ppSeqNo': 1, 'discarded': 1} with state root b'\xba\xacPz1\x97K\xccr\xbe\xdc\x96\x95\x0b\xdb\x85y2\x03\xb9\xeeI\xda\xbd\x87\x8e\xf8:Gc\xa3S' 2017-09-14 15:15:24,715 | DEBUG | replica.py ( 842) | processPrePrepare | Node1:0 processed incoming PRE-PREPARE(1, 1) 2017-09-14 15:15:24,715 | DEBUG | node.py (2608) | send | Node1 sending message PREPARE{'digest': '22c727caf41ca2e8f715bcfcf11f8730a7dc29d6ccc2553d64dc9a1e97f34261', 'stateRootHash': '5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq', 'txnRootHash': '8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm', 'viewNo': 1, 'instId': 0, 'ppTime': 1505402124, 'ppSeqNo': 1} to all recipients: ['Node3', 'Node2'] 2017-09-14 15:15:24,716 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"digest":"22c727caf41ca2e8f715bcfcf11f8730a7dc29d6ccc2553d64dc9a1e97f34261","stateRootHash":"5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq","viewNo":1,"ppSeqNo":1,"op":"PREPARE","instId":0,"ppTime":1505402124,"txnRootHash":"8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm"}' to Node3 2017-09-14 15:15:24,716 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"digest":"22c727caf41ca2e8f715bcfcf11f8730a7dc29d6ccc2553d64dc9a1e97f34261","stateRootHash":"5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq","viewNo":1,"ppSeqNo":1,"op":"PREPARE","instId":0,"ppTime":1505402124,"txnRootHash":"8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm"}' to Node3 2017-09-14 15:15:24,717 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"digest":"22c727caf41ca2e8f715bcfcf11f8730a7dc29d6ccc2553d64dc9a1e97f34261","stateRootHash":"5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq","viewNo":1,"ppSeqNo":1,"op":"PREPARE","instId":0,"ppTime":1505402124,"txnRootHash":"8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm"}' to Node2 2017-09-14 15:15:24,717 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"digest":"22c727caf41ca2e8f715bcfcf11f8730a7dc29d6ccc2553d64dc9a1e97f34261","stateRootHash":"5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq","viewNo":1,"ppSeqNo":1,"op":"PREPARE","instId":0,"ppTime":1505402124,"txnRootHash":"8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm"}' to Node2 2017-09-14 15:15:24,729 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 2 messages through listener 2017-09-14 15:15:24,729 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: PREPARE{'digest': '22c727caf41ca2e8f715bcfcf11f8730a7dc29d6ccc2553d64dc9a1e97f34261', 'stateRootHash': '5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq', 'txnRootHash': '8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm', 'viewNo': 1, 'instId': 0, 'ppTime': 1505402124, 'ppSeqNo': 1} 2017-09-14 15:15:24,730 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'digest': '22c727caf41ca2e8f715bcfcf11f8730a7dc29d6ccc2553d64dc9a1e97f34261', 'stateRootHash': '5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq', 'txnRootHash': '8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm', 'op': 'PREPARE', 'viewNo': 1, 'instId': 0, 'ppTime': 1505402124, 'ppSeqNo': 1}, 'Node3') 2017-09-14 15:15:24,730 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '22c727caf41ca2e8f715bcfcf11f8730a7dc29d6ccc2553d64dc9a1e97f34261', 'stateRootHash': '5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq', 'txnRootHash': '8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm', 'viewNo': 1, 'instId': 0, 'ppTime': 1505402124, 'ppSeqNo': 1} 2017-09-14 15:15:24,730 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 1} 2017-09-14 15:15:24,730 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'instId': 0, 'ppSeqNo': 1, 'op': 'COMMIT', 'viewNo': 1}, 'Node3') 2017-09-14 15:15:24,730 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 1} 2017-09-14 15:15:24,731 | DEBUG | replica.py ( 953) | doCommit | Node1:0 Sending COMMIT(1, 1) at 13501.582839642 2017-09-14 15:15:24,731 | DEBUG | replica.py (2193) | send | Node1:0 sending Commit 2017-09-14 15:15:24,731 | TRACE | replica.py (2194) | send | Node1:0 sending COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 1} 2017-09-14 15:15:24,731 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 1} commits where f is 0 2017-09-14 15:15:24,731 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (1, 1) 2017-09-14 15:15:24,731 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(1, 1) from Node3:0 2017-09-14 15:15:24,732 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 1} commits where f is 0 2017-09-14 15:15:24,732 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(1, 1) 2017-09-14 15:15:24,732 | DEBUG | node.py (2608) | send | Node1 sending message COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 1} to all recipients: ['Node3', 'Node2'] 2017-09-14 15:15:24,738 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"instId":0,"ppSeqNo":1,"op":"COMMIT","viewNo":1}' to Node3 2017-09-14 15:15:24,739 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"instId":0,"ppSeqNo":1,"op":"COMMIT","viewNo":1}' to Node3 2017-09-14 15:15:24,739 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"instId":0,"ppSeqNo":1,"op":"COMMIT","viewNo":1}' to Node2 2017-09-14 15:15:24,739 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"instId":0,"ppSeqNo":1,"op":"COMMIT","viewNo":1}' to Node2 2017-09-14 15:15:24,740 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 1 messages through listener 2017-09-14 15:15:24,740 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 1} 2017-09-14 15:15:24,740 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'instId': 0, 'ppSeqNo': 1, 'op': 'COMMIT', 'viewNo': 1}, 'Node2') 2017-09-14 15:15:24,740 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 1} 2017-09-14 15:15:24,741 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(1, 1) from Node2:0 2017-09-14 15:15:24,741 | TRACE | replica.py ( 925) | tryOrder | Node1:0 returning request to node 2017-09-14 15:15:24,741 | DEBUG | replica.py (1444) | doOrder | Node1:0 ordering COMMIT (1, 1) 2017-09-14 15:15:24,741 | DEBUG | replica.py ( 323) | last_ordered_3pc | Node1:0 set last ordered as (1, 1) 2017-09-14 15:15:24,742 | DEBUG | replica.py (2193) | send | Node1:0 sending Ordered 2017-09-14 15:15:24,742 | TRACE | replica.py (2194) | send | Node1:0 sending ORDERED{'stateRootHash': '5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq', 'txnRootHash': '8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm', 'ledgerId': 1, 'viewNo': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402110572745)], 'instId': 0, 'ppTime': 1505402124, 'ppSeqNo': 1} 2017-09-14 15:15:24,742 | INFO | replica.py (1483) | order_3pc_key | Node1:0 ordered batch request, view no 1, ppSeqNo 1, ledger 1, state root 5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq, txn root 8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1505402110572745)], discarded [] 2017-09-14 15:15:24,742 | DEBUG | replica.py (1556) | _newCheckpointState | Node1:0 adding new checkpoint state for (1, 100) 2017-09-14 15:15:24,742 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(1, 1) 2017-09-14 15:15:24,742 | DEBUG | node.py (1847) | processOrdered | Node1 executing Ordered batch 1 1 of 1 requests 2017-09-14 15:15:24,743 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-09-14 15:15:24,743 | DEBUG | node.py (2386) | sendReplyToClient | Node1 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1505402110572745) to client 2017-09-14 15:15:24,743 | TRACE | zstack.py ( 754) | transmitThroughListener | Node1C transmitting b'{"result":{"txnTime":1505402124,"reqId":1505402110572745,"type":"1","signature":"JrZZALkJWvqyur4yneS2TcrNTjSRZHRx1wjyDEiaWgj7wH6rPRwYacdJJ7Ku7Hr7QkuG9uX5HpXA2wxFLjZ2xKF","dest":"V4SGRU86Z58d6TV7PBUe64","seqNo":9,"auditPath":["4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja"],"rootHash":"8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm","identifier":"V4SGRU86Z58d6TV7PBUe6f"},"op":"REPLY"}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:15:24,743 | INFO | node.py (2290) | executeBatch | Node1 committed batch request, view no 1, ppSeqNo 1, ledger 1, state root 5fypm8PCwjD71AxdsvdEnPR93ywNhnJRG6vpmoWzohxq, txn root 8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm, requests: [('V4SGRU86Z58d6TV7PBUe6f', 1505402110572745)] 2017-09-14 15:15:24,744 | DEBUG | node.py (2305) | executeBatch | Node1 storing 3PC key (1, 1) for ledger 1 range (9, 9) 2017-09-14 15:15:24,744 | DEBUG | monitor.py ( 233) | requestOrdered | Got ordered request with identifier V4SGRU86Z58d6TV7PBUe6f and reqId 1505402110572745 but it was from a previous view 2017-09-14 15:15:24,744 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:15:27,425 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action dump_json_file with id 4 2017-09-14 15:15:27,731 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1C got 1 messages through listener 2017-09-14 15:15:27,733 | DEBUG | node.py (2236) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1505402127709038 2017-09-14 15:15:27,733 | TRACE | node.py (1388) | validateClientMsg | Node1C received CLIENT message: SafeRequest: {'reqId': 1505402127709038, 'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe65', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D'} 2017-09-14 15:15:27,733 | DEBUG | node.py (1434) | processClientInBox | Node1C processing b'v#ZljK34WWooA6%RgESn.Ox' request SafeRequest: {'reqId': 1505402127709038, 'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe65', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D'} 2017-09-14 15:15:27,734 | DEBUG | node.py (1708) | processRequest | Node1 received client request: SafeRequest: {'reqId': 1505402127709038, 'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe65', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D'} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:15:27,734 | TRACE | propagator.py ( 165) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'reqId': 1505402127709038, 'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe65', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D'} 2017-09-14 15:15:27,734 | INFO | propagator.py ( 148) | propagate | Node1 propagating request ('V4SGRU86Z58d6TV7PBUe6f', 1505402127709038) from client b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:15:27,734 | DEBUG | node.py (2608) | send | Node1 sending message PROPAGATE{'senderClient': 'v#ZljK34WWooA6%RgESn.Ox', 'request': {'reqId': 1505402127709038, 'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe65', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D'}} to all recipients: ['Node3', 'Node2'] 2017-09-14 15:15:27,734 | DEBUG | propagator.py ( 214) | forward | Node1 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1505402127709038) to 0 replicas 2017-09-14 15:15:27,740 | TRACE | zstack.py ( 754) | transmitThroughListener | Node1C transmitting b'{"reqId":1505402127709038,"identifier":"V4SGRU86Z58d6TV7PBUe6f","op":"REQACK"}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:15:27,741 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE","request":{"reqId":1505402127709038,"operation":{"dest":"V4SGRU86Z58d6TV7PBUe65","type":"1"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D"}}' to Node3 2017-09-14 15:15:27,741 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE","request":{"reqId":1505402127709038,"operation":{"dest":"V4SGRU86Z58d6TV7PBUe65","type":"1"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D"}}' to Node3 2017-09-14 15:15:27,741 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE","request":{"reqId":1505402127709038,"operation":{"dest":"V4SGRU86Z58d6TV7PBUe65","type":"1"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D"}}' to Node2 2017-09-14 15:15:27,741 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE","request":{"reqId":1505402127709038,"operation":{"dest":"V4SGRU86Z58d6TV7PBUe65","type":"1"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D"}}' to Node2 2017-09-14 15:15:27,759 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 3 messages through listener 2017-09-14 15:15:27,760 | DEBUG | node.py (2236) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1505402127709038 2017-09-14 15:15:27,760 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: PROPAGATE{'senderClient': 'v#ZljK34WWooA6%RgESn.Ox', 'request': {'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D', 'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe65', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402127709038}} 2017-09-14 15:15:27,760 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'senderClient': 'v#ZljK34WWooA6%RgESn.Ox', 'op': 'PROPAGATE', 'request': {'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D', 'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe65', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402127709038}}, 'Node3') 2017-09-14 15:15:27,760 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': 'v#ZljK34WWooA6%RgESn.Ox', 'request': {'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D', 'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe65', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402127709038}} 2017-09-14 15:15:27,768 | DEBUG | node.py (2236) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1505402127709038 2017-09-14 15:15:27,768 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: PROPAGATE{'senderClient': 'v#ZljK34WWooA6%RgESn.Ox', 'request': {'reqId': 1505402127709038, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe65'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D'}} 2017-09-14 15:15:27,768 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'senderClient': 'v#ZljK34WWooA6%RgESn.Ox', 'op': 'PROPAGATE', 'request': {'reqId': 1505402127709038, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe65'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D'}}, 'Node2') 2017-09-14 15:15:27,768 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'senderClient': 'v#ZljK34WWooA6%RgESn.Ox', 'request': {'reqId': 1505402127709038, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe65'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D'}} 2017-09-14 15:15:27,768 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: PREPREPARE{'digest': '89ccc072336b1421ffb726dffb332d7d90ed24649d0dc981ddfc174581b650de', 'stateRootHash': '2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf', 'txnRootHash': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'ledgerId': 1, 'viewNo': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1505402127709038]], 'instId': 0, 'ppTime': 1505402127, 'ppSeqNo': 2, 'discarded': 1} 2017-09-14 15:15:27,769 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'digest': '89ccc072336b1421ffb726dffb332d7d90ed24649d0dc981ddfc174581b650de', 'stateRootHash': '2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf', 'ppSeqNo': 2, 'op': 'PREPREPARE', 'viewNo': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1505402127709038]], 'ledgerId': 1, 'instId': 0, 'discarded': 1, 'txnRootHash': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'ppTime': 1505402127}, 'Node2') 2017-09-14 15:15:27,769 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox PREPREPARE{'digest': '89ccc072336b1421ffb726dffb332d7d90ed24649d0dc981ddfc174581b650de', 'stateRootHash': '2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf', 'txnRootHash': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'ledgerId': 1, 'viewNo': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1505402127709038]], 'instId': 0, 'ppTime': 1505402127, 'ppSeqNo': 2, 'discarded': 1} 2017-09-14 15:15:27,769 | DEBUG | node.py (1757) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': 'v#ZljK34WWooA6%RgESn.Ox', 'request': {'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D', 'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe65', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402127709038}} 2017-09-14 15:15:27,769 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'reqId': 1505402127709038, 'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe65', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D'} 2017-09-14 15:15:27,769 | DEBUG | propagator.py ( 246) | tryForwarding | Node1 not forwarding request SafeRequest: {'reqId': 1505402127709038, 'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe65', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D'} to its replicas since already forwarded 2017-09-14 15:15:27,770 | DEBUG | node.py (1757) | processPropagate | Node Node1 received propagated request: PROPAGATE{'senderClient': 'v#ZljK34WWooA6%RgESn.Ox', 'request': {'reqId': 1505402127709038, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe65'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D'}} 2017-09-14 15:15:27,770 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'reqId': 1505402127709038, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe65'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D'} 2017-09-14 15:15:27,770 | DEBUG | propagator.py ( 246) | tryForwarding | Node1 not forwarding request SafeRequest: {'reqId': 1505402127709038, 'operation': {'type': '1', 'dest': 'V4SGRU86Z58d6TV7PBUe65'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D'} to its replicas since already forwarded 2017-09-14 15:15:27,771 | DEBUG | replica.py ( 823) | processPrePrepare | Node1:0 received PRE-PREPARE(1, 2) from Node2:0 at 13504.623247271 2017-09-14 15:15:27,772 | DEBUG | replica.py (1034) | validate_pre_prepare | Node1:0 state root before processing PREPREPARE{'digest': '89ccc072336b1421ffb726dffb332d7d90ed24649d0dc981ddfc174581b650de', 'stateRootHash': '2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf', 'txnRootHash': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'ledgerId': 1, 'viewNo': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402127709038)], 'instId': 0, 'ppTime': 1505402127, 'ppSeqNo': 2, 'discarded': 1} is b'Ek\x11\xa6O\xa30u A\x81\x91\x97)\x94\x88}e\xd6\x17\xcfO\x13\xda\xe5\x91\x9e\x0e\xa4<\x93\xa6', 8WgYbzQuSn8Vbm288JRmvWDocvcRpJE1b54zCb9jnxm 2017-09-14 15:15:27,774 | DEBUG | replica.py ( 934) | doPrepare | Node1:0 Sending PREPARE(1, 2) at 13504.625942299 2017-09-14 15:15:27,774 | DEBUG | replica.py (2193) | send | Node1:0 sending Prepare 2017-09-14 15:15:27,774 | TRACE | replica.py (2194) | send | Node1:0 sending PREPARE{'digest': '89ccc072336b1421ffb726dffb332d7d90ed24649d0dc981ddfc174581b650de', 'stateRootHash': '2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf', 'txnRootHash': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'viewNo': 1, 'instId': 0, 'ppTime': 1505402127, 'ppSeqNo': 2} 2017-09-14 15:15:27,775 | DEBUG | replica.py ( 917) | tryCommit | Node1:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': '89ccc072336b1421ffb726dffb332d7d90ed24649d0dc981ddfc174581b650de', 'stateRootHash': '2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf', 'txnRootHash': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'viewNo': 1, 'instId': 0, 'ppTime': 1505402127, 'ppSeqNo': 2} 2017-09-14 15:15:27,775 | DEBUG | replica.py ( 614) | trackBatches | Node1:0 tracking batch for PREPREPARE{'digest': '89ccc072336b1421ffb726dffb332d7d90ed24649d0dc981ddfc174581b650de', 'stateRootHash': '2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf', 'txnRootHash': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'ledgerId': 1, 'viewNo': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402127709038)], 'instId': 0, 'ppTime': 1505402127, 'ppSeqNo': 2, 'discarded': 1} with state root b'Ek\x11\xa6O\xa30u A\x81\x91\x97)\x94\x88}e\xd6\x17\xcfO\x13\xda\xe5\x91\x9e\x0e\xa4<\x93\xa6' 2017-09-14 15:15:27,775 | DEBUG | replica.py ( 842) | processPrePrepare | Node1:0 processed incoming PRE-PREPARE(1, 2) 2017-09-14 15:15:27,775 | DEBUG | node.py (2608) | send | Node1 sending message PREPARE{'digest': '89ccc072336b1421ffb726dffb332d7d90ed24649d0dc981ddfc174581b650de', 'stateRootHash': '2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf', 'txnRootHash': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'viewNo': 1, 'instId': 0, 'ppTime': 1505402127, 'ppSeqNo': 2} to all recipients: ['Node3', 'Node2'] 2017-09-14 15:15:27,775 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 1 messages through listener 2017-09-14 15:15:27,776 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: PREPARE{'digest': '89ccc072336b1421ffb726dffb332d7d90ed24649d0dc981ddfc174581b650de', 'stateRootHash': '2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf', 'txnRootHash': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'viewNo': 1, 'instId': 0, 'ppTime': 1505402127, 'ppSeqNo': 2} 2017-09-14 15:15:27,776 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'digest': '89ccc072336b1421ffb726dffb332d7d90ed24649d0dc981ddfc174581b650de', 'stateRootHash': '2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf', 'txnRootHash': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'op': 'PREPARE', 'viewNo': 1, 'instId': 0, 'ppTime': 1505402127, 'ppSeqNo': 2}, 'Node3') 2017-09-14 15:15:27,776 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'digest': '89ccc072336b1421ffb726dffb332d7d90ed24649d0dc981ddfc174581b650de', 'stateRootHash': '2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf', 'txnRootHash': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'viewNo': 1, 'instId': 0, 'ppTime': 1505402127, 'ppSeqNo': 2} 2017-09-14 15:15:27,777 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"digest":"89ccc072336b1421ffb726dffb332d7d90ed24649d0dc981ddfc174581b650de","stateRootHash":"2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf","viewNo":1,"ppSeqNo":2,"op":"PREPARE","instId":0,"ppTime":1505402127,"txnRootHash":"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq"}' to Node3 2017-09-14 15:15:27,777 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"digest":"89ccc072336b1421ffb726dffb332d7d90ed24649d0dc981ddfc174581b650de","stateRootHash":"2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf","viewNo":1,"ppSeqNo":2,"op":"PREPARE","instId":0,"ppTime":1505402127,"txnRootHash":"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq"}' to Node3 2017-09-14 15:15:27,777 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"digest":"89ccc072336b1421ffb726dffb332d7d90ed24649d0dc981ddfc174581b650de","stateRootHash":"2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf","viewNo":1,"ppSeqNo":2,"op":"PREPARE","instId":0,"ppTime":1505402127,"txnRootHash":"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq"}' to Node2 2017-09-14 15:15:27,777 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"digest":"89ccc072336b1421ffb726dffb332d7d90ed24649d0dc981ddfc174581b650de","stateRootHash":"2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf","viewNo":1,"ppSeqNo":2,"op":"PREPARE","instId":0,"ppTime":1505402127,"txnRootHash":"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq"}' to Node2 2017-09-14 15:15:27,778 | DEBUG | replica.py ( 953) | doCommit | Node1:0 Sending COMMIT(1, 2) at 13504.629550319 2017-09-14 15:15:27,778 | DEBUG | replica.py (2193) | send | Node1:0 sending Commit 2017-09-14 15:15:27,778 | TRACE | replica.py (2194) | send | Node1:0 sending COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 1} 2017-09-14 15:15:27,778 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 1} commits where f is 0 2017-09-14 15:15:27,778 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (1, 2) 2017-09-14 15:15:27,778 | DEBUG | node.py (2608) | send | Node1 sending message COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 1} to all recipients: ['Node3', 'Node2'] 2017-09-14 15:15:27,779 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"instId":0,"ppSeqNo":2,"op":"COMMIT","viewNo":1}' to Node3 2017-09-14 15:15:27,779 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"instId":0,"ppSeqNo":2,"op":"COMMIT","viewNo":1}' to Node3 2017-09-14 15:15:27,779 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"instId":0,"ppSeqNo":2,"op":"COMMIT","viewNo":1}' to Node2 2017-09-14 15:15:27,779 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"instId":0,"ppSeqNo":2,"op":"COMMIT","viewNo":1}' to Node2 2017-09-14 15:15:27,791 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 2 messages through listener 2017-09-14 15:15:27,792 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 1} 2017-09-14 15:15:27,792 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'instId': 0, 'ppSeqNo': 2, 'op': 'COMMIT', 'viewNo': 1}, 'Node3') 2017-09-14 15:15:27,792 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 1} 2017-09-14 15:15:27,792 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 1} 2017-09-14 15:15:27,792 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'instId': 0, 'ppSeqNo': 2, 'op': 'COMMIT', 'viewNo': 1}, 'Node2') 2017-09-14 15:15:27,792 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 1} 2017-09-14 15:15:27,793 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(1, 2) from Node3:0 2017-09-14 15:15:27,793 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 1} commits where f is 0 2017-09-14 15:15:27,793 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(1, 2) 2017-09-14 15:15:27,793 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(1, 2) from Node2:0 2017-09-14 15:15:27,793 | TRACE | replica.py ( 925) | tryOrder | Node1:0 returning request to node 2017-09-14 15:15:27,794 | DEBUG | replica.py (1444) | doOrder | Node1:0 ordering COMMIT (1, 2) 2017-09-14 15:15:27,794 | DEBUG | replica.py ( 323) | last_ordered_3pc | Node1:0 set last ordered as (1, 2) 2017-09-14 15:15:27,794 | DEBUG | replica.py (2193) | send | Node1:0 sending Ordered 2017-09-14 15:15:27,794 | TRACE | replica.py (2194) | send | Node1:0 sending ORDERED{'stateRootHash': '2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf', 'txnRootHash': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'ledgerId': 1, 'viewNo': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402127709038)], 'instId': 0, 'ppTime': 1505402127, 'ppSeqNo': 2} 2017-09-14 15:15:27,794 | INFO | replica.py (1483) | order_3pc_key | Node1:0 ordered batch request, view no 1, ppSeqNo 2, ledger 1, state root 2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf, txn root 3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1505402127709038)], discarded [] 2017-09-14 15:15:27,794 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(1, 2) 2017-09-14 15:15:27,794 | DEBUG | node.py (1847) | processOrdered | Node1 executing Ordered batch 1 2 of 1 requests 2017-09-14 15:15:27,795 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-09-14 15:15:27,795 | DEBUG | node.py (2386) | sendReplyToClient | Node1 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1505402127709038) to client 2017-09-14 15:15:27,795 | TRACE | zstack.py ( 754) | transmitThroughListener | Node1C transmitting b'{"result":{"txnTime":1505402127,"reqId":1505402127709038,"type":"1","signature":"Y24JAeKUNSRv99ADBTnHxtgZinaecKqMwjWYE7jhGa8WGTmABVFasN71hL9GrW9B5nBzrAsw9e6yWoAuiact62D","dest":"V4SGRU86Z58d6TV7PBUe65","seqNo":10,"auditPath":["CkpbpBJkZfJw4NsSEot9TP4CKGonBTKj66VXAwRUSbUy","4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja"],"rootHash":"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq","identifier":"V4SGRU86Z58d6TV7PBUe6f"},"op":"REPLY"}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:15:27,796 | INFO | node.py (2290) | executeBatch | Node1 committed batch request, view no 1, ppSeqNo 2, ledger 1, state root 2AVUhoqwjquBQ4W4c65vEVtvqC4mUS5148kyv4HEA1Jf, txn root 3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq, requests: [('V4SGRU86Z58d6TV7PBUe6f', 1505402127709038)] 2017-09-14 15:15:27,796 | DEBUG | node.py (2305) | executeBatch | Node1 storing 3PC key (1, 2) for ledger 1 range (10, 10) 2017-09-14 15:15:27,796 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:15:34,241 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 1 2017-09-14 15:15:34,241 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:15:34,241 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:15:34,241 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:15:34,241 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:15:34,242 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:15:34,242 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:15:34,242 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 7 to run in 10 seconds 2017-09-14 15:15:39,431 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:15:39,431 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:15:39,432 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:15:44,246 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 7 2017-09-14 15:15:44,246 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:15:44,247 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:15:44,247 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:15:44,247 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:15:44,247 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:15:44,247 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:15:44,247 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 8 to run in 10 seconds 2017-09-14 15:15:54,256 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 8 2017-09-14 15:15:54,257 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:15:54,257 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:15:54,257 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:15:54,257 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:15:54,257 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:15:54,257 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:15:54,257 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 9 to run in 10 seconds 2017-09-14 15:15:54,437 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:15:54,438 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:15:54,438 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:16:04,258 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 9 2017-09-14 15:16:04,258 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:16:04,258 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:16:04,258 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:16:04,258 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:16:04,259 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:16:04,259 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:16:04,259 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 10 to run in 10 seconds 2017-09-14 15:16:16,848 | DEBUG | node_runner.py ( 18) | run_node | You can find logs in /home/sovrin/.sovrin/Node1.log 2017-09-14 15:16:16,848 | DEBUG | node_runner.py ( 21) | run_node | Sovrin related env vars: [] 2017-09-14 15:16:17,746 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: git 2017-09-14 15:16:17,760 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: hg 2017-09-14 15:16:17,795 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: svn 2017-09-14 15:16:17,796 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: bzr 2017-09-14 15:16:18,333 | DEBUG | selector_events.py ( 53) | __init__ | Using selector: EpollSelector 2017-09-14 15:16:18,333 | DEBUG | looper.py ( 125) | __init__ | Setting handler for SIGINT 2017-09-14 15:16:18,379 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-09-14 15:16:18,399 | DEBUG | ledger.py ( 72) | recoverTree | Recovering tree from hash store of size 10 2017-09-14 15:16:18,400 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.0015519010012212675 seconds 2017-09-14 15:16:18,456 | DEBUG | idr_cache.py ( 25) | __init__ | Initializing identity cache Node1 2017-09-14 15:16:18,543 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-09-14 15:16:18,563 | DEBUG | ledger.py ( 72) | recoverTree | Recovering tree from hash store of size 3 2017-09-14 15:16:18,564 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.0006290569999691797 seconds 2017-09-14 15:16:18,617 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-09-14 15:16:18,617 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: typing.Dict<~KT, ~VT>] because it does not have a 'pluginType' attribute 2017-09-14 15:16:18,617 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-09-14 15:16:18,617 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-09-14 15:16:18,617 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-09-14 15:16:18,618 | INFO | plugin_loader.py ( 117) | _load | plugin FirebaseStatsConsumer successfully loaded from module plugin_firebase_stats_consumer 2017-09-14 15:16:18,619 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkPerformance will be repeating every 60 seconds 2017-09-14 15:16:18,619 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 1 to run in 60 seconds 2017-09-14 15:16:18,619 | DEBUG | replica.py ( 313) | h | Node1:0 set watermarks as 0 300 2017-09-14 15:16:18,619 | DISPLAY | replicas.py ( 36) | grow | Node1 added replica Node1:0 to instance 0 (master) 2017-09-14 15:16:18,620 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkPerformance will be repeating every 10 seconds 2017-09-14 15:16:18,620 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 1 to run in 10 seconds 2017-09-14 15:16:18,620 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkNodeRequestSpike will be repeating every 60 seconds 2017-09-14 15:16:18,620 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkNodeRequestSpike with id 2 to run in 60 seconds 2017-09-14 15:16:18,620 | DEBUG | plugin_helper.py ( 24) | loadPlugins | Plugin loading started to load plugins from basedir: /home/sovrin/.sovrin 2017-09-14 15:16:18,620 | DEBUG | plugin_helper.py ( 68) | loadPlugins | Total plugins loaded from basedir /home/sovrin/.sovrin are : 0 2017-09-14 15:16:18,621 | DEBUG | node.py ( 340) | __init__ | total plugins loaded in node: 0 2017-09-14 15:16:18,687 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-09-14 15:16:18,707 | DEBUG | ledger.py ( 67) | recoverTree | Recovering tree from transaction log 2017-09-14 15:16:18,768 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.06021836599938979 seconds 2017-09-14 15:16:18,789 | DEBUG | upgrader.py ( 118) | check_upgrade_succeeded | Node Node1 has no upgrade events 2017-09-14 15:16:18,789 | INFO | node.py (2420) | initStateFromLedger | Node1 found state to be empty, recreating from ledger 2017-09-14 15:16:18,790 | DEBUG | motor.py ( 34) | set_status | Node1 changing status from stopped to starting 2017-09-14 15:16:18,790 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-09-14 15:16:18,790 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-09-14 15:16:18,790 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-09-14 15:16:18,790 | DEBUG | zstack.py ( 319) | start | Node1 starting with restricted as True and reSetupAuth as True 2017-09-14 15:16:18,791 | DEBUG | authenticator.py ( 31) | start | Starting ZAP at inproc://zeromq.zap.1 2017-09-14 15:16:18,791 | DEBUG | base.py ( 72) | allow | Allowing 0.0.0.0 2017-09-14 15:16:18,791 | DEBUG | base.py ( 112) | configure_curve | Configure curve: *[/home/sovrin/.sovrin/Node1/public_keys] 2017-09-14 15:16:18,792 | DEBUG | zstack.py ( 347) | open | Node1 will bind its listener at 9701 2017-09-14 15:16:18,792 | INFO | stacks.py ( 84) | start | CONNECTION: Node1 listening for other nodes at 0.0.0.0:9701 2017-09-14 15:16:18,792 | DEBUG | zstack.py ( 319) | start | Node1C starting with restricted as False and reSetupAuth as True 2017-09-14 15:16:18,792 | DEBUG | authenticator.py ( 31) | start | Starting ZAP at inproc://zeromq.zap.2 2017-09-14 15:16:18,792 | DEBUG | base.py ( 72) | allow | Allowing 0.0.0.0 2017-09-14 15:16:18,792 | DEBUG | base.py ( 112) | configure_curve | Configure curve: *[*] 2017-09-14 15:16:18,793 | DEBUG | zstack.py ( 347) | open | Node1C will bind its listener at 9702 2017-09-14 15:16:18,793 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action propose_view_change with id 3 to run in 60 seconds 2017-09-14 15:16:18,793 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action dump_json_file with id 4 to run in 3 seconds 2017-09-14 15:16:18,793 | DEBUG | has_action_queue.py ( 79) | startRepeating | dump_json_file will be repeating every 60 seconds 2017-09-14 15:16:18,793 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action dump_json_file with id 5 to run in 60 seconds 2017-09-14 15:16:18,793 | INFO | node.py ( 621) | start | Node1 first time running... 2017-09-14 15:16:18,794 | DEBUG | kit_zstack.py ( 97) | connectToMissing | CONNECTION: Node1 found the following missing connections: Node3, Node2 2017-09-14 15:16:18,795 | TRACE | remote.py ( 86) | connect | connecting socket 78 34752016 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-09-14 15:16:18,795 | INFO | zstack.py ( 590) | connect | CONNECTION: Node1 looking for Node3 at 10.0.0.4:9705 2017-09-14 15:16:18,796 | DEBUG | zstack.py ( 645) | sendPingPong | Node1 pinged Node3 2017-09-14 15:16:18,796 | TRACE | remote.py ( 86) | connect | connecting socket 81 34773984 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-09-14 15:16:18,796 | INFO | zstack.py ( 590) | connect | CONNECTION: Node1 looking for Node2 at 10.0.0.3:9703 2017-09-14 15:16:18,797 | DEBUG | zstack.py ( 645) | sendPingPong | Node1 pinged Node2 2017-09-14 15:16:18,797 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:16:18,808 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'pi' to Node3 2017-09-14 15:16:18,808 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'pi' to Node3 2017-09-14 15:16:18,808 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'pi' to Node2 2017-09-14 15:16:18,809 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'pi' to Node2 2017-09-14 15:16:18,867 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 4 messages through listener 2017-09-14 15:16:18,868 | DEBUG | zstack.py ( 663) | handlePingPong | Node1 got ping from Node2 2017-09-14 15:16:18,868 | DEBUG | zstack.py ( 645) | sendPingPong | Node1 ponged Node2 2017-09-14 15:16:18,870 | DEBUG | node.py (2236) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1505402168868387 2017-09-14 15:16:18,870 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: PROPAGATE{'request': {'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe66', 'type': '1'}, 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:16:18,870 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'request': {'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe66', 'type': '1'}, 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox', 'op': 'PROPAGATE'}, 'Node2') 2017-09-14 15:16:18,870 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'request': {'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe66', 'type': '1'}, 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:16:18,870 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: PREPREPARE{'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1505402168868387]], 'digest': '9ecbd5039951ac63430121b1efcd09398a11361343dec5681f72386f62408e30', 'stateRootHash': '2YkpdGxx16PJ9cKVYHjcGBvBFvAcnWtYHTTMMQznVMo6', 'ledgerId': 1, 'instId': 0, 'viewNo': 1, 'ppTime': 1505402168, 'discarded': 1, 'txnRootHash': 'J58EEZUkgZpjDMen6f7zj1jgYhj5GTsw2YxQRFUZ8XNS', 'ppSeqNo': 3} 2017-09-14 15:16:18,871 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1505402168868387]], 'stateRootHash': '2YkpdGxx16PJ9cKVYHjcGBvBFvAcnWtYHTTMMQznVMo6', 'ledgerId': 1, 'ppSeqNo': 3, 'op': 'PREPREPARE', 'digest': '9ecbd5039951ac63430121b1efcd09398a11361343dec5681f72386f62408e30', 'ppTime': 1505402168, 'viewNo': 1, 'discarded': 1, 'txnRootHash': 'J58EEZUkgZpjDMen6f7zj1jgYhj5GTsw2YxQRFUZ8XNS', 'instId': 0}, 'Node2') 2017-09-14 15:16:18,871 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox PREPREPARE{'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1505402168868387]], 'digest': '9ecbd5039951ac63430121b1efcd09398a11361343dec5681f72386f62408e30', 'stateRootHash': '2YkpdGxx16PJ9cKVYHjcGBvBFvAcnWtYHTTMMQznVMo6', 'ledgerId': 1, 'instId': 0, 'viewNo': 1, 'ppTime': 1505402168, 'discarded': 1, 'txnRootHash': 'J58EEZUkgZpjDMen6f7zj1jgYhj5GTsw2YxQRFUZ8XNS', 'ppSeqNo': 3} 2017-09-14 15:16:18,871 | DEBUG | zstack.py ( 669) | handlePingPong | Node1 got pong from Node2 2017-09-14 15:16:18,871 | DEBUG | node.py (1757) | processPropagate | Node Node1 received propagated request: PROPAGATE{'request': {'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe66', 'type': '1'}, 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:16:18,871 | TRACE | propagator.py ( 165) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe66', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'} 2017-09-14 15:16:18,871 | INFO | propagator.py ( 148) | propagate | Node1 propagating request ('V4SGRU86Z58d6TV7PBUe6f', 1505402168868387) from client v#ZljK34WWooA6%RgESn.Ox 2017-09-14 15:16:18,871 | DEBUG | node.py (2608) | send | Node1 sending message PROPAGATE{'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe66', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402168868387, 'signature': '36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} to all recipients: ['Node3', 'Node2'] 2017-09-14 15:16:18,872 | DEBUG | propagator.py ( 214) | forward | Node1 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1505402168868387) to 0 replicas 2017-09-14 15:16:18,872 | DEBUG | node.py (1170) | msgHasAcceptableViewNo | Node1 stashing a message for a future view: PREPREPARE{'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1505402168868387]], 'digest': '9ecbd5039951ac63430121b1efcd09398a11361343dec5681f72386f62408e30', 'stateRootHash': '2YkpdGxx16PJ9cKVYHjcGBvBFvAcnWtYHTTMMQznVMo6', 'ledgerId': 1, 'instId': 0, 'viewNo': 1, 'ppTime': 1505402168, 'discarded': 1, 'txnRootHash': 'J58EEZUkgZpjDMen6f7zj1jgYhj5GTsw2YxQRFUZ8XNS', 'ppSeqNo': 3} 2017-09-14 15:16:18,873 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"request":{"operation":{"dest":"V4SGRU86Z58d6TV7PBUe66","type":"1"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402168868387,"signature":"36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node3 2017-09-14 15:16:18,873 | DEBUG | zstack.py ( 728) | transmit | Remote Node3 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-09-14 15:16:18,873 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"request":{"operation":{"dest":"V4SGRU86Z58d6TV7PBUe66","type":"1"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402168868387,"signature":"36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node3 2017-09-14 15:16:18,874 | DEBUG | batched.py ( 100) | flushOutBoxes | Node1 batching 2 msgs to Node2 into one transmission 2017-09-14 15:16:18,874 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'po', b'{"request":{"operation":{"dest":"V4SGRU86Z58d6TV7PBUe66","type":"1"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402168868387,"signature":"36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}']) 2017-09-14 15:16:18,874 | TRACE | batched.py ( 110) | flushOutBoxes | Node1 sending payload to Node2: b'{"messages":["po","{\\"request\\":{\\"operation\\":{\\"dest\\":\\"V4SGRU86Z58d6TV7PBUe66\\",\\"type\\":\\"1\\"},\\"identifier\\":\\"V4SGRU86Z58d6TV7PBUe6f\\",\\"reqId\\":1505402168868387,\\"signature\\":\\"36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8\\"},\\"senderClient\\":\\"v#ZljK34WWooA6%RgESn.Ox\\",\\"op\\":\\"PROPAGATE\\"}"],"signature":null,"op":"BATCH"}' 2017-09-14 15:16:18,874 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"messages":["po","{\\"request\\":{\\"operation\\":{\\"dest\\":\\"V4SGRU86Z58d6TV7PBUe66\\",\\"type\\":\\"1\\"},\\"identifier\\":\\"V4SGRU86Z58d6TV7PBUe6f\\",\\"reqId\\":1505402168868387,\\"signature\\":\\"36VinsqhWTyNvXEpeVKrLFAdmkrmPbE3u4LBi4uvnVmreSZuZTruzWKm44dg7TJ8h8n5RSuGcRTP4JStw7Y31YS8\\"},\\"senderClient\\":\\"v#ZljK34WWooA6%RgESn.Ox\\",\\"op\\":\\"PROPAGATE\\"}"],"signature":null,"op":"BATCH"}' to Node2 2017-09-14 15:16:18,875 | DEBUG | keep_in_touch.py ( 68) | conns | Node1's connections changed from set() to {'Node2'} 2017-09-14 15:16:18,875 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node1 now connected to Node2 2017-09-14 15:16:18,875 | DEBUG | motor.py ( 34) | set_status | Node1 changing status from starting to started_hungry 2017-09-14 15:16:18,875 | DEBUG | node.py ( 941) | checkInstances | Node1 choosing to start election on the basis of count 2 and nodes {'Node2'} 2017-09-14 15:16:18,875 | DEBUG | primary_selector.py ( 77) | get_msgs_for_lagged_nodes | Node1 has no ViewChangeDone message to send for view 0 2017-09-14 15:16:18,875 | DEBUG | node.py ( 902) | send_current_state_to_lagging_node | Node1 sending current state CURRENT_STATE{'primary': [], 'viewNo': 0} to lagged node Node2 2017-09-14 15:16:18,875 | DEBUG | node.py (2608) | send | Node1 sending message CURRENT_STATE{'primary': [], 'viewNo': 0} to 1 recipients: ['Node2'] 2017-09-14 15:16:18,876 | DEBUG | node.py (2608) | send | Node1 sending message LEDGER_STATUS{'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ppSeqNo': None, 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} to 1 recipients: ['Node2'] 2017-09-14 15:16:18,876 | DEBUG | batched.py ( 100) | flushOutBoxes | Node1 batching 2 msgs to Node2 into one transmission 2017-09-14 15:16:18,876 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"viewNo":0,"op":"CURRENT_STATE","primary":[]}', b'{"ledgerId":0,"op":"LEDGER_STATUS","viewNo":null,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ppSeqNo":null,"txnSeqNo":3}']) 2017-09-14 15:16:18,877 | TRACE | batched.py ( 110) | flushOutBoxes | Node1 sending payload to Node2: b'{"messages":["{\\"viewNo\\":0,\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[]}","{\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null,\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":3}"],"signature":null,"op":"BATCH"}' 2017-09-14 15:16:18,877 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"messages":["{\\"viewNo\\":0,\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[]}","{\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null,\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":3}"],"signature":null,"op":"BATCH"}' to Node2 2017-09-14 15:16:18,900 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 1 messages through listener 2017-09-14 15:16:18,900 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: BATCH{'messages': ['{"op":"CURRENT_STATE","primary":[{"ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,8,"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node2","viewNo":1}],"viewNo":1}', '{"ledgerId":0,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","txnSeqNo":3,"viewNo":null}', '{"ledgerId":1,"op":"LEDGER_STATUS","ppSeqNo":2,"merkleRoot":"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq","txnSeqNo":10,"viewNo":1}', '{"ledgerId":2,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","txnSeqNo":0,"viewNo":null}'], 'signature': None} 2017-09-14 15:16:18,901 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'messages': ['{"op":"CURRENT_STATE","primary":[{"ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,8,"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node2","viewNo":1}],"viewNo":1}', '{"ledgerId":0,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","txnSeqNo":3,"viewNo":null}', '{"ledgerId":1,"op":"LEDGER_STATUS","ppSeqNo":2,"merkleRoot":"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq","txnSeqNo":10,"viewNo":1}', '{"ledgerId":2,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","txnSeqNo":0,"viewNo":null}'], 'signature': None, 'op': 'BATCH'}, 'Node2') 2017-09-14 15:16:18,901 | DEBUG | node.py (1272) | unpackNodeMsg | Node1 processing a batch BATCH{'messages': ['{"op":"CURRENT_STATE","primary":[{"ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,8,"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node2","viewNo":1}],"viewNo":1}', '{"ledgerId":0,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","txnSeqNo":3,"viewNo":null}', '{"ledgerId":1,"op":"LEDGER_STATUS","ppSeqNo":2,"merkleRoot":"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq","txnSeqNo":10,"viewNo":1}', '{"ledgerId":2,"op":"LEDGER_STATUS","ppSeqNo":null,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","txnSeqNo":0,"viewNo":null}'], 'signature': None} 2017-09-14 15:16:18,901 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: CURRENT_STATE{'primary': [{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1}], 'viewNo': 1} 2017-09-14 15:16:18,901 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'viewNo': 1, 'op': 'CURRENT_STATE', 'primary': [{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1}]}, 'Node2') 2017-09-14 15:16:18,901 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox CURRENT_STATE{'primary': [{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1}], 'viewNo': 1} 2017-09-14 15:16:18,902 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: LEDGER_STATUS{'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ppSeqNo': None, 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} 2017-09-14 15:16:18,902 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': None, 'op': 'LEDGER_STATUS', 'ledgerId': 0, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'viewNo': None, 'txnSeqNo': 3}, 'Node2') 2017-09-14 15:16:18,902 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox LEDGER_STATUS{'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ppSeqNo': None, 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} 2017-09-14 15:16:18,902 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: LEDGER_STATUS{'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'ppSeqNo': 2, 'txnSeqNo': 10, 'ledgerId': 1, 'viewNo': 1} 2017-09-14 15:16:18,902 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 2, 'op': 'LEDGER_STATUS', 'ledgerId': 1, 'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'viewNo': 1, 'txnSeqNo': 10}, 'Node2') 2017-09-14 15:16:18,902 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox LEDGER_STATUS{'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'ppSeqNo': 2, 'txnSeqNo': 10, 'ledgerId': 1, 'viewNo': 1} 2017-09-14 15:16:18,902 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None} 2017-09-14 15:16:18,903 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': None, 'op': 'LEDGER_STATUS', 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None, 'txnSeqNo': 0}, 'Node2') 2017-09-14 15:16:18,903 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None} 2017-09-14 15:16:18,903 | DEBUG | node.py ( 907) | process_current_state_message | Node1 processing current state CURRENT_STATE{'primary': [{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1}], 'viewNo': 1} from Node2 2017-09-14 15:16:18,903 | DEBUG | node.py (1170) | msgHasAcceptableViewNo | Node1 stashing a message for a future view: VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1} 2017-09-14 15:16:18,903 | INFO | node.py (1967) | _start_view_change_if_possible | VIEW CHANGE: Node1 starting view change for 1 after 1 view change indications from other nodes 2017-09-14 15:16:18,903 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action _check_view_change_completed with id 6 to run in 60 seconds 2017-09-14 15:16:18,904 | DEBUG | replica.py ( 455) | on_view_change_start | Node1:0 setting last prepared for master to None 2017-09-14 15:16:18,904 | DEBUG | node.py (2146) | startViewChange | Node1 resetting monitor stats after view change 2017-09-14 15:16:18,904 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-09-14 15:16:18,904 | DEBUG | node.py (1205) | sendToElector | Node1 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1}, 'Node2') 2017-09-14 15:16:18,904 | DEBUG | node.py ( 996) | processStashedMsgsForView | Node1 processed 2 stashed msgs for view no 1 2017-09-14 15:16:18,905 | INFO | node.py (2159) | startViewChange | VIEW CHANGE: Node1 changed to view 1, will start catchup now 2017-09-14 15:16:18,905 | DEBUG | node.py (2185) | start_catchup | Node1 does not start the catchup procedure because it is already in this state 2017-09-14 15:16:18,905 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ppSeqNo': None, 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} from Node2 2017-09-14 15:16:18,905 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 3 2017-09-14 15:16:18,905 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 3 2017-09-14 15:16:18,905 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'ppSeqNo': 2, 'txnSeqNo': 10, 'ledgerId': 1, 'viewNo': 1} from Node2 2017-09-14 15:16:18,906 | DEBUG | ledger_manager.py (1016) | stashLedgerStatus | Node1 stashing ledger status LEDGER_STATUS{'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'ppSeqNo': 2, 'txnSeqNo': 10, 'ledgerId': 1, 'viewNo': 1} from Node2 2017-09-14 15:16:18,906 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None} from Node2 2017-09-14 15:16:18,906 | DEBUG | ledger_manager.py (1016) | stashLedgerStatus | Node1 stashing ledger status LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None} from Node2 2017-09-14 15:16:18,906 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node1's primary selector started processing of ViewChangeDone msg from Node2 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1} 2017-09-14 15:16:18,907 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node1 got view change quorum (1 >= 1) 2017-09-14 15:16:18,907 | DEBUG | primary_selector.py ( 215) | has_view_change_from_primary | Node1 received ViewChangeDone from primary Node2 2017-09-14 15:16:18,907 | DEBUG | primary_selector.py ( 238) | has_sufficient_same_view_change_done_messages | Node1 found acceptable primary Node2 and ledger info ((0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')) 2017-09-14 15:16:18,907 | DEBUG | primary_selector.py ( 271) | _startSelection | Node1 cannot start primary selection since mode is 100 2017-09-14 15:16:18,912 | DEBUG | replica.py ( 799) | processThreePhaseMsg | Replica Node1:0 pended request PREPREPARE{'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1505402168868387]], 'digest': '9ecbd5039951ac63430121b1efcd09398a11361343dec5681f72386f62408e30', 'stateRootHash': '2YkpdGxx16PJ9cKVYHjcGBvBFvAcnWtYHTTMMQznVMo6', 'ledgerId': 1, 'instId': 0, 'viewNo': 1, 'ppTime': 1505402168, 'discarded': 1, 'txnRootHash': 'J58EEZUkgZpjDMen6f7zj1jgYhj5GTsw2YxQRFUZ8XNS', 'ppSeqNo': 3} from Node2 2017-09-14 15:16:18,935 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 2 messages through listener 2017-09-14 15:16:18,935 | DEBUG | zstack.py ( 663) | handlePingPong | Node1 got ping from Node3 2017-09-14 15:16:18,935 | DEBUG | zstack.py ( 645) | sendPingPong | Node1 ponged Node3 2017-09-14 15:16:18,935 | DEBUG | zstack.py ( 669) | handlePingPong | Node1 got pong from Node3 2017-09-14 15:16:18,936 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1C got 2 messages through listener 2017-09-14 15:16:18,936 | DEBUG | zstack.py ( 663) | handlePingPong | Node1C got ping from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:16:18,936 | TRACE | zstack.py ( 754) | transmitThroughListener | Node1C transmitting b'po' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:16:18,936 | DEBUG | zstack.py ( 645) | sendPingPong | Node1C ponged b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:16:18,936 | DEBUG | zstack.py ( 663) | handlePingPong | Node1C got ping from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:16:18,936 | TRACE | zstack.py ( 754) | transmitThroughListener | Node1C transmitting b'po' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:16:18,936 | DEBUG | zstack.py ( 645) | sendPingPong | Node1C ponged b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:16:18,937 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'po' to Node3 2017-09-14 15:16:18,937 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'po' to Node3 2017-09-14 15:16:18,937 | DEBUG | keep_in_touch.py ( 68) | conns | Node1's connections changed from {'Node2'} to {'Node3', 'Node2'} 2017-09-14 15:16:18,937 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node1 now connected to Node3 2017-09-14 15:16:18,937 | DEBUG | motor.py ( 34) | set_status | Node1 changing status from started_hungry to started 2017-09-14 15:16:18,938 | DEBUG | node.py ( 941) | checkInstances | Node1 choosing to start election on the basis of count 3 and nodes {'Node3', 'Node2'} 2017-09-14 15:16:18,938 | DEBUG | node.py ( 902) | send_current_state_to_lagging_node | Node1 sending current state CURRENT_STATE{'primary': [VIEW_CHANGE_DONE{'ledgerInfo': ((0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')), 'name': 'Node2', 'viewNo': 1}], 'viewNo': 1} to lagged node Node3 2017-09-14 15:16:18,938 | DEBUG | node.py (2608) | send | Node1 sending message CURRENT_STATE{'primary': [VIEW_CHANGE_DONE{'ledgerInfo': ((0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')), 'name': 'Node2', 'viewNo': 1}], 'viewNo': 1} to 1 recipients: ['Node3'] 2017-09-14 15:16:18,938 | DEBUG | node.py (2608) | send | Node1 sending message LEDGER_STATUS{'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ppSeqNo': None, 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} to 1 recipients: ['Node3'] 2017-09-14 15:16:18,939 | DEBUG | batched.py ( 100) | flushOutBoxes | Node1 batching 2 msgs to Node3 into one transmission 2017-09-14 15:16:18,939 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"viewNo":1,"op":"CURRENT_STATE","primary":[{"ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,8,"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node2","viewNo":1}]}', b'{"ledgerId":0,"op":"LEDGER_STATUS","viewNo":null,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ppSeqNo":null,"txnSeqNo":3}']) 2017-09-14 15:16:18,939 | TRACE | batched.py ( 110) | flushOutBoxes | Node1 sending payload to Node3: b'{"messages":["{\\"viewNo\\":1,\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[{\\"ledgerInfo\\":[[0,3,\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\"],[1,8,\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"name\\":\\"Node2\\",\\"viewNo\\":1}]}","{\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null,\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":3}"],"signature":null,"op":"BATCH"}' 2017-09-14 15:16:18,939 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"messages":["{\\"viewNo\\":1,\\"op\\":\\"CURRENT_STATE\\",\\"primary\\":[{\\"ledgerInfo\\":[[0,3,\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\"],[1,8,\\"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"name\\":\\"Node2\\",\\"viewNo\\":1}]}","{\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null,\\"merkleRoot\\":\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":3}"],"signature":null,"op":"BATCH"}' to Node3 2017-09-14 15:16:18,951 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1C got 1 messages through listener 2017-09-14 15:16:18,951 | TRACE | node.py (1388) | validateClientMsg | Node1C received CLIENT message: LEDGER_STATUS{'merkleRoot': 'CWoRZ62yMB12JyQEUAYU73CNBqJXWaKUafN9yiR9Dre2', 'ppSeqNo': None, 'txnSeqNo': 4, 'ledgerId': 0, 'viewNo': None} 2017-09-14 15:16:18,952 | DEBUG | node.py (1434) | processClientInBox | Node1C processing b'v#ZljK34WWooA6%RgESn.Ox' request LEDGER_STATUS{'merkleRoot': 'CWoRZ62yMB12JyQEUAYU73CNBqJXWaKUafN9yiR9Dre2', 'ppSeqNo': None, 'txnSeqNo': 4, 'ledgerId': 0, 'viewNo': None} 2017-09-14 15:16:18,952 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'merkleRoot': 'CWoRZ62yMB12JyQEUAYU73CNBqJXWaKUafN9yiR9Dre2', 'ppSeqNo': None, 'txnSeqNo': 4, 'ledgerId': 0, 'viewNo': None} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:16:18,952 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 4 2017-09-14 15:16:18,952 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 4 2017-09-14 15:16:18,952 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 4 2017-09-14 15:16:18,952 | TRACE | zstack.py ( 754) | transmitThroughListener | Node1C transmitting b'{"ledgerId":0,"op":"LEDGER_STATUS","viewNo":null,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","ppSeqNo":null,"txnSeqNo":3}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:16:18,953 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 1 messages through listener 2017-09-14 15:16:18,953 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: BATCH{'messages': ['{"op":"CURRENT_STATE","viewNo":1,"primary":[{"ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,8,"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node2","viewNo":1}]}', '{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","txnSeqNo":3,"viewNo":null,"op":"LEDGER_STATUS","ppSeqNo":null}', '{"ledgerId":1,"merkleRoot":"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq","txnSeqNo":10,"viewNo":1,"op":"LEDGER_STATUS","ppSeqNo":2}', '{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","txnSeqNo":0,"viewNo":null,"op":"LEDGER_STATUS","ppSeqNo":null}'], 'signature': None} 2017-09-14 15:16:18,953 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'messages': ['{"op":"CURRENT_STATE","viewNo":1,"primary":[{"ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,8,"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node2","viewNo":1}]}', '{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","txnSeqNo":3,"viewNo":null,"op":"LEDGER_STATUS","ppSeqNo":null}', '{"ledgerId":1,"merkleRoot":"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq","txnSeqNo":10,"viewNo":1,"op":"LEDGER_STATUS","ppSeqNo":2}', '{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","txnSeqNo":0,"viewNo":null,"op":"LEDGER_STATUS","ppSeqNo":null}'], 'signature': None, 'op': 'BATCH'}, 'Node3') 2017-09-14 15:16:18,953 | DEBUG | node.py (1272) | unpackNodeMsg | Node1 processing a batch BATCH{'messages': ['{"op":"CURRENT_STATE","viewNo":1,"primary":[{"ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,8,"4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node2","viewNo":1}]}', '{"ledgerId":0,"merkleRoot":"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq","txnSeqNo":3,"viewNo":null,"op":"LEDGER_STATUS","ppSeqNo":null}', '{"ledgerId":1,"merkleRoot":"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq","txnSeqNo":10,"viewNo":1,"op":"LEDGER_STATUS","ppSeqNo":2}', '{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","txnSeqNo":0,"viewNo":null,"op":"LEDGER_STATUS","ppSeqNo":null}'], 'signature': None} 2017-09-14 15:16:18,954 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: CURRENT_STATE{'primary': [{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1}], 'viewNo': 1} 2017-09-14 15:16:18,954 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'primary': [{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1}], 'op': 'CURRENT_STATE', 'viewNo': 1}, 'Node3') 2017-09-14 15:16:18,954 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox CURRENT_STATE{'primary': [{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1}], 'viewNo': 1} 2017-09-14 15:16:18,954 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: LEDGER_STATUS{'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ppSeqNo': None, 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} 2017-09-14 15:16:18,954 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'ledgerId': 0, 'op': 'LEDGER_STATUS', 'viewNo': None, 'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ppSeqNo': None, 'txnSeqNo': 3}, 'Node3') 2017-09-14 15:16:18,954 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox LEDGER_STATUS{'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ppSeqNo': None, 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} 2017-09-14 15:16:18,955 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: LEDGER_STATUS{'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'ppSeqNo': 2, 'txnSeqNo': 10, 'ledgerId': 1, 'viewNo': 1} 2017-09-14 15:16:18,955 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'ledgerId': 1, 'op': 'LEDGER_STATUS', 'viewNo': 1, 'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'ppSeqNo': 2, 'txnSeqNo': 10}, 'Node3') 2017-09-14 15:16:18,955 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox LEDGER_STATUS{'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'ppSeqNo': 2, 'txnSeqNo': 10, 'ledgerId': 1, 'viewNo': 1} 2017-09-14 15:16:18,955 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None} 2017-09-14 15:16:18,955 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'ledgerId': 2, 'op': 'LEDGER_STATUS', 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ppSeqNo': None, 'txnSeqNo': 0}, 'Node3') 2017-09-14 15:16:18,955 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None} 2017-09-14 15:16:18,955 | DEBUG | node.py ( 907) | process_current_state_message | Node1 processing current state CURRENT_STATE{'primary': [{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1}], 'viewNo': 1} from Node3 2017-09-14 15:16:18,956 | DEBUG | node.py (1205) | sendToElector | Node1 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1}, 'Node3') 2017-09-14 15:16:18,956 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ppSeqNo': None, 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} from Node3 2017-09-14 15:16:18,956 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 3 2017-09-14 15:16:18,956 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 3 2017-09-14 15:16:18,956 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node2', 'Node3'} that its ledger of type 0 is latest 2017-09-14 15:16:18,956 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node1 found from ledger status LEDGER_STATUS{'merkleRoot': '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq', 'ppSeqNo': None, 'txnSeqNo': 3, 'ledgerId': 0, 'viewNo': None} that it does not need catchup 2017-09-14 15:16:18,956 | DEBUG | node.py (1479) | preLedgerCatchUp | Node1 going to process any ordered requests before starting catchup. 2017-09-14 15:16:18,956 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node1:0 going to remove 0 Ordered messages from outbox 2017-09-14 15:16:18,956 | DEBUG | node.py (1888) | force_process_ordered | Node1 processed 0 Ordered batches for instance 0 before starting catch up 2017-09-14 15:16:18,957 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node1 processed 0 stashed ordered requests 2017-09-14 15:16:18,957 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-09-14 15:16:18,957 | INFO | node.py (1489) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 0 2017-09-14 15:16:18,957 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 0 of size 3 with 3 2017-09-14 15:16:18,957 | DEBUG | node.py ( 941) | checkInstances | Node1 choosing to start election on the basis of count 3 and nodes {'Node3', 'Node2'} 2017-09-14 15:16:18,957 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to all recipients: ['Node3', 'Node2'] 2017-09-14 15:16:18,957 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node1 asking Node1 for ledger status of ledger 2 2017-09-14 15:16:18,958 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node2'] 2017-09-14 15:16:18,958 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node1 asking Node2 for ledger status of ledger 2 2017-09-14 15:16:18,958 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-09-14 15:16:18,958 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node1 asking Node3 for ledger status of ledger 2 2017-09-14 15:16:18,958 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node1 going to process 1 stashed ledger statuses for ledger 2 2017-09-14 15:16:18,958 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None} from Node2 2017-09-14 15:16:18,958 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:16:18,958 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:16:18,958 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 0, caught up 0 in total 2017-09-14 15:16:18,959 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'ppSeqNo': 2, 'txnSeqNo': 10, 'ledgerId': 1, 'viewNo': 1} from Node3 2017-09-14 15:16:18,959 | DEBUG | ledger_manager.py (1016) | stashLedgerStatus | Node1 stashing ledger status LEDGER_STATUS{'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'ppSeqNo': 2, 'txnSeqNo': 10, 'ledgerId': 1, 'viewNo': 1} from Node3 2017-09-14 15:16:18,959 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None} from Node3 2017-09-14 15:16:18,959 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:16:18,959 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:16:18,959 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node2', 'Node3'} that its ledger of type 2 is latest 2017-09-14 15:16:18,959 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node1 found from ledger status LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None} that it does not need catchup 2017-09-14 15:16:18,959 | DEBUG | node.py (1479) | preLedgerCatchUp | Node1 going to process any ordered requests before starting catchup. 2017-09-14 15:16:18,960 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node1:0 going to remove 0 Ordered messages from outbox 2017-09-14 15:16:18,960 | DEBUG | node.py (1888) | force_process_ordered | Node1 processed 0 Ordered batches for instance 0 before starting catch up 2017-09-14 15:16:18,960 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node1 processed 0 stashed ordered requests 2017-09-14 15:16:18,960 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-09-14 15:16:18,960 | INFO | node.py (1489) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 2 2017-09-14 15:16:18,960 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:16:18,960 | DEBUG | pool_config.py ( 35) | processLedger | processing config ledger for any POOL_CONFIGs 2017-09-14 15:16:18,962 | DEBUG | upgrader.py ( 208) | processLedger | Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv processing config ledger for any upgrades 2017-09-14 15:16:18,963 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to all recipients: ['Node3', 'Node2'] 2017-09-14 15:16:18,963 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node1 asking Node1 for ledger status of ledger 1 2017-09-14 15:16:18,963 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node2'] 2017-09-14 15:16:18,963 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node1 asking Node2 for ledger status of ledger 1 2017-09-14 15:16:18,963 | DEBUG | node.py (2608) | send | Node1 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-09-14 15:16:18,963 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node1 asking Node3 for ledger status of ledger 1 2017-09-14 15:16:18,963 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node1 going to process 2 stashed ledger statuses for ledger 1 2017-09-14 15:16:18,963 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'ppSeqNo': 2, 'txnSeqNo': 10, 'ledgerId': 1, 'viewNo': 1} from Node2 2017-09-14 15:16:18,964 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 10 with 10 2017-09-14 15:16:18,964 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 10 with 10 2017-09-14 15:16:18,964 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'ppSeqNo': 2, 'txnSeqNo': 10, 'ledgerId': 1, 'viewNo': 1} from Node3 2017-09-14 15:16:18,964 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 10 with 10 2017-09-14 15:16:18,964 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 10 with 10 2017-09-14 15:16:18,964 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node2', 'Node3'} that its ledger of type 1 is latest 2017-09-14 15:16:18,964 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node1 found from ledger status LEDGER_STATUS{'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'ppSeqNo': 2, 'txnSeqNo': 10, 'ledgerId': 1, 'viewNo': 1} that it does not need catchup 2017-09-14 15:16:18,964 | DEBUG | node.py (1479) | preLedgerCatchUp | Node1 going to process any ordered requests before starting catchup. 2017-09-14 15:16:18,964 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node1:0 going to remove 0 Ordered messages from outbox 2017-09-14 15:16:18,965 | DEBUG | node.py (1888) | force_process_ordered | Node1 processed 0 Ordered batches for instance 0 before starting catch up 2017-09-14 15:16:18,965 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node1 processed 0 stashed ordered requests 2017-09-14 15:16:18,965 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-09-14 15:16:18,965 | INFO | node.py (1489) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 1 2017-09-14 15:16:18,965 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 10 with 10 2017-09-14 15:16:18,965 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 1, caught up 0 in total 2017-09-14 15:16:18,965 | DEBUG | node.py (1609) | num_txns_caught_up_in_last_catchup | Node1 caught up to 0 txns in the last catchup 2017-09-14 15:16:18,965 | DEBUG | replica.py ( 323) | last_ordered_3pc | Node1:0 set last ordered as (1, 2) 2017-09-14 15:16:18,965 | DEBUG | replica.py (2234) | _remove_till_caught_up_3pc | Node1:0 going to remove messages for 0 3PC keys 2017-09-14 15:16:18,966 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node1:0 going to remove 0 Ordered messages from outbox 2017-09-14 15:16:18,966 | INFO | node.py (1525) | allLedgersCaughtUp | CATCH-UP: Node1 caught up till (1, 2) 2017-09-14 15:16:18,966 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node1 processed 0 stashed ordered requests 2017-09-14 15:16:18,966 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-09-14 15:16:18,966 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node1 got view change quorum (1 >= 1) 2017-09-14 15:16:18,966 | DEBUG | node.py (1549) | is_catchup_needed | Node1 is caught up for the current view 1 2017-09-14 15:16:18,966 | INFO | node.py (1537) | allLedgersCaughtUp | CATCH-UP: Node1 does not need any more catchups 2017-09-14 15:16:18,967 | DEBUG | primary_decider.py ( 134) | send | Node1's elector sending VIEW_CHANGE_DONE{'ledgerInfo': [(0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 10, '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'name': 'Node2', 'viewNo': 1} 2017-09-14 15:16:18,967 | DEBUG | primary_selector.py ( 281) | _startSelection | Node1 starting selection 2017-09-14 15:16:18,967 | DISPLAY | primary_selector.py ( 291) | _startSelection | PRIMARY SELECTION: Node1:0 selected primary Node2:0 for instance 0 (view 1) 2017-09-14 15:16:18,967 | INFO | node.py ( 510) | start_participating | Node1 started participating 2017-09-14 15:16:18,967 | DEBUG | replica.py ( 408) | primaryName | Node1:0 setting primaryName for view no 1 to: Node2:0 2017-09-14 15:16:18,967 | DEBUG | replica.py (1694) | _gc | Node1:0 cleaning up till (1, 2) 2017-09-14 15:16:18,967 | DEBUG | replica.py (1709) | _gc | Node1:0 found 0 3-phase keys to clean 2017-09-14 15:16:18,968 | DEBUG | replica.py (1711) | _gc | Node1:0 found 0 request keys to clean 2017-09-14 15:16:18,968 | DEBUG | replica.py ( 313) | h | Node1:0 set watermarks as 0 300 2017-09-14 15:16:18,968 | DEBUG | replica.py ( 748) | processPostElectionMsgs | Node1:0 processing pended msg (PREPREPARE{'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1505402168868387]], 'digest': '9ecbd5039951ac63430121b1efcd09398a11361343dec5681f72386f62408e30', 'stateRootHash': '2YkpdGxx16PJ9cKVYHjcGBvBFvAcnWtYHTTMMQznVMo6', 'ledgerId': 1, 'instId': 0, 'viewNo': 1, 'ppTime': 1505402168, 'discarded': 1, 'txnRootHash': 'J58EEZUkgZpjDMen6f7zj1jgYhj5GTsw2YxQRFUZ8XNS', 'ppSeqNo': 3}, 'Node2') 2017-09-14 15:16:18,968 | DEBUG | replica.py ( 823) | processPrePrepare | Node1:0 received PRE-PREPARE(1, 3) from Node2:0 at 13555.819818671 2017-09-14 15:16:18,968 | DEBUG | replica.py (1034) | validate_pre_prepare | Node1:0 state root before processing PREPREPARE{'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402168868387)], 'digest': '9ecbd5039951ac63430121b1efcd09398a11361343dec5681f72386f62408e30', 'stateRootHash': '2YkpdGxx16PJ9cKVYHjcGBvBFvAcnWtYHTTMMQznVMo6', 'ledgerId': 1, 'instId': 0, 'viewNo': 1, 'ppTime': 1505402168, 'discarded': 1, 'txnRootHash': 'J58EEZUkgZpjDMen6f7zj1jgYhj5GTsw2YxQRFUZ8XNS', 'ppSeqNo': 3} is b'\x11J\x1d.C\xe9\xb1*\xa9\xb5Z\xb5\xf3\xe5h\xb7\xf4\xc9\xbe\xaf[\x01\xee\x89|T\x06\xffQG\x9cx', 3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq 2017-09-14 15:16:18,970 | DEBUG | replica.py ( 934) | doPrepare | Node1:0 Sending PREPARE(1, 3) at 13555.822166628 2017-09-14 15:16:18,970 | DEBUG | replica.py (2193) | send | Node1:0 sending Prepare 2017-09-14 15:16:18,971 | TRACE | replica.py (2194) | send | Node1:0 sending PREPARE{'stateRootHash': '2YkpdGxx16PJ9cKVYHjcGBvBFvAcnWtYHTTMMQznVMo6', 'instId': 0, 'ppTime': 1505402168, 'digest': '9ecbd5039951ac63430121b1efcd09398a11361343dec5681f72386f62408e30', 'viewNo': 1, 'txnRootHash': 'J58EEZUkgZpjDMen6f7zj1jgYhj5GTsw2YxQRFUZ8XNS', 'ppSeqNo': 3} 2017-09-14 15:16:18,971 | DEBUG | replica.py ( 917) | tryCommit | Node1:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'stateRootHash': '2YkpdGxx16PJ9cKVYHjcGBvBFvAcnWtYHTTMMQznVMo6', 'instId': 0, 'ppTime': 1505402168, 'digest': '9ecbd5039951ac63430121b1efcd09398a11361343dec5681f72386f62408e30', 'viewNo': 1, 'txnRootHash': 'J58EEZUkgZpjDMen6f7zj1jgYhj5GTsw2YxQRFUZ8XNS', 'ppSeqNo': 3} 2017-09-14 15:16:18,971 | DEBUG | replica.py ( 614) | trackBatches | Node1:0 tracking batch for PREPREPARE{'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402168868387)], 'digest': '9ecbd5039951ac63430121b1efcd09398a11361343dec5681f72386f62408e30', 'stateRootHash': '2YkpdGxx16PJ9cKVYHjcGBvBFvAcnWtYHTTMMQznVMo6', 'ledgerId': 1, 'instId': 0, 'viewNo': 1, 'ppTime': 1505402168, 'discarded': 1, 'txnRootHash': 'J58EEZUkgZpjDMen6f7zj1jgYhj5GTsw2YxQRFUZ8XNS', 'ppSeqNo': 3} with state root b'\x11J\x1d.C\xe9\xb1*\xa9\xb5Z\xb5\xf3\xe5h\xb7\xf4\xc9\xbe\xaf[\x01\xee\x89|T\x06\xffQG\x9cx' 2017-09-14 15:16:18,971 | DEBUG | replica.py ( 842) | processPrePrepare | Node1:0 processed incoming PRE-PREPARE(1, 3) 2017-09-14 15:16:18,971 | DISPLAY | primary_selector.py ( 315) | _startSelection | VIEW CHANGE: Node1:0 declares view change 1 as completed for instance 0, new primary is Node2:0, ledger info is [(0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 10, '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-09-14 15:16:18,972 | INFO | upgrader.py ( 150) | should_notify_about_upgrade_result | Node's 'Node1' last upgrade txn is None 2017-09-14 15:16:18,972 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 2, caught up 0 in total 2017-09-14 15:16:18,972 | DEBUG | node.py (1609) | num_txns_caught_up_in_last_catchup | Node1 caught up to 0 txns in the last catchup 2017-09-14 15:16:18,972 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node1 processed 0 stashed ordered requests 2017-09-14 15:16:18,972 | DEBUG | monitor.py ( 192) | reset | Node1's Monitor being reset 2017-09-14 15:16:18,972 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node1 got view change quorum (2 >= 1) 2017-09-14 15:16:18,972 | DEBUG | node.py (1549) | is_catchup_needed | Node1 is caught up for the current view 1 2017-09-14 15:16:18,974 | INFO | node.py (1537) | allLedgersCaughtUp | CATCH-UP: Node1 does not need any more catchups 2017-09-14 15:16:18,974 | DEBUG | primary_selector.py ( 281) | _startSelection | Node1 starting selection 2017-09-14 15:16:18,974 | DEBUG | primary_selector.py ( 284) | _startSelection | Node1:0 already has a primary 2017-09-14 15:16:18,974 | INFO | node.py (1624) | no_more_catchups_needed | Node1 starting to participate since catchup is done, primaries are selected but mode was not set to participating 2017-09-14 15:16:18,975 | INFO | node.py ( 510) | start_participating | Node1 started participating 2017-09-14 15:16:18,975 | DEBUG | node.py (2608) | send | Node1 sending message VIEW_CHANGE_DONE{'ledgerInfo': [(0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'), (1, 10, '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'name': 'Node2', 'viewNo': 1} to all recipients: ['Node3', 'Node2'] 2017-09-14 15:16:18,975 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node1's primary selector started processing of ViewChangeDone msg from Node3 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1} 2017-09-14 15:16:18,975 | DEBUG | message_processor.py ( 29) | discard | Node1 discarding message VIEW_CHANGE_DONE{'ledgerInfo': [[0, 3, '9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq'], [1, 8, '4sbpEBGvw2eUYLnNQcYDH82QDZ5yCmn41yWAjXzPXJja'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node2', 'viewNo': 1} because it already decided primary which is Node2:0 2017-09-14 15:16:18,976 | DEBUG | batched.py ( 100) | flushOutBoxes | Node1 batching 5 msgs to Node3 into one transmission 2017-09-14 15:16:18,976 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', b'{"ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,10,"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node2","op":"VIEW_CHANGE_DONE","viewNo":1}']) 2017-09-14 15:16:18,976 | TRACE | batched.py ( 110) | flushOutBoxes | Node1 sending payload to Node3: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"ledgerInfo\\":[[0,3,\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\"],[1,10,\\"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"name\\":\\"Node2\\",\\"op\\":\\"VIEW_CHANGE_DONE\\",\\"viewNo\\":1}"],"signature":null,"op":"BATCH"}' 2017-09-14 15:16:18,976 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"ledgerInfo\\":[[0,3,\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\"],[1,10,\\"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"name\\":\\"Node2\\",\\"op\\":\\"VIEW_CHANGE_DONE\\",\\"viewNo\\":1}"],"signature":null,"op":"BATCH"}' to Node3 2017-09-14 15:16:18,976 | DEBUG | batched.py ( 100) | flushOutBoxes | Node1 batching 5 msgs to Node2 into one transmission 2017-09-14 15:16:18,977 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', b'{"ledgerInfo":[[0,3,"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq"],[1,10,"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node2","op":"VIEW_CHANGE_DONE","viewNo":1}']) 2017-09-14 15:16:18,977 | TRACE | batched.py ( 110) | flushOutBoxes | Node1 sending payload to Node2: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"ledgerInfo\\":[[0,3,\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\"],[1,10,\\"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"name\\":\\"Node2\\",\\"op\\":\\"VIEW_CHANGE_DONE\\",\\"viewNo\\":1}"],"signature":null,"op":"BATCH"}' 2017-09-14 15:16:18,977 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"ledgerInfo\\":[[0,3,\\"9pM4rmVSZGgU1muLXeHittuQNM8V9Z3hswV3DZDmb5Rq\\"],[1,10,\\"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"name\\":\\"Node2\\",\\"op\\":\\"VIEW_CHANGE_DONE\\",\\"viewNo\\":1}"],"signature":null,"op":"BATCH"}' to Node2 2017-09-14 15:16:18,978 | DEBUG | node.py (2608) | send | Node1 sending message PREPARE{'stateRootHash': '2YkpdGxx16PJ9cKVYHjcGBvBFvAcnWtYHTTMMQznVMo6', 'instId': 0, 'ppTime': 1505402168, 'digest': '9ecbd5039951ac63430121b1efcd09398a11361343dec5681f72386f62408e30', 'viewNo': 1, 'txnRootHash': 'J58EEZUkgZpjDMen6f7zj1jgYhj5GTsw2YxQRFUZ8XNS', 'ppSeqNo': 3} to all recipients: ['Node3', 'Node2'] 2017-09-14 15:16:18,979 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"op":"PREPARE","viewNo":1,"ppTime":1505402168,"ppSeqNo":3,"stateRootHash":"2YkpdGxx16PJ9cKVYHjcGBvBFvAcnWtYHTTMMQznVMo6","digest":"9ecbd5039951ac63430121b1efcd09398a11361343dec5681f72386f62408e30","txnRootHash":"J58EEZUkgZpjDMen6f7zj1jgYhj5GTsw2YxQRFUZ8XNS","instId":0}' to Node3 2017-09-14 15:16:18,979 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"op":"PREPARE","viewNo":1,"ppTime":1505402168,"ppSeqNo":3,"stateRootHash":"2YkpdGxx16PJ9cKVYHjcGBvBFvAcnWtYHTTMMQznVMo6","digest":"9ecbd5039951ac63430121b1efcd09398a11361343dec5681f72386f62408e30","txnRootHash":"J58EEZUkgZpjDMen6f7zj1jgYhj5GTsw2YxQRFUZ8XNS","instId":0}' to Node3 2017-09-14 15:16:18,979 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"op":"PREPARE","viewNo":1,"ppTime":1505402168,"ppSeqNo":3,"stateRootHash":"2YkpdGxx16PJ9cKVYHjcGBvBFvAcnWtYHTTMMQznVMo6","digest":"9ecbd5039951ac63430121b1efcd09398a11361343dec5681f72386f62408e30","txnRootHash":"J58EEZUkgZpjDMen6f7zj1jgYhj5GTsw2YxQRFUZ8XNS","instId":0}' to Node2 2017-09-14 15:16:18,980 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"op":"PREPARE","viewNo":1,"ppTime":1505402168,"ppSeqNo":3,"stateRootHash":"2YkpdGxx16PJ9cKVYHjcGBvBFvAcnWtYHTTMMQznVMo6","digest":"9ecbd5039951ac63430121b1efcd09398a11361343dec5681f72386f62408e30","txnRootHash":"J58EEZUkgZpjDMen6f7zj1jgYhj5GTsw2YxQRFUZ8XNS","instId":0}' to Node2 2017-09-14 15:16:18,993 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 2 messages through listener 2017-09-14 15:16:18,994 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: BATCH{'messages': ['{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq","ppSeqNo":2,"txnSeqNo":10,"viewNo":1}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq","ppSeqNo":2,"txnSeqNo":10,"viewNo":1}}'], 'signature': None} 2017-09-14 15:16:18,994 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'messages': ['{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq","ppSeqNo":2,"txnSeqNo":10,"viewNo":1}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq","ppSeqNo":2,"txnSeqNo":10,"viewNo":1}}'], 'signature': None, 'op': 'BATCH'}, 'Node2') 2017-09-14 15:16:18,994 | DEBUG | node.py (1272) | unpackNodeMsg | Node1 processing a batch BATCH{'messages': ['{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq","ppSeqNo":2,"txnSeqNo":10,"viewNo":1}}', '{"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq","ppSeqNo":2,"txnSeqNo":10,"viewNo":1}}'], 'signature': None} 2017-09-14 15:16:18,995 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None}, 'params': {'ledgerId': 2}} 2017-09-14 15:16:18,995 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None}, 'params': {'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE'}, 'Node2') 2017-09-14 15:16:18,995 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None}, 'params': {'ledgerId': 2}} 2017-09-14 15:16:18,995 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None}, 'params': {'ledgerId': 2}} 2017-09-14 15:16:18,995 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None}, 'params': {'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE'}, 'Node2') 2017-09-14 15:16:18,995 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None}, 'params': {'ledgerId': 2}} 2017-09-14 15:16:18,996 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'txnSeqNo': 10, 'ppSeqNo': 2, 'ledgerId': 1, 'viewNo': 1}, 'params': {'ledgerId': 1}} 2017-09-14 15:16:18,996 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'txnSeqNo': 10, 'ppSeqNo': 2, 'ledgerId': 1, 'viewNo': 1}, 'params': {'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE'}, 'Node2') 2017-09-14 15:16:18,996 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'txnSeqNo': 10, 'ppSeqNo': 2, 'ledgerId': 1, 'viewNo': 1}, 'params': {'ledgerId': 1}} 2017-09-14 15:16:18,996 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'txnSeqNo': 10, 'ppSeqNo': 2, 'ledgerId': 1, 'viewNo': 1}, 'params': {'ledgerId': 1}} 2017-09-14 15:16:18,996 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'txnSeqNo': 10, 'ppSeqNo': 2, 'ledgerId': 1, 'viewNo': 1}, 'params': {'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE'}, 'Node2') 2017-09-14 15:16:18,996 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'txnSeqNo': 10, 'ppSeqNo': 2, 'ledgerId': 1, 'viewNo': 1}, 'params': {'ledgerId': 1}} 2017-09-14 15:16:18,997 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq","ppSeqNo":2,"txnSeqNo":10,"viewNo":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq","ppSeqNo":2,"txnSeqNo":10,"viewNo":1}}'], 'signature': None} 2017-09-14 15:16:18,997 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq","ppSeqNo":2,"txnSeqNo":10,"viewNo":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq","ppSeqNo":2,"txnSeqNo":10,"viewNo":1}}'], 'signature': None, 'op': 'BATCH'}, 'Node3') 2017-09-14 15:16:18,997 | DEBUG | node.py (1272) | unpackNodeMsg | Node1 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq","ppSeqNo":2,"txnSeqNo":10,"viewNo":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq","ppSeqNo":2,"txnSeqNo":10,"viewNo":1}}'], 'signature': None} 2017-09-14 15:16:18,997 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None}, 'params': {'ledgerId': 2}} 2017-09-14 15:16:18,998 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None}, 'params': {'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE'}, 'Node3') 2017-09-14 15:16:18,998 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None}, 'params': {'ledgerId': 2}} 2017-09-14 15:16:18,998 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None}, 'params': {'ledgerId': 2}} 2017-09-14 15:16:18,998 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None}, 'params': {'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE'}, 'Node3') 2017-09-14 15:16:18,998 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None}, 'params': {'ledgerId': 2}} 2017-09-14 15:16:18,999 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'txnSeqNo': 10, 'ppSeqNo': 2, 'ledgerId': 1, 'viewNo': 1}, 'params': {'ledgerId': 1}} 2017-09-14 15:16:18,999 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'txnSeqNo': 10, 'ppSeqNo': 2, 'ledgerId': 1, 'viewNo': 1}, 'params': {'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE'}, 'Node3') 2017-09-14 15:16:18,999 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'txnSeqNo': 10, 'ppSeqNo': 2, 'ledgerId': 1, 'viewNo': 1}, 'params': {'ledgerId': 1}} 2017-09-14 15:16:18,999 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'txnSeqNo': 10, 'ppSeqNo': 2, 'ledgerId': 1, 'viewNo': 1}, 'params': {'ledgerId': 1}} 2017-09-14 15:16:18,999 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'txnSeqNo': 10, 'ppSeqNo': 2, 'ledgerId': 1, 'viewNo': 1}, 'params': {'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE'}, 'Node3') 2017-09-14 15:16:18,999 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'txnSeqNo': 10, 'ppSeqNo': 2, 'ledgerId': 1, 'viewNo': 1}, 'params': {'ledgerId': 1}} 2017-09-14 15:16:18,999 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None} from Node2 2017-09-14 15:16:19,000 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:16:19,000 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:16:19,000 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None} from Node2 2017-09-14 15:16:19,000 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:16:19,000 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:16:19,001 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'ppSeqNo': 2, 'txnSeqNo': 10, 'ledgerId': 1, 'viewNo': 1} from Node2 2017-09-14 15:16:19,001 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 10 with 10 2017-09-14 15:16:19,001 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 10 with 10 2017-09-14 15:16:19,001 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'ppSeqNo': 2, 'txnSeqNo': 10, 'ledgerId': 1, 'viewNo': 1} from Node2 2017-09-14 15:16:19,001 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 10 with 10 2017-09-14 15:16:19,001 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 10 with 10 2017-09-14 15:16:19,002 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None} from Node3 2017-09-14 15:16:19,002 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:16:19,002 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:16:19,002 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node2', 'Node3'} that its ledger of type 2 is latest 2017-09-14 15:16:19,002 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None} from Node3 2017-09-14 15:16:19,003 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:16:19,003 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 2 of size 0 with 0 2017-09-14 15:16:19,003 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node2', 'Node3'} that its ledger of type 2 is latest 2017-09-14 15:16:19,003 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'ppSeqNo': 2, 'txnSeqNo': 10, 'ledgerId': 1, 'viewNo': 1} from Node3 2017-09-14 15:16:19,003 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 10 with 10 2017-09-14 15:16:19,004 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 10 with 10 2017-09-14 15:16:19,004 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node2', 'Node3'} that its ledger of type 1 is latest 2017-09-14 15:16:19,004 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node1 received ledger status: LEDGER_STATUS{'merkleRoot': '3rDuTMTYZTHUKtsP8tGi3AmdTZDxyGL4KehZM2Fwq5iq', 'ppSeqNo': 2, 'txnSeqNo': 10, 'ledgerId': 1, 'viewNo': 1} from Node3 2017-09-14 15:16:19,004 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 10 with 10 2017-09-14 15:16:19,004 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node1 comparing its ledger 1 of size 10 with 10 2017-09-14 15:16:19,004 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node1 found out from {'Node2', 'Node3'} that its ledger of type 1 is latest 2017-09-14 15:16:19,005 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 2 messages through listener 2017-09-14 15:16:19,006 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: COMMIT{'ppSeqNo': 3, 'instId': 0, 'viewNo': 1} 2017-09-14 15:16:19,006 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 3, 'instId': 0, 'op': 'COMMIT', 'viewNo': 1}, 'Node3') 2017-09-14 15:16:19,006 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 3, 'instId': 0, 'viewNo': 1} 2017-09-14 15:16:19,006 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: COMMIT{'ppSeqNo': 3, 'instId': 0, 'viewNo': 1} 2017-09-14 15:16:19,006 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 3, 'op': 'COMMIT', 'instId': 0, 'viewNo': 1}, 'Node2') 2017-09-14 15:16:19,006 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 3, 'instId': 0, 'viewNo': 1} 2017-09-14 15:16:19,007 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(1, 3) from Node3:0 2017-09-14 15:16:19,007 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 3, 'instId': 0, 'viewNo': 1} commits where f is 0 2017-09-14 15:16:19,007 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(1, 3) 2017-09-14 15:16:19,007 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(1, 3) from Node2:0 2017-09-14 15:16:19,007 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 3, 'instId': 0, 'viewNo': 1} commits where f is 0 2017-09-14 15:16:19,007 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(1, 3) 2017-09-14 15:16:21,800 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action dump_json_file with id 4 2017-09-14 15:16:22,364 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1C got 1 messages through listener 2017-09-14 15:16:22,366 | DEBUG | node.py (2236) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1505402182349581 2017-09-14 15:16:22,366 | TRACE | node.py (1388) | validateClientMsg | Node1C received CLIENT message: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:16:22,366 | DEBUG | node.py (1434) | processClientInBox | Node1C processing b'v#ZljK34WWooA6%RgESn.Ox' request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:16:22,367 | DEBUG | node.py (1708) | processRequest | Node1 received client request: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:16:22,367 | TRACE | propagator.py ( 165) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:16:22,367 | INFO | propagator.py ( 148) | propagate | Node1 propagating request ('V4SGRU86Z58d6TV7PBUe6f', 1505402182349581) from client b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:16:22,367 | DEBUG | node.py (2608) | send | Node1 sending message PROPAGATE{'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} to all recipients: ['Node3', 'Node2'] 2017-09-14 15:16:22,367 | DEBUG | propagator.py ( 214) | forward | Node1 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1505402182349581) to 0 replicas 2017-09-14 15:16:22,368 | TRACE | zstack.py ( 754) | transmitThroughListener | Node1C transmitting b'{"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402182349581,"op":"REQACK"}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:16:22,368 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"request":{"operation":{"dest":"V4SGRU86Z58d6TV7PBUe67","type":"1"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402182349581,"signature":"5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node3 2017-09-14 15:16:22,368 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"request":{"operation":{"dest":"V4SGRU86Z58d6TV7PBUe67","type":"1"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402182349581,"signature":"5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node3 2017-09-14 15:16:22,369 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"request":{"operation":{"dest":"V4SGRU86Z58d6TV7PBUe67","type":"1"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402182349581,"signature":"5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node2 2017-09-14 15:16:22,369 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"request":{"operation":{"dest":"V4SGRU86Z58d6TV7PBUe67","type":"1"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402182349581,"signature":"5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb"},"senderClient":"v#ZljK34WWooA6%RgESn.Ox","op":"PROPAGATE"}' to Node2 2017-09-14 15:16:22,369 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 1 messages through listener 2017-09-14 15:16:22,370 | DEBUG | node.py (2236) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1505402182349581 2017-09-14 15:16:22,370 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: PROPAGATE{'request': {'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:16:22,370 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'request': {'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox', 'op': 'PROPAGATE'}, 'Node2') 2017-09-14 15:16:22,371 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'request': {'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:16:22,371 | DEBUG | node.py (1757) | processPropagate | Node Node1 received propagated request: PROPAGATE{'request': {'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:16:22,371 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:16:22,371 | DEBUG | propagator.py ( 246) | tryForwarding | Node1 not forwarding request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} to its replicas since already forwarded 2017-09-14 15:16:22,383 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 1 messages through listener 2017-09-14 15:16:22,383 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: PREPREPARE{'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1505402182349581]], 'digest': 'cfbf1f06b34b99be51c2768e434b24584e26aaf349bdcdfe774d173c848d4247', 'stateRootHash': 'EoqS54wgE8uye5WQ1rHrFj1stwrnnTFZi9y3t6aNY4nv', 'ledgerId': 1, 'instId': 0, 'viewNo': 1, 'ppTime': 1505402182, 'discarded': 1, 'txnRootHash': '6yaNKaDqtduU2k7wmoJcdHdp7Jf3BBjwUEFJAyyukC1T', 'ppSeqNo': 4} 2017-09-14 15:16:22,383 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1505402182349581]], 'stateRootHash': 'EoqS54wgE8uye5WQ1rHrFj1stwrnnTFZi9y3t6aNY4nv', 'ledgerId': 1, 'ppSeqNo': 4, 'op': 'PREPREPARE', 'digest': 'cfbf1f06b34b99be51c2768e434b24584e26aaf349bdcdfe774d173c848d4247', 'ppTime': 1505402182, 'viewNo': 1, 'discarded': 1, 'txnRootHash': '6yaNKaDqtduU2k7wmoJcdHdp7Jf3BBjwUEFJAyyukC1T', 'instId': 0}, 'Node2') 2017-09-14 15:16:22,384 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox PREPREPARE{'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1505402182349581]], 'digest': 'cfbf1f06b34b99be51c2768e434b24584e26aaf349bdcdfe774d173c848d4247', 'stateRootHash': 'EoqS54wgE8uye5WQ1rHrFj1stwrnnTFZi9y3t6aNY4nv', 'ledgerId': 1, 'instId': 0, 'viewNo': 1, 'ppTime': 1505402182, 'discarded': 1, 'txnRootHash': '6yaNKaDqtduU2k7wmoJcdHdp7Jf3BBjwUEFJAyyukC1T', 'ppSeqNo': 4} 2017-09-14 15:16:22,384 | DEBUG | replica.py ( 823) | processPrePrepare | Node1:0 received PRE-PREPARE(1, 4) from Node2:0 at 13559.236264913 2017-09-14 15:16:22,385 | DEBUG | replica.py (1034) | validate_pre_prepare | Node1:0 state root before processing PREPREPARE{'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402182349581)], 'digest': 'cfbf1f06b34b99be51c2768e434b24584e26aaf349bdcdfe774d173c848d4247', 'stateRootHash': 'EoqS54wgE8uye5WQ1rHrFj1stwrnnTFZi9y3t6aNY4nv', 'ledgerId': 1, 'instId': 0, 'viewNo': 1, 'ppTime': 1505402182, 'discarded': 1, 'txnRootHash': '6yaNKaDqtduU2k7wmoJcdHdp7Jf3BBjwUEFJAyyukC1T', 'ppSeqNo': 4} is b'\x16\xfe=\x9f\x84{g1\xa0\xe7I\x8bc\xb3\xe8\xb5\x82\xdd\xf1x\x8f\xcf k\xf0\x9b\xef\x1f\x99\xaf;\xd1', J58EEZUkgZpjDMen6f7zj1jgYhj5GTsw2YxQRFUZ8XNS 2017-09-14 15:16:22,386 | DEBUG | replica.py ( 934) | doPrepare | Node1:0 Sending PREPARE(1, 4) at 13559.237975211 2017-09-14 15:16:22,386 | DEBUG | replica.py (2193) | send | Node1:0 sending Prepare 2017-09-14 15:16:22,386 | TRACE | replica.py (2194) | send | Node1:0 sending PREPARE{'stateRootHash': 'EoqS54wgE8uye5WQ1rHrFj1stwrnnTFZi9y3t6aNY4nv', 'instId': 0, 'ppTime': 1505402182, 'digest': 'cfbf1f06b34b99be51c2768e434b24584e26aaf349bdcdfe774d173c848d4247', 'viewNo': 1, 'txnRootHash': '6yaNKaDqtduU2k7wmoJcdHdp7Jf3BBjwUEFJAyyukC1T', 'ppSeqNo': 4} 2017-09-14 15:16:22,386 | DEBUG | replica.py ( 917) | tryCommit | Node1:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'stateRootHash': 'EoqS54wgE8uye5WQ1rHrFj1stwrnnTFZi9y3t6aNY4nv', 'instId': 0, 'ppTime': 1505402182, 'digest': 'cfbf1f06b34b99be51c2768e434b24584e26aaf349bdcdfe774d173c848d4247', 'viewNo': 1, 'txnRootHash': '6yaNKaDqtduU2k7wmoJcdHdp7Jf3BBjwUEFJAyyukC1T', 'ppSeqNo': 4} 2017-09-14 15:16:22,387 | DEBUG | replica.py ( 614) | trackBatches | Node1:0 tracking batch for PREPREPARE{'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1505402182349581)], 'digest': 'cfbf1f06b34b99be51c2768e434b24584e26aaf349bdcdfe774d173c848d4247', 'stateRootHash': 'EoqS54wgE8uye5WQ1rHrFj1stwrnnTFZi9y3t6aNY4nv', 'ledgerId': 1, 'instId': 0, 'viewNo': 1, 'ppTime': 1505402182, 'discarded': 1, 'txnRootHash': '6yaNKaDqtduU2k7wmoJcdHdp7Jf3BBjwUEFJAyyukC1T', 'ppSeqNo': 4} with state root b'\x16\xfe=\x9f\x84{g1\xa0\xe7I\x8bc\xb3\xe8\xb5\x82\xdd\xf1x\x8f\xcf k\xf0\x9b\xef\x1f\x99\xaf;\xd1' 2017-09-14 15:16:22,387 | DEBUG | replica.py ( 842) | processPrePrepare | Node1:0 processed incoming PRE-PREPARE(1, 4) 2017-09-14 15:16:22,387 | DEBUG | node.py (2608) | send | Node1 sending message PREPARE{'stateRootHash': 'EoqS54wgE8uye5WQ1rHrFj1stwrnnTFZi9y3t6aNY4nv', 'instId': 0, 'ppTime': 1505402182, 'digest': 'cfbf1f06b34b99be51c2768e434b24584e26aaf349bdcdfe774d173c848d4247', 'viewNo': 1, 'txnRootHash': '6yaNKaDqtduU2k7wmoJcdHdp7Jf3BBjwUEFJAyyukC1T', 'ppSeqNo': 4} to all recipients: ['Node3', 'Node2'] 2017-09-14 15:16:22,387 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"op":"PREPARE","viewNo":1,"ppTime":1505402182,"ppSeqNo":4,"stateRootHash":"EoqS54wgE8uye5WQ1rHrFj1stwrnnTFZi9y3t6aNY4nv","digest":"cfbf1f06b34b99be51c2768e434b24584e26aaf349bdcdfe774d173c848d4247","txnRootHash":"6yaNKaDqtduU2k7wmoJcdHdp7Jf3BBjwUEFJAyyukC1T","instId":0}' to Node3 2017-09-14 15:16:22,388 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"op":"PREPARE","viewNo":1,"ppTime":1505402182,"ppSeqNo":4,"stateRootHash":"EoqS54wgE8uye5WQ1rHrFj1stwrnnTFZi9y3t6aNY4nv","digest":"cfbf1f06b34b99be51c2768e434b24584e26aaf349bdcdfe774d173c848d4247","txnRootHash":"6yaNKaDqtduU2k7wmoJcdHdp7Jf3BBjwUEFJAyyukC1T","instId":0}' to Node3 2017-09-14 15:16:22,388 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"op":"PREPARE","viewNo":1,"ppTime":1505402182,"ppSeqNo":4,"stateRootHash":"EoqS54wgE8uye5WQ1rHrFj1stwrnnTFZi9y3t6aNY4nv","digest":"cfbf1f06b34b99be51c2768e434b24584e26aaf349bdcdfe774d173c848d4247","txnRootHash":"6yaNKaDqtduU2k7wmoJcdHdp7Jf3BBjwUEFJAyyukC1T","instId":0}' to Node2 2017-09-14 15:16:22,388 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"op":"PREPARE","viewNo":1,"ppTime":1505402182,"ppSeqNo":4,"stateRootHash":"EoqS54wgE8uye5WQ1rHrFj1stwrnnTFZi9y3t6aNY4nv","digest":"cfbf1f06b34b99be51c2768e434b24584e26aaf349bdcdfe774d173c848d4247","txnRootHash":"6yaNKaDqtduU2k7wmoJcdHdp7Jf3BBjwUEFJAyyukC1T","instId":0}' to Node2 2017-09-14 15:16:22,400 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 3 messages through listener 2017-09-14 15:16:22,401 | DEBUG | node.py (2236) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1505402182349581 2017-09-14 15:16:22,401 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: PROPAGATE{'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:16:22,402 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox', 'op': 'PROPAGATE'}, 'Node3') 2017-09-14 15:16:22,403 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox PROPAGATE{'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:16:22,403 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: PREPARE{'stateRootHash': 'EoqS54wgE8uye5WQ1rHrFj1stwrnnTFZi9y3t6aNY4nv', 'instId': 0, 'ppTime': 1505402182, 'digest': 'cfbf1f06b34b99be51c2768e434b24584e26aaf349bdcdfe774d173c848d4247', 'viewNo': 1, 'txnRootHash': '6yaNKaDqtduU2k7wmoJcdHdp7Jf3BBjwUEFJAyyukC1T', 'ppSeqNo': 4} 2017-09-14 15:16:22,403 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'stateRootHash': 'EoqS54wgE8uye5WQ1rHrFj1stwrnnTFZi9y3t6aNY4nv', 'instId': 0, 'op': 'PREPARE', 'ppTime': 1505402182, 'digest': 'cfbf1f06b34b99be51c2768e434b24584e26aaf349bdcdfe774d173c848d4247', 'viewNo': 1, 'txnRootHash': '6yaNKaDqtduU2k7wmoJcdHdp7Jf3BBjwUEFJAyyukC1T', 'ppSeqNo': 4}, 'Node3') 2017-09-14 15:16:22,403 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox PREPARE{'stateRootHash': 'EoqS54wgE8uye5WQ1rHrFj1stwrnnTFZi9y3t6aNY4nv', 'instId': 0, 'ppTime': 1505402182, 'digest': 'cfbf1f06b34b99be51c2768e434b24584e26aaf349bdcdfe774d173c848d4247', 'viewNo': 1, 'txnRootHash': '6yaNKaDqtduU2k7wmoJcdHdp7Jf3BBjwUEFJAyyukC1T', 'ppSeqNo': 4} 2017-09-14 15:16:22,403 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node3: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1} 2017-09-14 15:16:22,404 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 4, 'instId': 0, 'op': 'COMMIT', 'viewNo': 1}, 'Node3') 2017-09-14 15:16:22,404 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1} 2017-09-14 15:16:22,404 | DEBUG | node.py (1757) | processPropagate | Node Node1 received propagated request: PROPAGATE{'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'}, 'senderClient': 'v#ZljK34WWooA6%RgESn.Ox'} 2017-09-14 15:16:22,404 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:16:22,404 | DEBUG | propagator.py ( 246) | tryForwarding | Node1 not forwarding request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} to its replicas since already forwarded 2017-09-14 15:16:22,405 | DEBUG | replica.py ( 953) | doCommit | Node1:0 Sending COMMIT(1, 4) at 13559.257009235 2017-09-14 15:16:22,405 | DEBUG | replica.py (2193) | send | Node1:0 sending Commit 2017-09-14 15:16:22,405 | TRACE | replica.py (2194) | send | Node1:0 sending COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1} 2017-09-14 15:16:22,405 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1} commits where f is 0 2017-09-14 15:16:22,406 | DEBUG | replica.py ( 876) | processPrepare | Node1:0 processed incoming PREPARE (1, 4) 2017-09-14 15:16:22,406 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(1, 4) from Node3:0 2017-09-14 15:16:22,406 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1} commits where f is 0 2017-09-14 15:16:22,406 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(1, 4) 2017-09-14 15:16:22,406 | DEBUG | node.py (2608) | send | Node1 sending message COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1} to all recipients: ['Node3', 'Node2'] 2017-09-14 15:16:22,407 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"viewNo":1,"ppSeqNo":4,"op":"COMMIT","instId":0}' to Node3 2017-09-14 15:16:22,407 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"viewNo":1,"ppSeqNo":4,"op":"COMMIT","instId":0}' to Node3 2017-09-14 15:16:22,407 | DEBUG | zstack.py ( 723) | transmit | Node1 transmitting message b'{"viewNo":1,"ppSeqNo":4,"op":"COMMIT","instId":0}' to Node2 2017-09-14 15:16:22,407 | TRACE | batched.py ( 96) | flushOutBoxes | Node1 sending msg b'{"viewNo":1,"ppSeqNo":4,"op":"COMMIT","instId":0}' to Node2 2017-09-14 15:16:22,419 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1 got 1 messages through listener 2017-09-14 15:16:22,420 | DEBUG | node.py (1257) | validateNodeMsg | Node1 received node message from Node2: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1} 2017-09-14 15:16:22,420 | DEBUG | node.py (1219) | handleOneNodeMsg | Node1 msg validated ({'ppSeqNo': 4, 'op': 'COMMIT', 'instId': 0, 'viewNo': 1}, 'Node2') 2017-09-14 15:16:22,420 | DEBUG | node.py (1286) | postToNodeInBox | Node1 appending to nodeInbox COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1} 2017-09-14 15:16:22,421 | DEBUG | replica.py ( 894) | processCommit | Node1:0 received COMMIT(1, 4) from Node2:0 2017-09-14 15:16:22,421 | DEBUG | has_action_queue.py ( 79) | startRepeating | process_stashed_out_of_order_commits will be repeating every 1 seconds 2017-09-14 15:16:22,421 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 1 to run in 1 seconds 2017-09-14 15:16:22,421 | DEBUG | replica.py ( 929) | tryOrder | Node1:0 cannot return request to node: stashing COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1} since out of order 2017-09-14 15:16:22,421 | DEBUG | replica.py ( 905) | processCommit | Node1:0 processed incoming COMMIT(1, 4) 2017-09-14 15:16:23,423 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 1 2017-09-14 15:16:23,424 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:23,424 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 2 to run in 1 seconds 2017-09-14 15:16:24,434 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 2 2017-09-14 15:16:24,434 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:24,434 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 3 to run in 1 seconds 2017-09-14 15:16:25,439 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 3 2017-09-14 15:16:25,439 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:25,440 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 4 to run in 1 seconds 2017-09-14 15:16:26,446 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 4 2017-09-14 15:16:26,446 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:26,446 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 5 to run in 1 seconds 2017-09-14 15:16:27,456 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 5 2017-09-14 15:16:27,456 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:27,456 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 6 to run in 1 seconds 2017-09-14 15:16:28,458 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 6 2017-09-14 15:16:28,458 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:28,458 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 7 to run in 1 seconds 2017-09-14 15:16:28,627 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 1 2017-09-14 15:16:28,627 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:16:28,628 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:16:28,628 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:16:28,628 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:16:28,628 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:16:28,628 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:16:28,628 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 7 to run in 10 seconds 2017-09-14 15:16:29,468 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 7 2017-09-14 15:16:29,468 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:29,469 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 8 to run in 1 seconds 2017-09-14 15:16:30,480 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 8 2017-09-14 15:16:30,480 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:30,480 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 9 to run in 1 seconds 2017-09-14 15:16:31,482 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 9 2017-09-14 15:16:31,482 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:31,482 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 10 to run in 1 seconds 2017-09-14 15:16:32,494 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 10 2017-09-14 15:16:32,495 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:32,495 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 11 to run in 1 seconds 2017-09-14 15:16:33,501 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 11 2017-09-14 15:16:33,501 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:33,501 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 12 to run in 1 seconds 2017-09-14 15:16:33,803 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:16:33,804 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:16:33,804 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:16:34,510 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 12 2017-09-14 15:16:34,510 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:34,511 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 13 to run in 1 seconds 2017-09-14 15:16:35,513 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 13 2017-09-14 15:16:35,514 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:35,514 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 14 to run in 1 seconds 2017-09-14 15:16:36,520 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 14 2017-09-14 15:16:36,521 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:36,521 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 15 to run in 1 seconds 2017-09-14 15:16:37,530 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 15 2017-09-14 15:16:37,531 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:37,531 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 16 to run in 1 seconds 2017-09-14 15:16:38,534 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 16 2017-09-14 15:16:38,534 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:38,534 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 17 to run in 1 seconds 2017-09-14 15:16:38,631 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 7 2017-09-14 15:16:38,631 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:16:38,631 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:16:38,631 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:16:38,631 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:16:38,631 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:16:38,631 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:16:38,631 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 8 to run in 10 seconds 2017-09-14 15:16:38,927 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1C got 1 messages through listener 2017-09-14 15:16:38,928 | DEBUG | node.py (2236) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1505402182349581 2017-09-14 15:16:38,928 | TRACE | node.py (1388) | validateClientMsg | Node1C received CLIENT message: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:16:38,928 | DEBUG | node.py (1434) | processClientInBox | Node1C processing b'v#ZljK34WWooA6%RgESn.Ox' request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:16:38,928 | DEBUG | node.py (1708) | processRequest | Node1 received client request: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:16:38,929 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:16:38,929 | DEBUG | propagator.py ( 246) | tryForwarding | Node1 not forwarding request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} to its replicas since already forwarded 2017-09-14 15:16:38,929 | TRACE | zstack.py ( 754) | transmitThroughListener | Node1C transmitting b'{"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402182349581,"op":"REQACK"}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:16:39,535 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 17 2017-09-14 15:16:39,535 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:39,536 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 18 to run in 1 seconds 2017-09-14 15:16:40,545 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 18 2017-09-14 15:16:40,545 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:40,545 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 19 to run in 1 seconds 2017-09-14 15:16:41,547 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 19 2017-09-14 15:16:41,547 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:41,547 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 20 to run in 1 seconds 2017-09-14 15:16:42,548 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 20 2017-09-14 15:16:42,548 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:42,549 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 21 to run in 1 seconds 2017-09-14 15:16:43,554 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 21 2017-09-14 15:16:43,554 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:43,554 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 22 to run in 1 seconds 2017-09-14 15:16:44,562 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 22 2017-09-14 15:16:44,563 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:44,563 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 23 to run in 1 seconds 2017-09-14 15:16:45,566 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 23 2017-09-14 15:16:45,567 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:45,567 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 24 to run in 1 seconds 2017-09-14 15:16:46,579 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 24 2017-09-14 15:16:46,579 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:46,579 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 25 to run in 1 seconds 2017-09-14 15:16:47,589 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 25 2017-09-14 15:16:47,589 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:47,589 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 26 to run in 1 seconds 2017-09-14 15:16:48,603 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 26 2017-09-14 15:16:48,604 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:48,604 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 27 to run in 1 seconds 2017-09-14 15:16:48,636 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 8 2017-09-14 15:16:48,636 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:16:48,636 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:16:48,637 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:16:48,637 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:16:48,637 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:16:48,637 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:16:48,637 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 9 to run in 10 seconds 2017-09-14 15:16:48,810 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:16:48,810 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:16:48,811 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:16:49,613 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 27 2017-09-14 15:16:49,615 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:49,615 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 28 to run in 1 seconds 2017-09-14 15:16:50,615 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 28 2017-09-14 15:16:50,616 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:50,616 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 29 to run in 1 seconds 2017-09-14 15:16:51,619 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 29 2017-09-14 15:16:51,619 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:51,619 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 30 to run in 1 seconds 2017-09-14 15:16:52,629 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 30 2017-09-14 15:16:52,630 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:52,630 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 31 to run in 1 seconds 2017-09-14 15:16:53,631 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 31 2017-09-14 15:16:53,631 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:53,631 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 32 to run in 1 seconds 2017-09-14 15:16:53,939 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1C got 1 messages through listener 2017-09-14 15:16:53,940 | DEBUG | node.py (2236) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1505402182349581 2017-09-14 15:16:53,940 | TRACE | node.py (1388) | validateClientMsg | Node1C received CLIENT message: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:16:53,941 | DEBUG | node.py (1434) | processClientInBox | Node1C processing b'v#ZljK34WWooA6%RgESn.Ox' request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:16:53,941 | DEBUG | node.py (1708) | processRequest | Node1 received client request: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:16:53,941 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:16:53,941 | DEBUG | propagator.py ( 246) | tryForwarding | Node1 not forwarding request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} to its replicas since already forwarded 2017-09-14 15:16:53,941 | TRACE | zstack.py ( 754) | transmitThroughListener | Node1C transmitting b'{"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402182349581,"op":"REQACK"}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:16:54,637 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 32 2017-09-14 15:16:54,637 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:54,637 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 33 to run in 1 seconds 2017-09-14 15:16:55,639 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 33 2017-09-14 15:16:55,640 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:55,640 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 34 to run in 1 seconds 2017-09-14 15:16:56,650 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 34 2017-09-14 15:16:56,651 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:56,651 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 35 to run in 1 seconds 2017-09-14 15:16:57,661 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 35 2017-09-14 15:16:57,661 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:57,661 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 36 to run in 1 seconds 2017-09-14 15:16:58,650 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 9 2017-09-14 15:16:58,650 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:16:58,650 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:16:58,650 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:16:58,650 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:16:58,650 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:16:58,650 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:16:58,650 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 10 to run in 10 seconds 2017-09-14 15:16:58,662 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 36 2017-09-14 15:16:58,662 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:58,662 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 37 to run in 1 seconds 2017-09-14 15:16:59,668 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 37 2017-09-14 15:16:59,669 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:16:59,669 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 38 to run in 1 seconds 2017-09-14 15:17:00,680 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 38 2017-09-14 15:17:00,681 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:00,681 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 39 to run in 1 seconds 2017-09-14 15:17:01,687 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 39 2017-09-14 15:17:01,687 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:01,687 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 40 to run in 1 seconds 2017-09-14 15:17:02,687 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 40 2017-09-14 15:17:02,688 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:02,688 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 41 to run in 1 seconds 2017-09-14 15:17:03,693 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 41 2017-09-14 15:17:03,693 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:03,693 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 42 to run in 1 seconds 2017-09-14 15:17:03,821 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:17:03,823 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:17:03,823 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:17:04,702 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 42 2017-09-14 15:17:04,702 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:04,702 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 43 to run in 1 seconds 2017-09-14 15:17:05,708 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 43 2017-09-14 15:17:05,709 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:05,709 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 44 to run in 1 seconds 2017-09-14 15:17:06,713 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 44 2017-09-14 15:17:06,714 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:06,714 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 45 to run in 1 seconds 2017-09-14 15:17:07,720 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 45 2017-09-14 15:17:07,720 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:07,721 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 46 to run in 1 seconds 2017-09-14 15:17:08,655 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 10 2017-09-14 15:17:08,656 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:17:08,656 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:17:08,656 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:17:08,656 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:17:08,656 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:17:08,656 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:17:08,657 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 11 to run in 10 seconds 2017-09-14 15:17:08,733 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 46 2017-09-14 15:17:08,734 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:08,734 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 47 to run in 1 seconds 2017-09-14 15:17:08,956 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1C got 1 messages through listener 2017-09-14 15:17:08,957 | DEBUG | node.py (2236) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1505402182349581 2017-09-14 15:17:08,957 | TRACE | node.py (1388) | validateClientMsg | Node1C received CLIENT message: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:17:08,958 | DEBUG | node.py (1434) | processClientInBox | Node1C processing b'v#ZljK34WWooA6%RgESn.Ox' request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:17:08,958 | DEBUG | node.py (1708) | processRequest | Node1 received client request: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:17:08,958 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:17:08,958 | DEBUG | propagator.py ( 246) | tryForwarding | Node1 not forwarding request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} to its replicas since already forwarded 2017-09-14 15:17:08,958 | TRACE | zstack.py ( 754) | transmitThroughListener | Node1C transmitting b'{"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402182349581,"op":"REQACK"}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:17:09,746 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 47 2017-09-14 15:17:09,747 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:09,747 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 48 to run in 1 seconds 2017-09-14 15:17:10,752 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 48 2017-09-14 15:17:10,753 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:10,753 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 49 to run in 1 seconds 2017-09-14 15:17:11,760 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 49 2017-09-14 15:17:11,760 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:11,760 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 50 to run in 1 seconds 2017-09-14 15:17:12,766 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 50 2017-09-14 15:17:12,767 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:12,767 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 51 to run in 1 seconds 2017-09-14 15:17:13,770 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 51 2017-09-14 15:17:13,770 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:13,770 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 52 to run in 1 seconds 2017-09-14 15:17:14,776 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 52 2017-09-14 15:17:14,776 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:14,776 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 53 to run in 1 seconds 2017-09-14 15:17:15,787 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 53 2017-09-14 15:17:15,787 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:15,788 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 54 to run in 1 seconds 2017-09-14 15:17:16,788 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 54 2017-09-14 15:17:16,788 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:16,789 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 55 to run in 1 seconds 2017-09-14 15:17:17,802 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 55 2017-09-14 15:17:17,802 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:17,802 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 56 to run in 1 seconds 2017-09-14 15:17:18,621 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkNodeRequestSpike with id 2 2017-09-14 15:17:18,621 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node1 checking its request amount 2017-09-14 15:17:18,622 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:17:18,622 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkNodeRequestSpike with id 12 to run in 60 seconds 2017-09-14 15:17:18,622 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 1 2017-09-14 15:17:18,622 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-09-14 15:17:18,622 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 2 to run in 60 seconds 2017-09-14 15:17:18,668 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 11 2017-09-14 15:17:18,668 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:17:18,668 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:17:18,669 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:17:18,669 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:17:18,669 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:17:18,669 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:17:18,669 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 13 to run in 10 seconds 2017-09-14 15:17:18,798 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action dump_json_file with id 5 2017-09-14 15:17:18,799 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action dump_json_file with id 14 to run in 60 seconds 2017-09-14 15:17:18,799 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action propose_view_change with id 3 2017-09-14 15:17:18,799 | TRACE | node.py (2097) | propose_view_change | The primary is already connected so view change will not be proposed 2017-09-14 15:17:18,811 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 56 2017-09-14 15:17:18,811 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:18,812 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 57 to run in 1 seconds 2017-09-14 15:17:18,824 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:17:18,824 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:17:18,824 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:17:18,908 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action _check_view_change_completed with id 6 2017-09-14 15:17:18,909 | DEBUG | node.py (1011) | _check_view_change_completed | Node1 running the scheduled check for view change completion 2017-09-14 15:17:18,909 | DEBUG | node.py (1013) | _check_view_change_completed | Node1 already completion view change 2017-09-14 15:17:19,815 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 57 2017-09-14 15:17:19,815 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:19,815 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 58 to run in 1 seconds 2017-09-14 15:17:20,819 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 58 2017-09-14 15:17:20,820 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:20,820 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 59 to run in 1 seconds 2017-09-14 15:17:21,824 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 59 2017-09-14 15:17:21,825 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:21,825 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 60 to run in 1 seconds 2017-09-14 15:17:22,835 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 60 2017-09-14 15:17:22,835 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:22,835 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 61 to run in 1 seconds 2017-09-14 15:17:23,840 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 61 2017-09-14 15:17:23,841 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:23,841 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 62 to run in 1 seconds 2017-09-14 15:17:23,970 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1C got 1 messages through listener 2017-09-14 15:17:23,972 | DEBUG | node.py (2236) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1505402182349581 2017-09-14 15:17:23,972 | TRACE | node.py (1388) | validateClientMsg | Node1C received CLIENT message: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:17:23,972 | DEBUG | node.py (1434) | processClientInBox | Node1C processing b'v#ZljK34WWooA6%RgESn.Ox' request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:17:23,972 | DEBUG | node.py (1708) | processRequest | Node1 received client request: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:17:23,973 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:17:23,973 | DEBUG | propagator.py ( 246) | tryForwarding | Node1 not forwarding request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} to its replicas since already forwarded 2017-09-14 15:17:23,973 | TRACE | zstack.py ( 754) | transmitThroughListener | Node1C transmitting b'{"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402182349581,"op":"REQACK"}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:17:24,845 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 62 2017-09-14 15:17:24,845 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:24,845 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 63 to run in 1 seconds 2017-09-14 15:17:25,851 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 63 2017-09-14 15:17:25,851 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:25,852 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 64 to run in 1 seconds 2017-09-14 15:17:26,861 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 64 2017-09-14 15:17:26,861 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:26,861 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 65 to run in 1 seconds 2017-09-14 15:17:27,870 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 65 2017-09-14 15:17:27,871 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:27,871 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 66 to run in 1 seconds 2017-09-14 15:17:28,677 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 13 2017-09-14 15:17:28,678 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:17:28,678 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:17:28,678 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:17:28,678 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:17:28,678 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:17:28,678 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:17:28,679 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 15 to run in 10 seconds 2017-09-14 15:17:28,880 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 66 2017-09-14 15:17:28,880 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:28,880 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 67 to run in 1 seconds 2017-09-14 15:17:29,886 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 67 2017-09-14 15:17:29,886 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:29,887 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 68 to run in 1 seconds 2017-09-14 15:17:30,892 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 68 2017-09-14 15:17:30,892 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:30,892 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 69 to run in 1 seconds 2017-09-14 15:17:31,901 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 69 2017-09-14 15:17:31,901 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:31,901 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 70 to run in 1 seconds 2017-09-14 15:17:32,904 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 70 2017-09-14 15:17:32,904 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:32,904 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 71 to run in 1 seconds 2017-09-14 15:17:33,832 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:17:33,832 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:17:33,832 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:17:33,915 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 71 2017-09-14 15:17:33,915 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:33,915 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 72 to run in 1 seconds 2017-09-14 15:17:34,920 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 72 2017-09-14 15:17:34,921 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:34,921 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 73 to run in 1 seconds 2017-09-14 15:17:35,921 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 73 2017-09-14 15:17:35,921 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:35,921 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 74 to run in 1 seconds 2017-09-14 15:17:36,925 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 74 2017-09-14 15:17:36,925 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:36,925 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 75 to run in 1 seconds 2017-09-14 15:17:37,927 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 75 2017-09-14 15:17:37,927 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:37,927 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 76 to run in 1 seconds 2017-09-14 15:17:38,688 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 15 2017-09-14 15:17:38,688 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:17:38,688 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:17:38,689 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:17:38,689 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:17:38,689 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:17:38,689 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:17:38,689 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 16 to run in 10 seconds 2017-09-14 15:17:38,940 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 76 2017-09-14 15:17:38,940 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:38,940 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 77 to run in 1 seconds 2017-09-14 15:17:38,994 | TRACE | zstack.py ( 479) | _receiveFromListener | Node1C got 1 messages through listener 2017-09-14 15:17:38,996 | DEBUG | node.py (2236) | verifySignature | Node1 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1505402182349581 2017-09-14 15:17:38,996 | TRACE | node.py (1388) | validateClientMsg | Node1C received CLIENT message: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:17:38,996 | DEBUG | node.py (1434) | processClientInBox | Node1C processing b'v#ZljK34WWooA6%RgESn.Ox' request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:17:38,996 | DEBUG | node.py (1708) | processRequest | Node1 received client request: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} from b'v#ZljK34WWooA6%RgESn.Ox' 2017-09-14 15:17:38,996 | TRACE | propagator.py ( 141) | propagate | Node1 already propagated SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} 2017-09-14 15:17:38,997 | DEBUG | propagator.py ( 246) | tryForwarding | Node1 not forwarding request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe67', 'type': '1'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1505402182349581, 'signature': '5iwkCfgGQPCWBdn5RGQ76fbYqaB9sPR8SLBK8UynSK56DZFoVRQTrsfXCyhYeDsu7QN366QrVp3yK5ELAtWSkGrb'} to its replicas since already forwarded 2017-09-14 15:17:38,997 | TRACE | zstack.py ( 754) | transmitThroughListener | Node1C transmitting b'{"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1505402182349581,"op":"REQACK"}' to b'v#ZljK34WWooA6%RgESn.Ox' through listener socket 2017-09-14 15:17:39,942 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 77 2017-09-14 15:17:39,942 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:39,942 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 78 to run in 1 seconds 2017-09-14 15:17:40,944 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 78 2017-09-14 15:17:40,945 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:40,945 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 79 to run in 1 seconds 2017-09-14 15:17:41,947 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 79 2017-09-14 15:17:41,947 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:41,948 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 80 to run in 1 seconds 2017-09-14 15:17:42,951 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 80 2017-09-14 15:17:42,951 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:42,951 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 81 to run in 1 seconds 2017-09-14 15:17:43,952 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 81 2017-09-14 15:17:43,952 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:43,952 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 82 to run in 1 seconds 2017-09-14 15:17:44,960 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 82 2017-09-14 15:17:44,960 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:44,960 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 83 to run in 1 seconds 2017-09-14 15:17:45,964 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 83 2017-09-14 15:17:45,965 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:45,965 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 84 to run in 1 seconds 2017-09-14 15:17:46,978 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 84 2017-09-14 15:17:46,978 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:46,978 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 85 to run in 1 seconds 2017-09-14 15:17:47,986 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 85 2017-09-14 15:17:47,987 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:47,987 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 86 to run in 1 seconds 2017-09-14 15:17:48,691 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 16 2017-09-14 15:17:48,691 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:17:48,691 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:17:48,691 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:17:48,691 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:17:48,691 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:17:48,692 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:17:48,692 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 17 to run in 10 seconds 2017-09-14 15:17:48,842 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:17:48,842 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:17:48,843 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:17:48,997 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 86 2017-09-14 15:17:48,997 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:48,998 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 87 to run in 1 seconds 2017-09-14 15:17:49,998 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 87 2017-09-14 15:17:49,999 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:49,999 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 88 to run in 1 seconds 2017-09-14 15:17:51,000 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 88 2017-09-14 15:17:51,000 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:51,000 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 89 to run in 1 seconds 2017-09-14 15:17:52,008 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 89 2017-09-14 15:17:52,008 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:52,008 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 90 to run in 1 seconds 2017-09-14 15:17:53,018 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 90 2017-09-14 15:17:53,018 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:53,018 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 91 to run in 1 seconds 2017-09-14 15:17:54,027 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 91 2017-09-14 15:17:54,027 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:54,028 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 92 to run in 1 seconds 2017-09-14 15:17:55,033 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 92 2017-09-14 15:17:55,033 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:55,033 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 93 to run in 1 seconds 2017-09-14 15:17:56,042 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 93 2017-09-14 15:17:56,042 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:56,042 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 94 to run in 1 seconds 2017-09-14 15:17:57,044 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 94 2017-09-14 15:17:57,044 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:57,044 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 95 to run in 1 seconds 2017-09-14 15:17:58,054 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 95 2017-09-14 15:17:58,055 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:58,055 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 96 to run in 1 seconds 2017-09-14 15:17:58,703 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 17 2017-09-14 15:17:58,703 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:17:58,704 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:17:58,704 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:17:58,704 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:17:58,704 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:17:58,704 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:17:58,704 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 18 to run in 10 seconds 2017-09-14 15:17:59,064 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 96 2017-09-14 15:17:59,064 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:17:59,064 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 97 to run in 1 seconds 2017-09-14 15:18:00,071 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 97 2017-09-14 15:18:00,071 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:00,071 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 98 to run in 1 seconds 2017-09-14 15:18:01,083 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 98 2017-09-14 15:18:01,083 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:01,083 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 99 to run in 1 seconds 2017-09-14 15:18:02,085 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 99 2017-09-14 15:18:02,086 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:02,086 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 100 to run in 1 seconds 2017-09-14 15:18:03,094 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 100 2017-09-14 15:18:03,094 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:03,095 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 101 to run in 1 seconds 2017-09-14 15:18:03,851 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:18:03,851 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:18:03,851 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:18:04,095 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 101 2017-09-14 15:18:04,095 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:04,095 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 102 to run in 1 seconds 2017-09-14 15:18:05,100 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 102 2017-09-14 15:18:05,100 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:05,100 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 103 to run in 1 seconds 2017-09-14 15:18:06,108 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 103 2017-09-14 15:18:06,108 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:06,108 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 104 to run in 1 seconds 2017-09-14 15:18:07,117 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 104 2017-09-14 15:18:07,117 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:07,117 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 105 to run in 1 seconds 2017-09-14 15:18:08,128 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 105 2017-09-14 15:18:08,129 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:08,129 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 106 to run in 1 seconds 2017-09-14 15:18:08,715 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 18 2017-09-14 15:18:08,716 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:18:08,716 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:18:08,716 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:18:08,716 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:18:08,716 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:18:08,716 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:18:08,716 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 19 to run in 10 seconds 2017-09-14 15:18:09,140 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 106 2017-09-14 15:18:09,141 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:09,141 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 107 to run in 1 seconds 2017-09-14 15:18:10,149 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 107 2017-09-14 15:18:10,150 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:10,150 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 108 to run in 1 seconds 2017-09-14 15:18:11,161 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 108 2017-09-14 15:18:11,161 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:11,161 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 109 to run in 1 seconds 2017-09-14 15:18:12,162 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 109 2017-09-14 15:18:12,162 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:12,164 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 110 to run in 1 seconds 2017-09-14 15:18:13,166 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 110 2017-09-14 15:18:13,166 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:13,166 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 111 to run in 1 seconds 2017-09-14 15:18:14,167 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 111 2017-09-14 15:18:14,167 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:14,168 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 112 to run in 1 seconds 2017-09-14 15:18:15,170 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 112 2017-09-14 15:18:15,170 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:15,171 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 113 to run in 1 seconds 2017-09-14 15:18:16,175 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 113 2017-09-14 15:18:16,176 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:16,176 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 114 to run in 1 seconds 2017-09-14 15:18:17,180 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 114 2017-09-14 15:18:17,180 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:17,180 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 115 to run in 1 seconds 2017-09-14 15:18:18,190 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 115 2017-09-14 15:18:18,190 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:18,190 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 116 to run in 1 seconds 2017-09-14 15:18:18,638 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkNodeRequestSpike with id 12 2017-09-14 15:18:18,638 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node1 checking its request amount 2017-09-14 15:18:18,638 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:18:18,638 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkNodeRequestSpike with id 20 to run in 60 seconds 2017-09-14 15:18:18,638 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 2 2017-09-14 15:18:18,638 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-09-14 15:18:18,639 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 3 to run in 60 seconds 2017-09-14 15:18:18,722 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 19 2017-09-14 15:18:18,722 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:18:18,723 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:18:18,723 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:18:18,723 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:18:18,723 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:18:18,723 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:18:18,723 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 21 to run in 10 seconds 2017-09-14 15:18:18,808 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action dump_json_file with id 14 2017-09-14 15:18:18,809 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action dump_json_file with id 22 to run in 60 seconds 2017-09-14 15:18:18,859 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:18:18,859 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:18:18,859 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:18:19,201 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 116 2017-09-14 15:18:19,202 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:19,202 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 117 to run in 1 seconds 2017-09-14 15:18:20,202 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 117 2017-09-14 15:18:20,205 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:20,205 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 118 to run in 1 seconds 2017-09-14 15:18:21,215 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 118 2017-09-14 15:18:21,215 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:21,216 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 119 to run in 1 seconds 2017-09-14 15:18:22,217 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 119 2017-09-14 15:18:22,217 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:22,217 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 120 to run in 1 seconds 2017-09-14 15:18:23,228 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 120 2017-09-14 15:18:23,228 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:23,228 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 121 to run in 1 seconds 2017-09-14 15:18:24,232 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 121 2017-09-14 15:18:24,233 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:24,233 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 122 to run in 1 seconds 2017-09-14 15:18:25,236 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 122 2017-09-14 15:18:25,236 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:25,236 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 123 to run in 1 seconds 2017-09-14 15:18:26,250 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 123 2017-09-14 15:18:26,250 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:26,250 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 124 to run in 1 seconds 2017-09-14 15:18:27,258 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 124 2017-09-14 15:18:27,258 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:27,258 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 125 to run in 1 seconds 2017-09-14 15:18:28,266 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 125 2017-09-14 15:18:28,266 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:28,266 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 126 to run in 1 seconds 2017-09-14 15:18:28,726 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 21 2017-09-14 15:18:28,726 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:18:28,727 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:18:28,727 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:18:28,727 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:18:28,727 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:18:28,727 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:18:28,727 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 23 to run in 10 seconds 2017-09-14 15:18:29,273 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 126 2017-09-14 15:18:29,273 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:29,273 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 127 to run in 1 seconds 2017-09-14 15:18:30,279 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 127 2017-09-14 15:18:30,279 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:30,279 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 128 to run in 1 seconds 2017-09-14 15:18:31,292 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 128 2017-09-14 15:18:31,292 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:31,292 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 129 to run in 1 seconds 2017-09-14 15:18:32,295 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 129 2017-09-14 15:18:32,295 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:32,295 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 130 to run in 1 seconds 2017-09-14 15:18:33,296 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 130 2017-09-14 15:18:33,296 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:33,296 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 131 to run in 1 seconds 2017-09-14 15:18:33,868 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:18:33,868 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:18:33,869 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:18:34,299 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 131 2017-09-14 15:18:34,299 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:34,300 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 132 to run in 1 seconds 2017-09-14 15:18:35,300 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 132 2017-09-14 15:18:35,301 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:35,301 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 133 to run in 1 seconds 2017-09-14 15:18:36,314 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 133 2017-09-14 15:18:36,315 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:36,315 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 134 to run in 1 seconds 2017-09-14 15:18:37,317 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 134 2017-09-14 15:18:37,317 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:37,317 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 135 to run in 1 seconds 2017-09-14 15:18:38,329 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 135 2017-09-14 15:18:38,329 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:38,330 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 136 to run in 1 seconds 2017-09-14 15:18:38,733 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 23 2017-09-14 15:18:38,734 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:18:38,734 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:18:38,734 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:18:38,734 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:18:38,734 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:18:38,734 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:18:38,734 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 24 to run in 10 seconds 2017-09-14 15:18:39,331 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 136 2017-09-14 15:18:39,331 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:39,331 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 137 to run in 1 seconds 2017-09-14 15:18:40,339 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 137 2017-09-14 15:18:40,340 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:40,340 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 138 to run in 1 seconds 2017-09-14 15:18:41,345 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 138 2017-09-14 15:18:41,345 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:41,345 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 139 to run in 1 seconds 2017-09-14 15:18:42,354 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 139 2017-09-14 15:18:42,354 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:42,354 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 140 to run in 1 seconds 2017-09-14 15:18:43,358 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 140 2017-09-14 15:18:43,358 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:43,358 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 141 to run in 1 seconds 2017-09-14 15:18:44,364 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 141 2017-09-14 15:18:44,364 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:44,365 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 142 to run in 1 seconds 2017-09-14 15:18:45,368 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 142 2017-09-14 15:18:45,368 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:45,368 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 143 to run in 1 seconds 2017-09-14 15:18:46,373 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 143 2017-09-14 15:18:46,373 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:46,373 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 144 to run in 1 seconds 2017-09-14 15:18:47,381 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 144 2017-09-14 15:18:47,381 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:47,382 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 145 to run in 1 seconds 2017-09-14 15:18:48,382 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 145 2017-09-14 15:18:48,383 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:48,383 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 146 to run in 1 seconds 2017-09-14 15:18:48,745 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 24 2017-09-14 15:18:48,745 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:18:48,746 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:18:48,746 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:18:48,746 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:18:48,746 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:18:48,746 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:18:48,746 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 25 to run in 10 seconds 2017-09-14 15:18:48,872 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:18:48,872 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:18:48,873 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:18:49,394 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 146 2017-09-14 15:18:49,394 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:49,394 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 147 to run in 1 seconds 2017-09-14 15:18:50,401 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 147 2017-09-14 15:18:50,401 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:50,401 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 148 to run in 1 seconds 2017-09-14 15:18:51,408 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 148 2017-09-14 15:18:51,408 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:51,408 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 149 to run in 1 seconds 2017-09-14 15:18:52,416 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 149 2017-09-14 15:18:52,417 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:52,417 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 150 to run in 1 seconds 2017-09-14 15:18:53,423 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 150 2017-09-14 15:18:53,424 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:53,424 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 151 to run in 1 seconds 2017-09-14 15:18:54,430 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 151 2017-09-14 15:18:54,430 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:54,430 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 152 to run in 1 seconds 2017-09-14 15:18:55,435 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 152 2017-09-14 15:18:55,435 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:55,436 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 153 to run in 1 seconds 2017-09-14 15:18:56,436 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 153 2017-09-14 15:18:56,437 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:56,437 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 154 to run in 1 seconds 2017-09-14 15:18:57,439 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 154 2017-09-14 15:18:57,439 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:57,439 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 155 to run in 1 seconds 2017-09-14 15:18:58,446 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 155 2017-09-14 15:18:58,446 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:58,446 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 156 to run in 1 seconds 2017-09-14 15:18:58,750 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 25 2017-09-14 15:18:58,751 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:18:58,751 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:18:58,751 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:18:58,751 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:18:58,751 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:18:58,751 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:18:58,751 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 26 to run in 10 seconds 2017-09-14 15:18:59,452 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 156 2017-09-14 15:18:59,453 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:18:59,453 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 157 to run in 1 seconds 2017-09-14 15:19:00,459 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 157 2017-09-14 15:19:00,459 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:00,459 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 158 to run in 1 seconds 2017-09-14 15:19:01,461 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 158 2017-09-14 15:19:01,461 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:01,461 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 159 to run in 1 seconds 2017-09-14 15:19:02,462 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 159 2017-09-14 15:19:02,462 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:02,462 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 160 to run in 1 seconds 2017-09-14 15:19:03,469 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 160 2017-09-14 15:19:03,469 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:03,469 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 161 to run in 1 seconds 2017-09-14 15:19:03,885 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:19:03,886 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:19:03,886 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:19:04,480 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 161 2017-09-14 15:19:04,481 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:04,481 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 162 to run in 1 seconds 2017-09-14 15:19:05,491 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 162 2017-09-14 15:19:05,492 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:05,492 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 163 to run in 1 seconds 2017-09-14 15:19:06,493 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 163 2017-09-14 15:19:06,493 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:06,494 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 164 to run in 1 seconds 2017-09-14 15:19:07,506 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 164 2017-09-14 15:19:07,506 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:07,506 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 165 to run in 1 seconds 2017-09-14 15:19:08,516 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 165 2017-09-14 15:19:08,516 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:08,516 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 166 to run in 1 seconds 2017-09-14 15:19:08,755 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 26 2017-09-14 15:19:08,755 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:19:08,755 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:19:08,756 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:19:08,756 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:19:08,756 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:19:08,756 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:19:08,756 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 27 to run in 10 seconds 2017-09-14 15:19:09,527 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 166 2017-09-14 15:19:09,527 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:09,527 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 167 to run in 1 seconds 2017-09-14 15:19:10,538 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 167 2017-09-14 15:19:10,539 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:10,539 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 168 to run in 1 seconds 2017-09-14 15:19:11,544 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 168 2017-09-14 15:19:11,544 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:11,544 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 169 to run in 1 seconds 2017-09-14 15:19:12,551 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 169 2017-09-14 15:19:12,551 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:12,551 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 170 to run in 1 seconds 2017-09-14 15:19:13,553 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 170 2017-09-14 15:19:13,554 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:13,554 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 171 to run in 1 seconds 2017-09-14 15:19:14,560 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 171 2017-09-14 15:19:14,560 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:14,560 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 172 to run in 1 seconds 2017-09-14 15:19:15,560 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 172 2017-09-14 15:19:15,561 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:15,561 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 173 to run in 1 seconds 2017-09-14 15:19:16,566 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 173 2017-09-14 15:19:16,567 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:16,567 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 174 to run in 1 seconds 2017-09-14 15:19:17,577 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 174 2017-09-14 15:19:17,577 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:17,577 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 175 to run in 1 seconds 2017-09-14 15:19:18,582 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 175 2017-09-14 15:19:18,582 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:18,582 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 176 to run in 1 seconds 2017-09-14 15:19:18,643 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkNodeRequestSpike with id 20 2017-09-14 15:19:18,644 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node1 checking its request amount 2017-09-14 15:19:18,644 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:19:18,644 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkNodeRequestSpike with id 28 to run in 60 seconds 2017-09-14 15:19:18,644 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 3 2017-09-14 15:19:18,644 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-09-14 15:19:18,644 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 4 to run in 60 seconds 2017-09-14 15:19:18,767 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 27 2017-09-14 15:19:18,768 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:19:18,768 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:19:18,768 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:19:18,768 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:19:18,768 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:19:18,768 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:19:18,768 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 29 to run in 10 seconds 2017-09-14 15:19:18,811 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action dump_json_file with id 22 2017-09-14 15:19:18,813 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action dump_json_file with id 30 to run in 60 seconds 2017-09-14 15:19:18,891 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:19:18,892 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:19:18,892 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:19:19,589 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 176 2017-09-14 15:19:19,590 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:19,590 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 177 to run in 1 seconds 2017-09-14 15:19:20,598 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 177 2017-09-14 15:19:20,598 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:20,599 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 178 to run in 1 seconds 2017-09-14 15:19:21,602 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 178 2017-09-14 15:19:21,602 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:21,602 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 179 to run in 1 seconds 2017-09-14 15:19:22,611 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 179 2017-09-14 15:19:22,611 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:22,612 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 180 to run in 1 seconds 2017-09-14 15:19:23,623 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 180 2017-09-14 15:19:23,624 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:23,624 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 181 to run in 1 seconds 2017-09-14 15:19:24,634 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 181 2017-09-14 15:19:24,635 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:24,635 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 182 to run in 1 seconds 2017-09-14 15:19:25,646 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 182 2017-09-14 15:19:25,647 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:25,647 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 183 to run in 1 seconds 2017-09-14 15:19:26,655 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 183 2017-09-14 15:19:26,656 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:26,656 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 184 to run in 1 seconds 2017-09-14 15:19:27,664 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 184 2017-09-14 15:19:27,664 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:27,664 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 185 to run in 1 seconds 2017-09-14 15:19:28,675 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 185 2017-09-14 15:19:28,675 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:28,675 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 186 to run in 1 seconds 2017-09-14 15:19:28,774 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 29 2017-09-14 15:19:28,775 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:19:28,775 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:19:28,775 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:19:28,775 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:19:28,775 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:19:28,775 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:19:28,775 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 31 to run in 10 seconds 2017-09-14 15:19:29,685 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 186 2017-09-14 15:19:29,686 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:29,686 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 187 to run in 1 seconds 2017-09-14 15:19:30,688 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 187 2017-09-14 15:19:30,688 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:30,689 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 188 to run in 1 seconds 2017-09-14 15:19:31,693 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 188 2017-09-14 15:19:31,693 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:31,693 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 189 to run in 1 seconds 2017-09-14 15:19:32,696 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 189 2017-09-14 15:19:32,696 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:32,697 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 190 to run in 1 seconds 2017-09-14 15:19:33,702 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 190 2017-09-14 15:19:33,702 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:33,702 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 191 to run in 1 seconds 2017-09-14 15:19:33,903 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:19:33,903 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:19:33,903 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:19:34,710 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 191 2017-09-14 15:19:34,711 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:34,711 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 192 to run in 1 seconds 2017-09-14 15:19:35,711 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 192 2017-09-14 15:19:35,711 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:35,712 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 193 to run in 1 seconds 2017-09-14 15:19:36,715 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 193 2017-09-14 15:19:36,715 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:36,716 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 194 to run in 1 seconds 2017-09-14 15:19:37,724 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 194 2017-09-14 15:19:37,724 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:37,725 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 195 to run in 1 seconds 2017-09-14 15:19:38,737 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 195 2017-09-14 15:19:38,737 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:38,738 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 196 to run in 1 seconds 2017-09-14 15:19:38,788 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 31 2017-09-14 15:19:38,788 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:19:38,788 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:19:38,788 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:19:38,789 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:19:38,789 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:19:38,789 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:19:38,789 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 32 to run in 10 seconds 2017-09-14 15:19:39,745 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 196 2017-09-14 15:19:39,745 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:39,745 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 197 to run in 1 seconds 2017-09-14 15:19:40,748 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 197 2017-09-14 15:19:40,748 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:40,749 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 198 to run in 1 seconds 2017-09-14 15:19:41,754 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 198 2017-09-14 15:19:41,754 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:41,754 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 199 to run in 1 seconds 2017-09-14 15:19:42,758 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 199 2017-09-14 15:19:42,758 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:42,758 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 200 to run in 1 seconds 2017-09-14 15:19:43,769 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 200 2017-09-14 15:19:43,769 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:43,769 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 201 to run in 1 seconds 2017-09-14 15:19:44,779 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 201 2017-09-14 15:19:44,780 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:44,780 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 202 to run in 1 seconds 2017-09-14 15:19:45,790 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 202 2017-09-14 15:19:45,790 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:45,790 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 203 to run in 1 seconds 2017-09-14 15:19:46,796 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 203 2017-09-14 15:19:46,796 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:46,796 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 204 to run in 1 seconds 2017-09-14 15:19:47,802 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 204 2017-09-14 15:19:47,802 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:47,802 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 205 to run in 1 seconds 2017-09-14 15:19:48,792 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 32 2017-09-14 15:19:48,792 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:19:48,792 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:19:48,792 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:19:48,792 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:19:48,792 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:19:48,793 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:19:48,793 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 33 to run in 10 seconds 2017-09-14 15:19:48,805 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 205 2017-09-14 15:19:48,805 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:48,805 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 206 to run in 1 seconds 2017-09-14 15:19:48,915 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:19:48,915 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:19:48,916 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:19:49,816 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 206 2017-09-14 15:19:49,816 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:49,816 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 207 to run in 1 seconds 2017-09-14 15:19:50,827 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 207 2017-09-14 15:19:50,828 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:50,828 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 208 to run in 1 seconds 2017-09-14 15:19:51,833 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 208 2017-09-14 15:19:51,834 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:51,834 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 209 to run in 1 seconds 2017-09-14 15:19:52,837 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 209 2017-09-14 15:19:52,838 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:52,838 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 210 to run in 1 seconds 2017-09-14 15:19:53,839 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 210 2017-09-14 15:19:53,839 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:53,840 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 211 to run in 1 seconds 2017-09-14 15:19:54,841 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 211 2017-09-14 15:19:54,841 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:54,841 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 212 to run in 1 seconds 2017-09-14 15:19:55,843 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 212 2017-09-14 15:19:55,843 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:55,843 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 213 to run in 1 seconds 2017-09-14 15:19:56,848 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 213 2017-09-14 15:19:56,848 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:56,849 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 214 to run in 1 seconds 2017-09-14 15:19:57,858 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 214 2017-09-14 15:19:57,858 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:57,858 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 215 to run in 1 seconds 2017-09-14 15:19:58,800 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 33 2017-09-14 15:19:58,800 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:19:58,800 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:19:58,800 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:19:58,800 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:19:58,800 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:19:58,800 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:19:58,800 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 34 to run in 10 seconds 2017-09-14 15:19:58,863 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 215 2017-09-14 15:19:58,863 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:58,863 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 216 to run in 1 seconds 2017-09-14 15:19:59,874 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 216 2017-09-14 15:19:59,875 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:19:59,875 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 217 to run in 1 seconds 2017-09-14 15:20:00,882 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 217 2017-09-14 15:20:00,882 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:00,883 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 218 to run in 1 seconds 2017-09-14 15:20:01,888 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 218 2017-09-14 15:20:01,888 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:01,888 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 219 to run in 1 seconds 2017-09-14 15:20:02,889 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 219 2017-09-14 15:20:02,890 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:02,890 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 220 to run in 1 seconds 2017-09-14 15:20:03,901 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 220 2017-09-14 15:20:03,901 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:03,901 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 221 to run in 1 seconds 2017-09-14 15:20:03,931 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:20:03,931 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:20:03,931 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:20:04,912 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 221 2017-09-14 15:20:04,913 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:04,913 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 222 to run in 1 seconds 2017-09-14 15:20:05,915 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 222 2017-09-14 15:20:05,915 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:05,915 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 223 to run in 1 seconds 2017-09-14 15:20:06,919 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 223 2017-09-14 15:20:06,920 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:06,920 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 224 to run in 1 seconds 2017-09-14 15:20:07,921 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 224 2017-09-14 15:20:07,922 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:07,922 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 225 to run in 1 seconds 2017-09-14 15:20:08,807 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 34 2017-09-14 15:20:08,807 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:20:08,808 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:20:08,808 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:20:08,808 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:20:08,808 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:20:08,808 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:20:08,808 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 35 to run in 10 seconds 2017-09-14 15:20:08,926 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 225 2017-09-14 15:20:08,927 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:08,927 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 226 to run in 1 seconds 2017-09-14 15:20:09,927 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 226 2017-09-14 15:20:09,927 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:09,929 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 227 to run in 1 seconds 2017-09-14 15:20:10,938 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 227 2017-09-14 15:20:10,938 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:10,938 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 228 to run in 1 seconds 2017-09-14 15:20:11,947 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 228 2017-09-14 15:20:11,947 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:11,947 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 229 to run in 1 seconds 2017-09-14 15:20:12,958 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 229 2017-09-14 15:20:12,958 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:12,958 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 230 to run in 1 seconds 2017-09-14 15:20:13,969 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 230 2017-09-14 15:20:13,969 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:13,969 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 231 to run in 1 seconds 2017-09-14 15:20:14,970 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 231 2017-09-14 15:20:14,970 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:14,970 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 232 to run in 1 seconds 2017-09-14 15:20:15,973 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 232 2017-09-14 15:20:15,974 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:15,974 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 233 to run in 1 seconds 2017-09-14 15:20:16,985 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 233 2017-09-14 15:20:16,985 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:16,986 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 234 to run in 1 seconds 2017-09-14 15:20:17,989 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 234 2017-09-14 15:20:17,989 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:17,990 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 235 to run in 1 seconds 2017-09-14 15:20:18,646 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkNodeRequestSpike with id 28 2017-09-14 15:20:18,647 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node1 checking its request amount 2017-09-14 15:20:18,647 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:20:18,647 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkNodeRequestSpike with id 36 to run in 60 seconds 2017-09-14 15:20:18,647 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 4 2017-09-14 15:20:18,647 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-09-14 15:20:18,647 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 5 to run in 60 seconds 2017-09-14 15:20:18,818 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 35 2017-09-14 15:20:18,818 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:20:18,818 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:20:18,818 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:20:18,818 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:20:18,819 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:20:18,819 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:20:18,819 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 37 to run in 10 seconds 2017-09-14 15:20:18,819 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action dump_json_file with id 30 2017-09-14 15:20:18,819 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action dump_json_file with id 38 to run in 60 seconds 2017-09-14 15:20:18,936 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:20:18,936 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:20:18,937 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:20:18,998 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 235 2017-09-14 15:20:18,998 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:18,998 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 236 to run in 1 seconds 2017-09-14 15:20:20,008 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 236 2017-09-14 15:20:20,008 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:20,008 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 237 to run in 1 seconds 2017-09-14 15:20:21,011 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 237 2017-09-14 15:20:21,011 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:21,011 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 238 to run in 1 seconds 2017-09-14 15:20:22,013 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 238 2017-09-14 15:20:22,014 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:22,014 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 239 to run in 1 seconds 2017-09-14 15:20:23,022 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 239 2017-09-14 15:20:23,022 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:23,022 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 240 to run in 1 seconds 2017-09-14 15:20:24,024 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 240 2017-09-14 15:20:24,024 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:24,024 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 241 to run in 1 seconds 2017-09-14 15:20:25,032 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 241 2017-09-14 15:20:25,032 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:25,032 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 242 to run in 1 seconds 2017-09-14 15:20:26,043 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 242 2017-09-14 15:20:26,043 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:26,043 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 243 to run in 1 seconds 2017-09-14 15:20:27,044 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 243 2017-09-14 15:20:27,044 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:27,044 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 244 to run in 1 seconds 2017-09-14 15:20:28,045 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 244 2017-09-14 15:20:28,045 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:28,045 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 245 to run in 1 seconds 2017-09-14 15:20:28,821 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 37 2017-09-14 15:20:28,821 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:20:28,821 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:20:28,821 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:20:28,821 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:20:28,821 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:20:28,821 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:20:28,821 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 39 to run in 10 seconds 2017-09-14 15:20:29,054 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 245 2017-09-14 15:20:29,054 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:29,054 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 246 to run in 1 seconds 2017-09-14 15:20:30,062 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 246 2017-09-14 15:20:30,062 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:30,063 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 247 to run in 1 seconds 2017-09-14 15:20:31,073 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 247 2017-09-14 15:20:31,073 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:31,073 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 248 to run in 1 seconds 2017-09-14 15:20:32,084 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 248 2017-09-14 15:20:32,084 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:32,084 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 249 to run in 1 seconds 2017-09-14 15:20:33,088 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 249 2017-09-14 15:20:33,089 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:33,089 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 250 to run in 1 seconds 2017-09-14 15:20:33,946 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:20:33,946 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:20:33,947 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:20:34,101 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 250 2017-09-14 15:20:34,101 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:34,101 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 251 to run in 1 seconds 2017-09-14 15:20:35,115 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 251 2017-09-14 15:20:35,115 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:35,115 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 252 to run in 1 seconds 2017-09-14 15:20:36,117 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 252 2017-09-14 15:20:36,118 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:36,118 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 253 to run in 1 seconds 2017-09-14 15:20:37,122 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 253 2017-09-14 15:20:37,122 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:37,122 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 254 to run in 1 seconds 2017-09-14 15:20:38,136 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 254 2017-09-14 15:20:38,136 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:38,136 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 255 to run in 1 seconds 2017-09-14 15:20:38,822 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 39 2017-09-14 15:20:38,822 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:20:38,823 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:20:38,823 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:20:38,823 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:20:38,823 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:20:38,823 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:20:38,823 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 40 to run in 10 seconds 2017-09-14 15:20:39,142 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 255 2017-09-14 15:20:39,142 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:39,143 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 256 to run in 1 seconds 2017-09-14 15:20:40,143 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 256 2017-09-14 15:20:40,143 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:40,143 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 257 to run in 1 seconds 2017-09-14 15:20:41,156 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 257 2017-09-14 15:20:41,157 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:41,157 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 258 to run in 1 seconds 2017-09-14 15:20:42,167 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 258 2017-09-14 15:20:42,168 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:42,168 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 259 to run in 1 seconds 2017-09-14 15:20:43,182 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 259 2017-09-14 15:20:43,182 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:43,182 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 260 to run in 1 seconds 2017-09-14 15:20:44,194 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 260 2017-09-14 15:20:44,194 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:44,194 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 261 to run in 1 seconds 2017-09-14 15:20:45,197 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 261 2017-09-14 15:20:45,197 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:45,197 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 262 to run in 1 seconds 2017-09-14 15:20:46,205 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 262 2017-09-14 15:20:46,205 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:46,206 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 263 to run in 1 seconds 2017-09-14 15:20:47,211 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 263 2017-09-14 15:20:47,212 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:47,212 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 264 to run in 1 seconds 2017-09-14 15:20:48,215 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 264 2017-09-14 15:20:48,215 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:48,215 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 265 to run in 1 seconds 2017-09-14 15:20:48,827 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 40 2017-09-14 15:20:48,828 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:20:48,828 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:20:48,828 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:20:48,828 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:20:48,828 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:20:48,828 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:20:48,828 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 41 to run in 10 seconds 2017-09-14 15:20:48,953 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:20:48,953 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:20:48,954 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:20:49,226 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 265 2017-09-14 15:20:49,226 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:49,226 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 266 to run in 1 seconds 2017-09-14 15:20:50,235 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 266 2017-09-14 15:20:50,236 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:50,236 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 267 to run in 1 seconds 2017-09-14 15:20:51,237 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 267 2017-09-14 15:20:51,237 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:51,238 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 268 to run in 1 seconds 2017-09-14 15:20:52,245 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 268 2017-09-14 15:20:52,245 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:52,245 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 269 to run in 1 seconds 2017-09-14 15:20:53,255 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 269 2017-09-14 15:20:53,255 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:53,255 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 270 to run in 1 seconds 2017-09-14 15:20:54,266 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 270 2017-09-14 15:20:54,266 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:54,266 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 271 to run in 1 seconds 2017-09-14 15:20:55,267 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 271 2017-09-14 15:20:55,268 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:55,268 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 272 to run in 1 seconds 2017-09-14 15:20:56,268 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 272 2017-09-14 15:20:56,268 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:56,268 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 273 to run in 1 seconds 2017-09-14 15:20:57,277 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 273 2017-09-14 15:20:57,277 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:57,277 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 274 to run in 1 seconds 2017-09-14 15:20:58,287 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 274 2017-09-14 15:20:58,287 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:58,287 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 275 to run in 1 seconds 2017-09-14 15:20:58,830 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 41 2017-09-14 15:20:58,830 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:20:58,830 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:20:58,830 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:20:58,830 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:20:58,830 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:20:58,830 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:20:58,831 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 42 to run in 10 seconds 2017-09-14 15:20:59,292 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 275 2017-09-14 15:20:59,293 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:20:59,293 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 276 to run in 1 seconds 2017-09-14 15:21:00,298 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 276 2017-09-14 15:21:00,298 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:00,299 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 277 to run in 1 seconds 2017-09-14 15:21:01,299 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 277 2017-09-14 15:21:01,299 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:01,299 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 278 to run in 1 seconds 2017-09-14 15:21:02,301 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 278 2017-09-14 15:21:02,301 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:02,301 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 279 to run in 1 seconds 2017-09-14 15:21:03,305 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 279 2017-09-14 15:21:03,308 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:03,308 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 280 to run in 1 seconds 2017-09-14 15:21:03,957 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:21:03,957 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:21:03,957 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:21:04,317 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 280 2017-09-14 15:21:04,317 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:04,317 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 281 to run in 1 seconds 2017-09-14 15:21:05,328 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 281 2017-09-14 15:21:05,328 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:05,328 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 282 to run in 1 seconds 2017-09-14 15:21:06,329 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 282 2017-09-14 15:21:06,330 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:06,330 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 283 to run in 1 seconds 2017-09-14 15:21:07,332 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 283 2017-09-14 15:21:07,332 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:07,332 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 284 to run in 1 seconds 2017-09-14 15:21:08,341 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 284 2017-09-14 15:21:08,341 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:08,341 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 285 to run in 1 seconds 2017-09-14 15:21:08,839 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 42 2017-09-14 15:21:08,840 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:21:08,840 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:21:08,840 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:21:08,840 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:21:08,840 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:21:08,840 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:21:08,840 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 43 to run in 10 seconds 2017-09-14 15:21:09,342 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 285 2017-09-14 15:21:09,342 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:09,343 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 286 to run in 1 seconds 2017-09-14 15:21:10,349 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 286 2017-09-14 15:21:10,349 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:10,349 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 287 to run in 1 seconds 2017-09-14 15:21:11,357 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 287 2017-09-14 15:21:11,357 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:11,357 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 288 to run in 1 seconds 2017-09-14 15:21:12,373 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 288 2017-09-14 15:21:12,373 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:12,373 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 289 to run in 1 seconds 2017-09-14 15:21:13,382 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 289 2017-09-14 15:21:13,382 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:13,383 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 290 to run in 1 seconds 2017-09-14 15:21:14,395 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 290 2017-09-14 15:21:14,395 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:14,395 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 291 to run in 1 seconds 2017-09-14 15:21:15,407 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 291 2017-09-14 15:21:15,407 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:15,407 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 292 to run in 1 seconds 2017-09-14 15:21:16,408 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 292 2017-09-14 15:21:16,411 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:16,411 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 293 to run in 1 seconds 2017-09-14 15:21:17,414 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 293 2017-09-14 15:21:17,414 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:17,414 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 294 to run in 1 seconds 2017-09-14 15:21:18,424 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 294 2017-09-14 15:21:18,425 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:18,425 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 295 to run in 1 seconds 2017-09-14 15:21:18,652 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkNodeRequestSpike with id 36 2017-09-14 15:21:18,652 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node1 checking its request amount 2017-09-14 15:21:18,652 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:21:18,652 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkNodeRequestSpike with id 44 to run in 60 seconds 2017-09-14 15:21:18,652 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 5 2017-09-14 15:21:18,653 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-09-14 15:21:18,653 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 6 to run in 60 seconds 2017-09-14 15:21:18,825 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action dump_json_file with id 38 2017-09-14 15:21:18,826 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action dump_json_file with id 45 to run in 60 seconds 2017-09-14 15:21:18,849 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 43 2017-09-14 15:21:18,850 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:21:18,850 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:21:18,850 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:21:18,850 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:21:18,850 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:21:18,850 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:21:18,850 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 46 to run in 10 seconds 2017-09-14 15:21:18,966 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:21:18,966 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:21:18,967 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:21:19,431 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 295 2017-09-14 15:21:19,432 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:19,432 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 296 to run in 1 seconds 2017-09-14 15:21:20,436 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 296 2017-09-14 15:21:20,436 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:20,436 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 297 to run in 1 seconds 2017-09-14 15:21:21,438 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 297 2017-09-14 15:21:21,438 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:21,438 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 298 to run in 1 seconds 2017-09-14 15:21:22,440 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 298 2017-09-14 15:21:22,441 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:22,441 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 299 to run in 1 seconds 2017-09-14 15:21:23,443 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 299 2017-09-14 15:21:23,443 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:23,443 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 300 to run in 1 seconds 2017-09-14 15:21:24,452 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 300 2017-09-14 15:21:24,453 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:24,453 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 301 to run in 1 seconds 2017-09-14 15:21:25,458 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 301 2017-09-14 15:21:25,458 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:25,459 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 302 to run in 1 seconds 2017-09-14 15:21:26,465 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 302 2017-09-14 15:21:26,465 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:26,465 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 303 to run in 1 seconds 2017-09-14 15:21:27,477 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 303 2017-09-14 15:21:27,477 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:27,477 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 304 to run in 1 seconds 2017-09-14 15:21:28,488 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 304 2017-09-14 15:21:28,488 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:28,488 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 305 to run in 1 seconds 2017-09-14 15:21:28,857 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 46 2017-09-14 15:21:28,857 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:21:28,857 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:21:28,858 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:21:28,858 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:21:28,858 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:21:28,858 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:21:28,858 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 47 to run in 10 seconds 2017-09-14 15:21:29,497 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 305 2017-09-14 15:21:29,497 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:29,497 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 306 to run in 1 seconds 2017-09-14 15:21:30,507 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 306 2017-09-14 15:21:30,507 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:30,507 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 307 to run in 1 seconds 2017-09-14 15:21:31,519 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 307 2017-09-14 15:21:31,519 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:31,519 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 308 to run in 1 seconds 2017-09-14 15:21:32,523 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 308 2017-09-14 15:21:32,523 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:32,523 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 309 to run in 1 seconds 2017-09-14 15:21:33,526 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 309 2017-09-14 15:21:33,527 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:33,527 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 310 to run in 1 seconds 2017-09-14 15:21:33,967 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:21:33,967 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:21:33,967 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:21:34,535 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 310 2017-09-14 15:21:34,536 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:34,536 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 311 to run in 1 seconds 2017-09-14 15:21:35,545 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 311 2017-09-14 15:21:35,545 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:35,546 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 312 to run in 1 seconds 2017-09-14 15:21:36,556 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 312 2017-09-14 15:21:36,556 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:36,556 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 313 to run in 1 seconds 2017-09-14 15:21:37,557 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 313 2017-09-14 15:21:37,557 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:37,557 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 314 to run in 1 seconds 2017-09-14 15:21:38,558 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 314 2017-09-14 15:21:38,558 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:38,558 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 315 to run in 1 seconds 2017-09-14 15:21:38,866 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 47 2017-09-14 15:21:38,866 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:21:38,867 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:21:38,867 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:21:38,867 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:21:38,867 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:21:38,867 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:21:38,867 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 48 to run in 10 seconds 2017-09-14 15:21:39,567 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 315 2017-09-14 15:21:39,567 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:39,567 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 316 to run in 1 seconds 2017-09-14 15:21:40,574 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 316 2017-09-14 15:21:40,574 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:40,574 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 317 to run in 1 seconds 2017-09-14 15:21:41,583 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 317 2017-09-14 15:21:41,584 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:41,584 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 318 to run in 1 seconds 2017-09-14 15:21:42,596 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 318 2017-09-14 15:21:42,596 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:42,596 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 319 to run in 1 seconds 2017-09-14 15:21:43,602 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 319 2017-09-14 15:21:43,602 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:43,602 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 320 to run in 1 seconds 2017-09-14 15:21:44,609 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 320 2017-09-14 15:21:44,609 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:44,609 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 321 to run in 1 seconds 2017-09-14 15:21:45,613 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 321 2017-09-14 15:21:45,613 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:45,613 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 322 to run in 1 seconds 2017-09-14 15:21:46,625 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 322 2017-09-14 15:21:46,625 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:46,625 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 323 to run in 1 seconds 2017-09-14 15:21:47,632 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 323 2017-09-14 15:21:47,633 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:47,633 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 324 to run in 1 seconds 2017-09-14 15:21:48,640 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 324 2017-09-14 15:21:48,640 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:48,640 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 325 to run in 1 seconds 2017-09-14 15:21:48,883 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 48 2017-09-14 15:21:48,883 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:21:48,883 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:21:48,884 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:21:48,884 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:21:48,884 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:21:48,884 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:21:48,884 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 49 to run in 10 seconds 2017-09-14 15:21:48,969 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:21:48,969 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:21:48,970 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:21:49,642 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 325 2017-09-14 15:21:49,643 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:49,643 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 326 to run in 1 seconds 2017-09-14 15:21:50,654 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 326 2017-09-14 15:21:50,654 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:50,654 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 327 to run in 1 seconds 2017-09-14 15:21:51,665 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 327 2017-09-14 15:21:51,665 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:51,665 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 328 to run in 1 seconds 2017-09-14 15:21:52,674 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 328 2017-09-14 15:21:52,674 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:52,674 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 329 to run in 1 seconds 2017-09-14 15:21:53,685 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 329 2017-09-14 15:21:53,685 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:53,685 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 330 to run in 1 seconds 2017-09-14 15:21:54,696 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 330 2017-09-14 15:21:54,697 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:54,697 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 331 to run in 1 seconds 2017-09-14 15:21:55,707 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 331 2017-09-14 15:21:55,707 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:55,707 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 332 to run in 1 seconds 2017-09-14 15:21:56,719 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 332 2017-09-14 15:21:56,720 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:56,720 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 333 to run in 1 seconds 2017-09-14 15:21:57,732 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 333 2017-09-14 15:21:57,733 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:57,733 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 334 to run in 1 seconds 2017-09-14 15:21:58,742 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 334 2017-09-14 15:21:58,743 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:58,743 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 335 to run in 1 seconds 2017-09-14 15:21:58,892 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 49 2017-09-14 15:21:58,892 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:21:58,893 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:21:58,893 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:21:58,893 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:21:58,893 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:21:58,893 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:21:58,893 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 50 to run in 10 seconds 2017-09-14 15:21:59,752 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 335 2017-09-14 15:21:59,752 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:21:59,752 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 336 to run in 1 seconds 2017-09-14 15:22:00,761 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 336 2017-09-14 15:22:00,762 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:00,762 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 337 to run in 1 seconds 2017-09-14 15:22:01,765 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 337 2017-09-14 15:22:01,766 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:01,766 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 338 to run in 1 seconds 2017-09-14 15:22:02,767 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 338 2017-09-14 15:22:02,767 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:02,767 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 339 to run in 1 seconds 2017-09-14 15:22:03,783 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 339 2017-09-14 15:22:03,783 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:03,783 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 340 to run in 1 seconds 2017-09-14 15:22:03,971 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:22:03,971 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:22:03,972 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:22:04,789 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 340 2017-09-14 15:22:04,789 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:04,789 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 341 to run in 1 seconds 2017-09-14 15:22:05,802 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 341 2017-09-14 15:22:05,802 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:05,802 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 342 to run in 1 seconds 2017-09-14 15:22:06,809 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 342 2017-09-14 15:22:06,809 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:06,809 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 343 to run in 1 seconds 2017-09-14 15:22:07,819 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 343 2017-09-14 15:22:07,820 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:07,820 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 344 to run in 1 seconds 2017-09-14 15:22:08,835 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 344 2017-09-14 15:22:08,836 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:08,836 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 345 to run in 1 seconds 2017-09-14 15:22:08,901 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 50 2017-09-14 15:22:08,901 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:22:08,901 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:22:08,901 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:22:08,901 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:22:08,901 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:22:08,902 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:22:08,902 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 51 to run in 10 seconds 2017-09-14 15:22:09,841 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 345 2017-09-14 15:22:09,841 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:09,842 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 346 to run in 1 seconds 2017-09-14 15:22:10,853 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 346 2017-09-14 15:22:10,854 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:10,854 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 347 to run in 1 seconds 2017-09-14 15:22:11,860 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 347 2017-09-14 15:22:11,861 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:11,861 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 348 to run in 1 seconds 2017-09-14 15:22:12,867 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 348 2017-09-14 15:22:12,867 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:12,867 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 349 to run in 1 seconds 2017-09-14 15:22:13,870 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 349 2017-09-14 15:22:13,871 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:13,871 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 350 to run in 1 seconds 2017-09-14 15:22:14,878 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 350 2017-09-14 15:22:14,878 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:14,878 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 351 to run in 1 seconds 2017-09-14 15:22:15,883 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 351 2017-09-14 15:22:15,884 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:15,886 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 352 to run in 1 seconds 2017-09-14 15:22:16,891 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 352 2017-09-14 15:22:16,891 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:16,891 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 353 to run in 1 seconds 2017-09-14 15:22:17,897 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 353 2017-09-14 15:22:17,898 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:17,898 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 354 to run in 1 seconds 2017-09-14 15:22:18,665 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkNodeRequestSpike with id 44 2017-09-14 15:22:18,665 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node1 checking its request amount 2017-09-14 15:22:18,665 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:22:18,665 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkNodeRequestSpike with id 52 to run in 60 seconds 2017-09-14 15:22:18,665 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 6 2017-09-14 15:22:18,666 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-09-14 15:22:18,666 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 7 to run in 60 seconds 2017-09-14 15:22:18,838 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action dump_json_file with id 45 2017-09-14 15:22:18,839 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action dump_json_file with id 53 to run in 60 seconds 2017-09-14 15:22:18,912 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 354 2017-09-14 15:22:18,913 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:18,913 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 355 to run in 1 seconds 2017-09-14 15:22:18,914 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 51 2017-09-14 15:22:18,915 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:22:18,915 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:22:18,915 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:22:18,915 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:22:18,915 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:22:18,916 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:22:18,916 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 54 to run in 10 seconds 2017-09-14 15:22:18,977 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:22:18,978 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:22:18,978 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:22:19,918 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 355 2017-09-14 15:22:19,919 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:19,919 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 356 to run in 1 seconds 2017-09-14 15:22:20,930 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 356 2017-09-14 15:22:20,931 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:20,931 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 357 to run in 1 seconds 2017-09-14 15:22:21,943 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 357 2017-09-14 15:22:21,944 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:21,944 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 358 to run in 1 seconds 2017-09-14 15:22:22,946 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 358 2017-09-14 15:22:22,947 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:22,947 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 359 to run in 1 seconds 2017-09-14 15:22:23,950 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 359 2017-09-14 15:22:23,951 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:23,951 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 360 to run in 1 seconds 2017-09-14 15:22:24,953 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 360 2017-09-14 15:22:24,953 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:24,953 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 361 to run in 1 seconds 2017-09-14 15:22:25,957 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 361 2017-09-14 15:22:25,957 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:25,958 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 362 to run in 1 seconds 2017-09-14 15:22:26,961 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 362 2017-09-14 15:22:26,961 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:26,961 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 363 to run in 1 seconds 2017-09-14 15:22:27,970 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 363 2017-09-14 15:22:27,970 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:27,970 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 364 to run in 1 seconds 2017-09-14 15:22:28,927 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 54 2017-09-14 15:22:28,927 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:22:28,927 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:22:28,927 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:22:28,927 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:22:28,928 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:22:28,928 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:22:28,928 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 55 to run in 10 seconds 2017-09-14 15:22:28,976 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 364 2017-09-14 15:22:28,976 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:28,976 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 365 to run in 1 seconds 2017-09-14 15:22:29,987 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 365 2017-09-14 15:22:29,988 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:29,988 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 366 to run in 1 seconds 2017-09-14 15:22:30,994 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 366 2017-09-14 15:22:30,994 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:30,994 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 367 to run in 1 seconds 2017-09-14 15:22:32,005 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 367 2017-09-14 15:22:32,005 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:32,005 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 368 to run in 1 seconds 2017-09-14 15:22:33,010 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 368 2017-09-14 15:22:33,010 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:33,010 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 369 to run in 1 seconds 2017-09-14 15:22:33,981 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:22:33,981 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:22:33,981 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:22:34,017 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 369 2017-09-14 15:22:34,018 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:34,018 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 370 to run in 1 seconds 2017-09-14 15:22:35,020 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 370 2017-09-14 15:22:35,020 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:35,021 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 371 to run in 1 seconds 2017-09-14 15:22:36,024 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 371 2017-09-14 15:22:36,024 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:36,024 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 372 to run in 1 seconds 2017-09-14 15:22:37,026 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 372 2017-09-14 15:22:37,026 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:37,026 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 373 to run in 1 seconds 2017-09-14 15:22:38,031 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 373 2017-09-14 15:22:38,031 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:38,031 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 374 to run in 1 seconds 2017-09-14 15:22:38,934 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 55 2017-09-14 15:22:38,934 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:22:38,934 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:22:38,934 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:22:38,934 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:22:38,935 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:22:38,935 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:22:38,935 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 56 to run in 10 seconds 2017-09-14 15:22:39,040 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 374 2017-09-14 15:22:39,040 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:39,040 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 375 to run in 1 seconds 2017-09-14 15:22:40,046 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 375 2017-09-14 15:22:40,047 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:40,047 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 376 to run in 1 seconds 2017-09-14 15:22:41,057 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 376 2017-09-14 15:22:41,057 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:41,057 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 377 to run in 1 seconds 2017-09-14 15:22:42,059 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 377 2017-09-14 15:22:42,059 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:42,059 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 378 to run in 1 seconds 2017-09-14 15:22:43,072 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 378 2017-09-14 15:22:43,072 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:43,073 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 379 to run in 1 seconds 2017-09-14 15:22:44,084 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 379 2017-09-14 15:22:44,085 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:44,085 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 380 to run in 1 seconds 2017-09-14 15:22:45,096 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 380 2017-09-14 15:22:45,097 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:45,097 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 381 to run in 1 seconds 2017-09-14 15:22:46,105 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 381 2017-09-14 15:22:46,105 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:46,106 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 382 to run in 1 seconds 2017-09-14 15:22:47,112 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 382 2017-09-14 15:22:47,113 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:47,113 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 383 to run in 1 seconds 2017-09-14 15:22:48,113 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 383 2017-09-14 15:22:48,114 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:48,114 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 384 to run in 1 seconds 2017-09-14 15:22:48,937 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 56 2017-09-14 15:22:48,937 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:22:48,937 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:22:48,937 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:22:48,938 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:22:48,938 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:22:48,938 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:22:48,938 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 57 to run in 10 seconds 2017-09-14 15:22:48,997 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:22:48,997 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:22:48,997 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:22:49,123 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 384 2017-09-14 15:22:49,123 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:49,123 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 385 to run in 1 seconds 2017-09-14 15:22:50,129 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 385 2017-09-14 15:22:50,130 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:50,130 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 386 to run in 1 seconds 2017-09-14 15:22:51,134 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 386 2017-09-14 15:22:51,134 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:51,134 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 387 to run in 1 seconds 2017-09-14 15:22:52,136 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 387 2017-09-14 15:22:52,137 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:52,137 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 388 to run in 1 seconds 2017-09-14 15:22:53,153 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 388 2017-09-14 15:22:53,154 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:53,154 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 389 to run in 1 seconds 2017-09-14 15:22:54,163 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 389 2017-09-14 15:22:54,164 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:54,164 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 390 to run in 1 seconds 2017-09-14 15:22:55,170 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 390 2017-09-14 15:22:55,171 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:55,171 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 391 to run in 1 seconds 2017-09-14 15:22:56,175 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 391 2017-09-14 15:22:56,175 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:56,175 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 392 to run in 1 seconds 2017-09-14 15:22:57,182 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 392 2017-09-14 15:22:57,183 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:57,183 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 393 to run in 1 seconds 2017-09-14 15:22:58,191 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 393 2017-09-14 15:22:58,191 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:58,191 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 394 to run in 1 seconds 2017-09-14 15:22:58,944 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 57 2017-09-14 15:22:58,944 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:22:58,944 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:22:58,944 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:22:58,945 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:22:58,945 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:22:58,945 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:22:58,945 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 58 to run in 10 seconds 2017-09-14 15:22:59,202 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 394 2017-09-14 15:22:59,202 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:22:59,202 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 395 to run in 1 seconds 2017-09-14 15:23:00,213 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 395 2017-09-14 15:23:00,213 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:23:00,213 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 396 to run in 1 seconds 2017-09-14 15:23:01,222 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 396 2017-09-14 15:23:01,222 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:23:01,222 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 397 to run in 1 seconds 2017-09-14 15:23:02,223 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 397 2017-09-14 15:23:02,224 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:23:02,224 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 398 to run in 1 seconds 2017-09-14 15:23:03,227 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 398 2017-09-14 15:23:03,228 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:23:03,228 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 399 to run in 1 seconds 2017-09-14 15:23:03,998 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:23:03,999 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:23:03,999 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:23:04,229 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 399 2017-09-14 15:23:04,229 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:23:04,229 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 400 to run in 1 seconds 2017-09-14 15:23:05,231 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 400 2017-09-14 15:23:05,231 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:23:05,232 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 401 to run in 1 seconds 2017-09-14 15:23:06,241 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 401 2017-09-14 15:23:06,241 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:23:06,241 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 402 to run in 1 seconds 2017-09-14 15:23:07,245 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 402 2017-09-14 15:23:07,245 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:23:07,245 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 403 to run in 1 seconds 2017-09-14 15:23:08,251 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 403 2017-09-14 15:23:08,251 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:23:08,252 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 404 to run in 1 seconds 2017-09-14 15:23:08,954 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 58 2017-09-14 15:23:08,954 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:23:08,954 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:23:08,954 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:23:08,954 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:23:08,954 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:23:08,954 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:23:08,954 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 59 to run in 10 seconds 2017-09-14 15:23:09,256 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 404 2017-09-14 15:23:09,256 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:23:09,256 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 405 to run in 1 seconds 2017-09-14 15:23:10,257 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 405 2017-09-14 15:23:10,257 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:23:10,257 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 406 to run in 1 seconds 2017-09-14 15:23:11,265 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 406 2017-09-14 15:23:11,266 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:23:11,266 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 407 to run in 1 seconds 2017-09-14 15:23:12,270 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 407 2017-09-14 15:23:12,271 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:23:12,271 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 408 to run in 1 seconds 2017-09-14 15:23:13,276 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 408 2017-09-14 15:23:13,276 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:23:13,276 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 409 to run in 1 seconds 2017-09-14 15:23:14,278 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 409 2017-09-14 15:23:14,278 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:23:14,278 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 410 to run in 1 seconds 2017-09-14 15:23:15,283 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 410 2017-09-14 15:23:15,283 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:23:15,284 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 411 to run in 1 seconds 2017-09-14 15:23:16,293 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 411 2017-09-14 15:23:16,294 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:23:16,294 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 412 to run in 1 seconds 2017-09-14 15:23:17,302 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 412 2017-09-14 15:23:17,302 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:23:17,302 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 413 to run in 1 seconds 2017-09-14 15:23:18,307 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 413 2017-09-14 15:23:18,307 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:23:18,307 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 414 to run in 1 seconds 2017-09-14 15:23:18,666 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkNodeRequestSpike with id 52 2017-09-14 15:23:18,667 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node1 checking its request amount 2017-09-14 15:23:18,667 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:23:18,667 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkNodeRequestSpike with id 60 to run in 60 seconds 2017-09-14 15:23:18,667 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 7 2017-09-14 15:23:18,667 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-09-14 15:23:18,667 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 8 to run in 60 seconds 2017-09-14 15:23:18,839 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action dump_json_file with id 53 2017-09-14 15:23:18,840 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action dump_json_file with id 61 to run in 60 seconds 2017-09-14 15:23:18,955 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 59 2017-09-14 15:23:18,955 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:23:18,955 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:23:18,955 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:23:18,955 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:23:18,955 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:23:18,956 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:23:18,956 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 62 to run in 10 seconds 2017-09-14 15:23:19,007 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-09-14 15:23:19,008 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node1 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-09-14 15:23:19,008 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node1 next check for retries in 15.00 seconds 2017-09-14 15:23:19,312 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 414 2017-09-14 15:23:19,312 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:23:19,312 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 415 to run in 1 seconds 2017-09-14 15:23:20,316 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 415 2017-09-14 15:23:20,317 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:23:20,317 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 416 to run in 1 seconds 2017-09-14 15:23:21,320 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 416 2017-09-14 15:23:21,320 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:23:21,320 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 417 to run in 1 seconds 2017-09-14 15:23:22,326 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 417 2017-09-14 15:23:22,326 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:23:22,326 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 418 to run in 1 seconds 2017-09-14 15:23:23,328 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 418 2017-09-14 15:23:23,328 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:23:23,328 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 419 to run in 1 seconds 2017-09-14 15:23:24,338 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 419 2017-09-14 15:23:24,338 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:23:24,339 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 420 to run in 1 seconds 2017-09-14 15:23:25,346 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 420 2017-09-14 15:23:25,346 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:23:25,347 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 421 to run in 1 seconds 2017-09-14 15:23:26,355 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 421 2017-09-14 15:23:26,355 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:23:26,355 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 422 to run in 1 seconds 2017-09-14 15:23:27,360 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 422 2017-09-14 15:23:27,360 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:23:27,360 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 423 to run in 1 seconds 2017-09-14 15:23:28,362 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 423 2017-09-14 15:23:28,363 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:23:28,363 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 424 to run in 1 seconds 2017-09-14 15:23:28,956 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1 running action checkPerformance with id 62 2017-09-14 15:23:28,956 | TRACE | node.py (1978) | checkPerformance | Node1 checking its performance 2017-09-14 15:23:28,956 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-09-14 15:23:28,956 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node1 master throughput is not measurable. 2017-09-14 15:23:28,956 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node1 found master's latency to be lower than the threshold for all requests. 2017-09-14 15:23:28,956 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node1 found difference between master and backups avg latencies to be acceptable 2017-09-14 15:23:28,957 | DEBUG | node.py (1995) | checkPerformance | Node1's master has higher performance than backups 2017-09-14 15:23:28,957 | TRACE | has_action_queue.py ( 36) | _schedule | Node1 scheduling action checkPerformance with id 63 to run in 10 seconds 2017-09-14 15:23:29,364 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node1:0 running action process_stashed_out_of_order_commits with id 424 2017-09-14 15:23:29,365 | DEBUG | replica.py (1389) | process_stashed_out_of_order_commits | Node1:0 trying to order from out of order commits. OrderedSet() {1: {4: COMMIT{'ppSeqNo': 4, 'instId': 0, 'viewNo': 1}}} 2017-09-14 15:23:29,365 | TRACE | has_action_queue.py ( 36) | _schedule | Node1:0 scheduling action process_stashed_out_of_order_commits with id 425 to run in 1 seconds