2017-10-23 15:56:40,547 | INFO | log.py ( 79) | setupRaet | Setting RAET log level 2 2017-10-23 15:56:40,558 | DEBUG | start_sovrin_node ( 39) | | You can find logs in /home/sovrin/.sovrin/Node3.log 2017-10-23 15:56:40,559 | DEBUG | start_sovrin_node ( 42) | | Sovrin related env vars: [] 2017-10-23 15:56:41,622 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: git 2017-10-23 15:56:41,660 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: hg 2017-10-23 15:56:41,785 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: svn 2017-10-23 15:56:41,786 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: bzr 2017-10-23 15:56:42,191 | DEBUG | selector_events.py ( 53) | __init__ | Using selector: EpollSelector 2017-10-23 15:56:42,192 | DEBUG | looper.py ( 123) | __init__ | Setting handler for SIGINT 2017-10-23 15:56:42,252 | DEBUG | ledger.py ( 206) | start | Starting ledger... 2017-10-23 15:56:42,253 | DEBUG | file_store.py ( 190) | appendNewLineIfReq | new line check for file: /home/sovrin/.sovrin/data/nodes/Node3/transactions_sandbox/1 2017-10-23 15:56:42,253 | DEBUG | ledger.py ( 78) | recoverTree | Recovering tree from transaction log 2017-10-23 15:56:42,320 | DEBUG | ledger.py ( 93) | recoverTree | Recovered tree in 0.06676647800486535 seconds 2017-10-23 15:56:42,369 | DEBUG | idr_cache.py ( 25) | __init__ | Initializing identity cache Node3 2017-10-23 15:56:42,398 | INFO | node.py (2408) | initStateFromLedger | Node3 found state to be empty, recreating from ledger 2017-10-23 15:56:42,426 | DEBUG | ledger.py ( 206) | start | Starting ledger... 2017-10-23 15:56:42,427 | DEBUG | file_store.py ( 190) | appendNewLineIfReq | new line check for file: /home/sovrin/.sovrin/data/nodes/Node3/pool_transactions_sandbox/1 2017-10-23 15:56:42,427 | DEBUG | ledger.py ( 78) | recoverTree | Recovering tree from transaction log 2017-10-23 15:56:42,480 | DEBUG | ledger.py ( 93) | recoverTree | Recovered tree in 0.05266980698797852 seconds 2017-10-23 15:56:42,480 | INFO | node.py (2408) | initStateFromLedger | Node3 found state to be empty, recreating from ledger 2017-10-23 15:56:42,538 | DEBUG | plugin_loader.py ( 95) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-10-23 15:56:42,538 | DEBUG | plugin_loader.py ( 95) | _load | skipping plugin plugin_firebase_stats_consumer[class: typing.Dict<~KT, ~VT>] because it does not have a 'pluginType' attribute 2017-10-23 15:56:42,538 | DEBUG | plugin_loader.py ( 95) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-10-23 15:56:42,538 | INFO | plugin_loader.py ( 116) | _load | plugin FirebaseStatsConsumer successfully loaded from module plugin_firebase_stats_consumer 2017-10-23 15:56:42,538 | DEBUG | plugin_loader.py ( 95) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-10-23 15:56:42,541 | DEBUG | plugin_loader.py ( 95) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-10-23 15:56:42,541 | DEBUG | has_action_queue.py ( 77) | startRepeating | checkPerformance will be repeating every 60 seconds 2017-10-23 15:56:42,541 | TRACE | has_action_queue.py ( 34) | _schedule | Node3 scheduling action checkPerformance with id 1 to run in 60 seconds 2017-10-23 15:56:42,541 | INFO | replica.py ( 300) | h | Node3:0 set watermarks as 0 300 2017-10-23 15:56:42,542 | DISPLAY | node.py (1034) | addReplica | Node3 added replica Node3:0 to instance 0 (master) 2017-10-23 15:56:42,542 | INFO | replica.py ( 300) | h | Node3:1 set watermarks as 0 300 2017-10-23 15:56:42,542 | DISPLAY | node.py (1034) | addReplica | Node3 added replica Node3:1 to instance 1 (backup) 2017-10-23 15:56:42,542 | DEBUG | has_action_queue.py ( 77) | startRepeating | checkPerformance will be repeating every 10 seconds 2017-10-23 15:56:42,542 | TRACE | has_action_queue.py ( 34) | _schedule | Node3 scheduling action checkPerformance with id 1 to run in 10 seconds 2017-10-23 15:56:42,542 | DEBUG | has_action_queue.py ( 77) | startRepeating | checkNodeRequestSpike will be repeating every 60 seconds 2017-10-23 15:56:42,542 | TRACE | has_action_queue.py ( 34) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 2 to run in 60 seconds 2017-10-23 15:56:42,542 | DEBUG | plugin_helper.py ( 24) | loadPlugins | Plugin loading started to load plugins from basedir: /home/sovrin/.sovrin 2017-10-23 15:56:42,543 | DEBUG | plugin_helper.py ( 33) | loadPlugins | Plugin directory created at: /home/sovrin/.sovrin/plugins 2017-10-23 15:56:42,543 | DEBUG | plugin_helper.py ( 67) | loadPlugins | Total plugins loaded from basedir /home/sovrin/.sovrin are : 0 2017-10-23 15:56:42,543 | DEBUG | node.py ( 325) | __init__ | total plugins loaded in node: 0 2017-10-23 15:56:42,570 | DEBUG | ledger.py ( 206) | start | Starting ledger... 2017-10-23 15:56:42,570 | DEBUG | file_store.py ( 190) | appendNewLineIfReq | new line check for file: /home/sovrin/.sovrin/data/nodes/Node3/config_transactions/1 2017-10-23 15:56:42,570 | DEBUG | ledger.py ( 78) | recoverTree | Recovering tree from transaction log 2017-10-23 15:56:42,571 | DEBUG | ledger.py ( 93) | recoverTree | Recovered tree in 0.00023650191724300385 seconds 2017-10-23 15:56:42,600 | INFO | node.py (2408) | initStateFromLedger | Node3 found state to be empty, recreating from ledger 2017-10-23 15:56:42,600 | DEBUG | motor.py ( 34) | set_status | Node3 changing status from stopped to starting 2017-10-23 15:56:42,600 | DEBUG | ledger.py ( 204) | start | Ledger already started. 2017-10-23 15:56:42,600 | INFO | zstack.py ( 312) | start | Node3 starting with restricted as True and reSetupAuth as True 2017-10-23 15:56:42,601 | DEBUG | authenticator.py ( 31) | start | Starting ZAP at inproc://zeromq.zap.1 2017-10-23 15:56:42,601 | DEBUG | base.py ( 72) | allow | Allowing 0.0.0.0 2017-10-23 15:56:42,601 | DEBUG | base.py ( 112) | configure_curve | Configure curve: *[/home/sovrin/.sovrin/Node3/public_keys] 2017-10-23 15:56:42,601 | DEBUG | zstack.py ( 339) | open | Node3 will bind its listener at 9705 2017-10-23 15:56:42,602 | INFO | stacks.py ( 76) | start | Node3 listening for other nodes at 0.0.0.0:9705 2017-10-23 15:56:42,602 | INFO | zstack.py ( 312) | start | Node3C starting with restricted as False and reSetupAuth as True 2017-10-23 15:56:42,602 | DEBUG | authenticator.py ( 31) | start | Starting ZAP at inproc://zeromq.zap.2 2017-10-23 15:56:42,602 | DEBUG | base.py ( 72) | allow | Allowing 0.0.0.0 2017-10-23 15:56:42,602 | DEBUG | base.py ( 112) | configure_curve | Configure curve: *[*] 2017-10-23 15:56:42,602 | DEBUG | zstack.py ( 339) | open | Node3C will bind its listener at 9706 2017-10-23 15:56:42,602 | INFO | node.py ( 594) | start | Node3 first time running... 2017-10-23 15:56:42,606 | DEBUG | kit_zstack.py ( 96) | connectToMissing | Node3 found the following missing connections: Node1, Node2, Node4 2017-10-23 15:56:42,606 | TRACE | remote.py ( 84) | connect | connecting socket 58 43570672 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-23 15:56:42,614 | INFO | zstack.py ( 580) | connect | Node3 looking for Node1 at 10.0.0.2:9701 2017-10-23 15:56:42,615 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 will be sending in batch 2017-10-23 15:56:42,615 | TRACE | remote.py ( 84) | connect | connecting socket 61 43674640 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-23 15:56:42,615 | INFO | zstack.py ( 580) | connect | Node3 looking for Node2 at 10.0.0.3:9703 2017-10-23 15:56:42,615 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 will be sending in batch 2017-10-23 15:56:42,616 | TRACE | remote.py ( 84) | connect | connecting socket 62 43711680 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-23 15:56:42,616 | INFO | zstack.py ( 580) | connect | Node3 looking for Node4 at 10.0.0.5:9707 2017-10-23 15:56:42,616 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 will be sending in batch 2017-10-23 15:56:42,616 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 15:56:42,648 | DEBUG | zstack.py ( 699) | transmit | Node3 transmitting message b'pi' to Node1 2017-10-23 15:56:42,649 | TRACE | batched.py ( 85) | flushOutBoxes | Node3 sending msg b'pi' to Node1 2017-10-23 15:56:42,649 | DEBUG | zstack.py ( 699) | transmit | Node3 transmitting message b'pi' to Node2 2017-10-23 15:56:42,649 | TRACE | batched.py ( 85) | flushOutBoxes | Node3 sending msg b'pi' to Node2 2017-10-23 15:56:42,649 | DEBUG | zstack.py ( 699) | transmit | Node3 transmitting message b'pi' to Node4 2017-10-23 15:56:42,650 | TRACE | batched.py ( 85) | flushOutBoxes | Node3 sending msg b'pi' to Node4 2017-10-23 15:56:42,691 | TRACE | zstack.py ( 472) | _receiveFromListener | Node3 got 2 messages through listener 2017-10-23 15:56:42,691 | DEBUG | zstack.py ( 652) | handlePingPong | Node3 got ping from Node1 2017-10-23 15:56:42,691 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 will be sending in batch 2017-10-23 15:56:42,691 | DEBUG | zstack.py ( 658) | handlePingPong | Node3 got pong from Node1 2017-10-23 15:56:42,692 | DEBUG | zstack.py ( 699) | transmit | Node3 transmitting message b'po' to Node1 2017-10-23 15:56:42,693 | TRACE | batched.py ( 85) | flushOutBoxes | Node3 sending msg b'po' to Node1 2017-10-23 15:56:42,693 | DEBUG | keep_in_touch.py ( 68) | conns | Node3's connections changed from set() to {'Node1'} 2017-10-23 15:56:42,693 | INFO | keep_in_touch.py ( 96) | _connsChanged | Node3 now connected to Node1 2017-10-23 15:56:42,693 | DEBUG | node.py (2593) | send | Node3 sending message LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'txnSeqNo': 4} to 1 recipients: ['Node1'] 2017-10-23 15:56:42,695 | DEBUG | zstack.py ( 699) | transmit | Node3 transmitting message b'{"ledgerId":0,"merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA","ppSeqNo":null,"op":"LEDGER_STATUS","viewNo":null,"txnSeqNo":4}' to Node1 2017-10-23 15:56:42,696 | TRACE | batched.py ( 85) | flushOutBoxes | Node3 sending msg b'{"ledgerId":0,"merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA","ppSeqNo":null,"op":"LEDGER_STATUS","viewNo":null,"txnSeqNo":4}' to Node1 2017-10-23 15:56:42,723 | TRACE | zstack.py ( 472) | _receiveFromListener | Node3 got 1 messages through listener 2017-10-23 15:56:42,724 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node1: LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'txnSeqNo': 4} 2017-10-23 15:56:42,724 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'ledgerId': 0, 'ppSeqNo': None, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'viewNo': None, 'op': 'LEDGER_STATUS', 'txnSeqNo': 4}, 'Node1') 2017-10-23 15:56:42,724 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'txnSeqNo': 4} 2017-10-23 15:56:42,724 | DEBUG | ledger_manager.py ( 244) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'txnSeqNo': 4} from Node1 2017-10-23 15:56:42,725 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-10-23 15:56:42,725 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-10-23 15:56:44,945 | TRACE | zstack.py ( 472) | _receiveFromListener | Node3 got 1 messages through listener 2017-10-23 15:56:44,945 | DEBUG | zstack.py ( 652) | handlePingPong | Node3 got ping from Node2 2017-10-23 15:56:44,945 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 will be sending in batch 2017-10-23 15:56:44,947 | DEBUG | zstack.py ( 699) | transmit | Node3 transmitting message b'po' to Node2 2017-10-23 15:56:44,947 | TRACE | batched.py ( 85) | flushOutBoxes | Node3 sending msg b'po' to Node2 2017-10-23 15:56:45,013 | TRACE | zstack.py ( 472) | _receiveFromListener | Node3 got 1 messages through listener 2017-10-23 15:56:45,013 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node1: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}']} 2017-10-23 15:56:45,013 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}'], 'signature': None, 'op': 'BATCH'}, 'Node1') 2017-10-23 15:56:45,013 | DEBUG | node.py (1328) | unpackNodeMsg | Node3 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}']} 2017-10-23 15:56:45,014 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-23 15:56:45,014 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'op': 'MESSAGE_REQUEST'}, 'Node1') 2017-10-23 15:56:45,014 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-23 15:56:45,014 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-23 15:56:45,014 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'op': 'MESSAGE_REQUEST'}, 'Node1') 2017-10-23 15:56:45,014 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-23 15:56:45,015 | DEBUG | node.py (2593) | send | Node3 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node1'] 2017-10-23 15:56:45,015 | DEBUG | node.py (2593) | send | Node3 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node1'] 2017-10-23 15:56:45,016 | DEBUG | batched.py ( 89) | flushOutBoxes | Node3 batching 2 msgs to Node1 into one transmission 2017-10-23 15:56:45,016 | TRACE | batched.py ( 90) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}']) 2017-10-23 15:56:45,016 | TRACE | batched.py ( 98) | flushOutBoxes | Node3 sending payload to Node1: b'{"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}"],"op":"BATCH","signature":null}' 2017-10-23 15:56:45,017 | DEBUG | zstack.py ( 699) | transmit | Node3 transmitting message b'{"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}"],"op":"BATCH","signature":null}' to Node1 2017-10-23 15:56:45,082 | TRACE | zstack.py ( 472) | _receiveFromListener | Node3 got 3 messages through listener 2017-10-23 15:56:45,082 | DEBUG | zstack.py ( 658) | handlePingPong | Node3 got pong from Node2 2017-10-23 15:56:45,083 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node1: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}']} 2017-10-23 15:56:45,083 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}'], 'signature': None, 'op': 'BATCH'}, 'Node1') 2017-10-23 15:56:45,083 | DEBUG | node.py (1328) | unpackNodeMsg | Node3 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}']} 2017-10-23 15:56:45,083 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 15:56:45,083 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'op': 'MESSAGE_REQUEST'}, 'Node1') 2017-10-23 15:56:45,083 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 15:56:45,083 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 15:56:45,083 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'op': 'MESSAGE_REQUEST'}, 'Node1') 2017-10-23 15:56:45,084 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 15:56:45,084 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node2: BATCH{'signature': None, 'messages': ['{"viewNo":0,"primary":[],"op":"CURRENT_STATE"}', '{"txnSeqNo":4,"viewNo":null,"ppSeqNo":null,"ledgerId":0,"op":"LEDGER_STATUS","merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"}']} 2017-10-23 15:56:45,084 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'signature': None, 'op': 'BATCH', 'messages': ['{"viewNo":0,"primary":[],"op":"CURRENT_STATE"}', '{"txnSeqNo":4,"viewNo":null,"ppSeqNo":null,"ledgerId":0,"op":"LEDGER_STATUS","merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"}']}, 'Node2') 2017-10-23 15:56:45,084 | DEBUG | node.py (1328) | unpackNodeMsg | Node3 processing a batch BATCH{'signature': None, 'messages': ['{"viewNo":0,"primary":[],"op":"CURRENT_STATE"}', '{"txnSeqNo":4,"viewNo":null,"ppSeqNo":null,"ledgerId":0,"op":"LEDGER_STATUS","merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"}']} 2017-10-23 15:56:45,084 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node2: CURRENT_STATE{'primary': [], 'viewNo': 0} 2017-10-23 15:56:45,084 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'primary': [], 'viewNo': 0, 'op': 'CURRENT_STATE'}, 'Node2') 2017-10-23 15:56:45,084 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox CURRENT_STATE{'primary': [], 'viewNo': 0} 2017-10-23 15:56:45,085 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node2: LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'txnSeqNo': 4} 2017-10-23 15:56:45,085 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'ledgerId': 0, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'ppSeqNo': None, 'txnSeqNo': 4, 'viewNo': None, 'op': 'LEDGER_STATUS'}, 'Node2') 2017-10-23 15:56:45,085 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'txnSeqNo': 4} 2017-10-23 15:56:45,085 | DEBUG | node.py (2593) | send | Node3 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'txnSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node1'] 2017-10-23 15:56:45,086 | DEBUG | node.py (2593) | send | Node3 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'txnSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node1'] 2017-10-23 15:56:45,089 | DEBUG | node.py ( 884) | process_current_state_message | Node3 processing current state CURRENT_STATE{'primary': [], 'viewNo': 0} from Node2 2017-10-23 15:56:45,090 | DEBUG | ledger_manager.py ( 244) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'txnSeqNo': 4} from Node2 2017-10-23 15:56:45,090 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-10-23 15:56:45,090 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-10-23 15:56:45,090 | DEBUG | ledger_manager.py ( 309) | processLedgerStatus | Node3 found out from {'Node1', 'Node2'} that its ledger of type 0 is latest 2017-10-23 15:56:45,090 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found from ledger status LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'txnSeqNo': 4} that it does not need catchup 2017-10-23 15:56:45,090 | DEBUG | node.py (1537) | preLedgerCatchUp | Node3 going to process any ordered requests before starting catchup. 2017-10-23 15:56:45,090 | DEBUG | replica.py (2130) | _remove_ordered_from_queue | Node3:0 going to remove 0 Ordered messages from outbox 2017-10-23 15:56:45,090 | DEBUG | node.py (1918) | force_process_ordered | Node3 processed 0 Ordered batches for instance 0 before starting catch up 2017-10-23 15:56:45,090 | DEBUG | replica.py (2130) | _remove_ordered_from_queue | Node3:1 going to remove 0 Ordered messages from outbox 2017-10-23 15:56:45,091 | DEBUG | node.py (1918) | force_process_ordered | Node3 processed 0 Ordered batches for instance 1 before starting catch up 2017-10-23 15:56:45,091 | DEBUG | node.py (2451) | processStashedOrderedReqs | Node3 processed 0 stashed ordered requests 2017-10-23 15:56:45,091 | DEBUG | monitor.py ( 183) | reset | Monitor being reset 2017-10-23 15:56:45,091 | DEBUG | node.py (1547) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 0 2017-10-23 15:56:45,091 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-10-23 15:56:45,091 | DEBUG | node.py ( 918) | checkInstances | Node3 choosing to start election on the basis of count 2 and nodes {'Node1'} 2017-10-23 15:56:45,091 | DEBUG | node.py (2593) | send | Node3 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node1'] 2017-10-23 15:56:45,091 | DEBUG | node.py ( 832) | _ask_for_ledger_status | Node3 asking Node1 for ledger status of ledger 2 2017-10-23 15:56:45,091 | DEBUG | node.py (2593) | send | Node3 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node2'] 2017-10-23 15:56:45,091 | DEBUG | node.py ( 832) | _ask_for_ledger_status | Node3 asking Node2 for ledger status of ledger 2 2017-10-23 15:56:45,092 | DEBUG | node.py (2593) | send | Node3 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to all recipients: ['Node1', 'Node2', 'Node4'] 2017-10-23 15:56:45,092 | DEBUG | node.py ( 832) | _ask_for_ledger_status | Node3 asking Node3 for ledger status of ledger 2 2017-10-23 15:56:45,092 | DEBUG | node.py (2593) | send | Node3 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-10-23 15:56:45,092 | DEBUG | node.py ( 832) | _ask_for_ledger_status | Node3 asking Node4 for ledger status of ledger 2 2017-10-23 15:56:45,092 | DEBUG | ledger_manager.py (1006) | processStashedLedgerStatuses | Node3 going to process 0 stashed ledger statuses for ledger 2 2017-10-23 15:56:45,092 | INFO | ledger_manager.py ( 831) | catchupCompleted | Node3 completed catching up ledger 0, caught up 0 in total 2017-10-23 15:56:45,093 | DEBUG | batched.py ( 89) | flushOutBoxes | Node3 batching 4 msgs to Node1 into one transmission 2017-10-23 15:56:45,093 | TRACE | batched.py ( 90) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', b'{"msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}']) 2017-10-23 15:56:45,093 | TRACE | batched.py ( 98) | flushOutBoxes | Node3 sending payload to Node1: b'{"messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":6,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":6,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}"],"op":"BATCH","signature":null}' 2017-10-23 15:56:45,106 | DEBUG | zstack.py ( 699) | transmit | Node3 transmitting message b'{"messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":6,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":6,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}"],"op":"BATCH","signature":null}' to Node1 2017-10-23 15:56:45,106 | DEBUG | batched.py ( 89) | flushOutBoxes | Node3 batching 2 msgs to Node2 into one transmission 2017-10-23 15:56:45,107 | TRACE | batched.py ( 90) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}']) 2017-10-23 15:56:45,107 | TRACE | batched.py ( 98) | flushOutBoxes | Node3 sending payload to Node2: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}"],"op":"BATCH","signature":null}' 2017-10-23 15:56:45,107 | DEBUG | zstack.py ( 699) | transmit | Node3 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}"],"op":"BATCH","signature":null}' to Node2 2017-10-23 15:56:45,107 | DEBUG | batched.py ( 89) | flushOutBoxes | Node3 batching 2 msgs to Node4 into one transmission 2017-10-23 15:56:45,107 | TRACE | batched.py ( 90) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}']) 2017-10-23 15:56:45,107 | TRACE | batched.py ( 98) | flushOutBoxes | Node3 sending payload to Node4: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}"],"op":"BATCH","signature":null}' 2017-10-23 15:56:45,108 | DEBUG | zstack.py ( 699) | transmit | Node3 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_REQUEST\\"}"],"op":"BATCH","signature":null}' to Node4 2017-10-23 15:56:45,108 | WARNING | zstack.py ( 704) | transmit | Remote Node4 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-23 15:56:45,108 | DEBUG | keep_in_touch.py ( 68) | conns | Node3's connections changed from {'Node1'} to {'Node1', 'Node2'} 2017-10-23 15:56:45,108 | INFO | keep_in_touch.py ( 96) | _connsChanged | Node3 now connected to Node2 2017-10-23 15:56:45,108 | DEBUG | motor.py ( 34) | set_status | Node3 changing status from starting to started_hungry 2017-10-23 15:56:45,108 | DEBUG | node.py ( 918) | checkInstances | Node3 choosing to start election on the basis of count 3 and nodes {'Node1', 'Node2'} 2017-10-23 15:56:45,108 | DEBUG | primary_selector.py ( 74) | get_msgs_for_lagged_nodes | Node3 has no ViewChangeDone message to send for view 0 2017-10-23 15:56:45,109 | DEBUG | node.py ( 879) | send_current_state_to_lagging_node | Node3 sending current state CURRENT_STATE{'primary': [], 'viewNo': 0} to lagged node Node2 2017-10-23 15:56:45,109 | DEBUG | node.py (2593) | send | Node3 sending message CURRENT_STATE{'primary': [], 'viewNo': 0} to 1 recipients: ['Node2'] 2017-10-23 15:56:45,109 | DEBUG | node.py (2593) | send | Node3 sending message LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'txnSeqNo': 4} to 1 recipients: ['Node2'] 2017-10-23 15:56:45,109 | DEBUG | node.py (2593) | send | Node3 sending message LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'txnSeqNo': 6} to 1 recipients: ['Node2'] 2017-10-23 15:56:45,111 | DEBUG | batched.py ( 89) | flushOutBoxes | Node3 batching 3 msgs to Node2 into one transmission 2017-10-23 15:56:45,112 | TRACE | batched.py ( 90) | flushOutBoxes | messages: deque([b'{"primary":[],"viewNo":0,"op":"CURRENT_STATE"}', b'{"ledgerId":0,"merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA","ppSeqNo":null,"op":"LEDGER_STATUS","viewNo":null,"txnSeqNo":4}', b'{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"op":"LEDGER_STATUS","viewNo":null,"txnSeqNo":6}']) 2017-10-23 15:56:45,112 | TRACE | batched.py ( 98) | flushOutBoxes | Node3 sending payload to Node2: b'{"messages":["{\\"primary\\":[],\\"viewNo\\":0,\\"op\\":\\"CURRENT_STATE\\"}","{\\"ledgerId\\":0,\\"merkleRoot\\":\\"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA\\",\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null,\\"txnSeqNo\\":4}","{\\"ledgerId\\":1,\\"merkleRoot\\":\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\",\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null,\\"txnSeqNo\\":6}"],"op":"BATCH","signature":null}' 2017-10-23 15:56:45,112 | DEBUG | zstack.py ( 699) | transmit | Node3 transmitting message b'{"messages":["{\\"primary\\":[],\\"viewNo\\":0,\\"op\\":\\"CURRENT_STATE\\"}","{\\"ledgerId\\":0,\\"merkleRoot\\":\\"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA\\",\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null,\\"txnSeqNo\\":4}","{\\"ledgerId\\":1,\\"merkleRoot\\":\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\",\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null,\\"txnSeqNo\\":6}"],"op":"BATCH","signature":null}' to Node2 2017-10-23 15:56:45,125 | TRACE | zstack.py ( 472) | _receiveFromListener | Node3 got 1 messages through listener 2017-10-23 15:56:45,125 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node1: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"name":"Node1","ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,6,"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":0,"op":"VIEW_CHANGE_DONE"}']} 2017-10-23 15:56:45,125 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"name":"Node1","ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,6,"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":0,"op":"VIEW_CHANGE_DONE"}'], 'signature': None, 'op': 'BATCH'}, 'Node1') 2017-10-23 15:56:45,125 | DEBUG | node.py (1328) | unpackNodeMsg | Node3 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"name":"Node1","ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,6,"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":0,"op":"VIEW_CHANGE_DONE"}']} 2017-10-23 15:56:45,126 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-23 15:56:45,126 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE'}, 'Node1') 2017-10-23 15:56:45,126 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-23 15:56:45,126 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-23 15:56:45,126 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE'}, 'Node1') 2017-10-23 15:56:45,126 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-23 15:56:45,127 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node1: VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-10-23 15:56:45,127 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0, 'op': 'VIEW_CHANGE_DONE'}, 'Node1') 2017-10-23 15:56:45,127 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-10-23 15:56:45,127 | DEBUG | ledger_manager.py ( 244) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0} from Node1 2017-10-23 15:56:45,127 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-10-23 15:56:45,127 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-10-23 15:56:45,128 | DEBUG | ledger_manager.py ( 244) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0} from Node1 2017-10-23 15:56:45,128 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-10-23 15:56:45,128 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-10-23 15:56:45,129 | DEBUG | node.py (1261) | sendToElector | Node3 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0}, 'Node1') 2017-10-23 15:56:45,130 | DEBUG | primary_selector.py ( 103) | _processViewChangeDoneMessage | Node3's primary selector started processing of ViewChangeDone msg from Node1 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-10-23 15:56:45,130 | DEBUG | primary_selector.py ( 186) | _hasViewChangeQuorum | Node3 needs 1 ViewChangeDone messages 2017-10-23 15:56:45,130 | DEBUG | primary_selector.py ( 258) | _startSelection | Node3 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-10-23 15:56:45,132 | TRACE | zstack.py ( 472) | _receiveFromListener | Node3 got 1 messages through listener 2017-10-23 15:56:45,133 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node2: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}']} 2017-10-23 15:56:45,133 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'signature': None, 'op': 'BATCH', 'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}']}, 'Node2') 2017-10-23 15:56:45,133 | DEBUG | node.py (1328) | unpackNodeMsg | Node3 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}']} 2017-10-23 15:56:45,133 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-23 15:56:45,133 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE'}, 'Node2') 2017-10-23 15:56:45,134 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-23 15:56:45,134 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-23 15:56:45,134 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE'}, 'Node2') 2017-10-23 15:56:45,134 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-23 15:56:45,134 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-23 15:56:45,135 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'op': 'MESSAGE_REQUEST'}, 'Node2') 2017-10-23 15:56:45,135 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-23 15:56:45,135 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-23 15:56:45,135 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'op': 'MESSAGE_REQUEST'}, 'Node2') 2017-10-23 15:56:45,135 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-23 15:56:45,135 | DEBUG | ledger_manager.py ( 244) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0} from Node2 2017-10-23 15:56:45,136 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-10-23 15:56:45,136 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-10-23 15:56:45,136 | DEBUG | ledger_manager.py ( 309) | processLedgerStatus | Node3 found out from {'Node1', 'Node2'} that its ledger of type 2 is latest 2017-10-23 15:56:45,136 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found from ledger status LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0} that it does not need catchup 2017-10-23 15:56:45,136 | DEBUG | node.py (1537) | preLedgerCatchUp | Node3 going to process any ordered requests before starting catchup. 2017-10-23 15:56:45,136 | DEBUG | replica.py (2130) | _remove_ordered_from_queue | Node3:0 going to remove 0 Ordered messages from outbox 2017-10-23 15:56:45,136 | DEBUG | node.py (1918) | force_process_ordered | Node3 processed 0 Ordered batches for instance 0 before starting catch up 2017-10-23 15:56:45,136 | DEBUG | replica.py (2130) | _remove_ordered_from_queue | Node3:1 going to remove 0 Ordered messages from outbox 2017-10-23 15:56:45,136 | DEBUG | node.py (1918) | force_process_ordered | Node3 processed 0 Ordered batches for instance 1 before starting catch up 2017-10-23 15:56:45,136 | DEBUG | node.py (2451) | processStashedOrderedReqs | Node3 processed 0 stashed ordered requests 2017-10-23 15:56:45,136 | DEBUG | monitor.py ( 183) | reset | Monitor being reset 2017-10-23 15:56:45,137 | DEBUG | node.py (1547) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 2 2017-10-23 15:56:45,137 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-10-23 15:56:45,137 | INFO | pool_config.py ( 33) | processLedger | processing config ledger for any POOL_CONFIGs 2017-10-23 15:56:45,137 | INFO | upgrader.py ( 145) | processLedger | DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya processing config ledger for any upgrades 2017-10-23 15:56:45,137 | DEBUG | node.py (2593) | send | Node3 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node1'] 2017-10-23 15:56:45,137 | DEBUG | node.py ( 832) | _ask_for_ledger_status | Node3 asking Node1 for ledger status of ledger 1 2017-10-23 15:56:45,137 | DEBUG | node.py (2593) | send | Node3 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node2'] 2017-10-23 15:56:45,138 | DEBUG | node.py ( 832) | _ask_for_ledger_status | Node3 asking Node2 for ledger status of ledger 1 2017-10-23 15:56:45,138 | DEBUG | node.py (2593) | send | Node3 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to all recipients: ['Node1', 'Node2', 'Node4'] 2017-10-23 15:56:45,138 | DEBUG | node.py ( 832) | _ask_for_ledger_status | Node3 asking Node3 for ledger status of ledger 1 2017-10-23 15:56:45,138 | DEBUG | node.py (2593) | send | Node3 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-10-23 15:56:45,138 | DEBUG | node.py ( 832) | _ask_for_ledger_status | Node3 asking Node4 for ledger status of ledger 1 2017-10-23 15:56:45,138 | DEBUG | ledger_manager.py (1006) | processStashedLedgerStatuses | Node3 going to process 0 stashed ledger statuses for ledger 1 2017-10-23 15:56:45,138 | INFO | ledger_manager.py ( 831) | catchupCompleted | Node3 completed catching up ledger 2, caught up 0 in total 2017-10-23 15:56:45,139 | DEBUG | ledger_manager.py ( 244) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0} from Node2 2017-10-23 15:56:45,139 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-10-23 15:56:45,139 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-10-23 15:56:45,139 | DEBUG | node.py (2593) | send | Node3 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node2'] 2017-10-23 15:56:45,140 | DEBUG | node.py (2593) | send | Node3 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node2'] 2017-10-23 15:56:45,141 | DEBUG | batched.py ( 89) | flushOutBoxes | Node3 batching 2 msgs to Node1 into one transmission 2017-10-23 15:56:45,141 | TRACE | batched.py ( 90) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}']) 2017-10-23 15:56:45,141 | TRACE | batched.py ( 98) | flushOutBoxes | Node3 sending payload to Node1: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}"],"op":"BATCH","signature":null}' 2017-10-23 15:56:45,141 | DEBUG | zstack.py ( 699) | transmit | Node3 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}"],"op":"BATCH","signature":null}' to Node1 2017-10-23 15:56:45,142 | DEBUG | batched.py ( 89) | flushOutBoxes | Node3 batching 4 msgs to Node2 into one transmission 2017-10-23 15:56:45,142 | TRACE | batched.py ( 90) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}']) 2017-10-23 15:56:45,142 | TRACE | batched.py ( 98) | flushOutBoxes | Node3 sending payload to Node2: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}"],"op":"BATCH","signature":null}' 2017-10-23 15:56:45,142 | DEBUG | zstack.py ( 699) | transmit | Node3 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}"],"op":"BATCH","signature":null}' to Node2 2017-10-23 15:56:45,143 | DEBUG | batched.py ( 89) | flushOutBoxes | Node3 batching 2 msgs to Node4 into one transmission 2017-10-23 15:56:45,143 | TRACE | batched.py ( 90) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', b'{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}']) 2017-10-23 15:56:45,143 | TRACE | batched.py ( 98) | flushOutBoxes | Node3 sending payload to Node4: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}"],"op":"BATCH","signature":null}' 2017-10-23 15:56:45,143 | DEBUG | zstack.py ( 699) | transmit | Node3 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_REQUEST\\"}"],"op":"BATCH","signature":null}' to Node4 2017-10-23 15:56:45,143 | WARNING | zstack.py ( 704) | transmit | Remote Node4 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-23 15:56:45,158 | TRACE | zstack.py ( 472) | _receiveFromListener | Node3 got 1 messages through listener 2017-10-23 15:56:45,159 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node1: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}']} 2017-10-23 15:56:45,159 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}'], 'signature': None, 'op': 'BATCH'}, 'Node1') 2017-10-23 15:56:45,160 | DEBUG | node.py (1328) | unpackNodeMsg | Node3 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}']} 2017-10-23 15:56:45,160 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 6, 'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 15:56:45,160 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'msg': {'txnSeqNo': 6, 'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE'}, 'Node1') 2017-10-23 15:56:45,160 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 6, 'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 15:56:45,161 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 6, 'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 15:56:45,161 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'msg': {'txnSeqNo': 6, 'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE'}, 'Node1') 2017-10-23 15:56:45,161 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 6, 'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 15:56:45,162 | DEBUG | ledger_manager.py ( 244) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'txnSeqNo': 6} from Node1 2017-10-23 15:56:45,162 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 1 of size 6 with 6 2017-10-23 15:56:45,162 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 1 of size 6 with 6 2017-10-23 15:56:45,163 | DEBUG | ledger_manager.py ( 244) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'txnSeqNo': 6} from Node1 2017-10-23 15:56:45,163 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 1 of size 6 with 6 2017-10-23 15:56:45,163 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 1 of size 6 with 6 2017-10-23 15:56:45,180 | TRACE | zstack.py ( 472) | _receiveFromListener | Node3 got 1 messages through listener 2017-10-23 15:56:45,181 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node2: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}']} 2017-10-23 15:56:45,181 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'signature': None, 'op': 'BATCH', 'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}']}, 'Node2') 2017-10-23 15:56:45,181 | DEBUG | node.py (1328) | unpackNodeMsg | Node3 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}']} 2017-10-23 15:56:45,181 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 6, 'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 15:56:45,181 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'msg': {'txnSeqNo': 6, 'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE'}, 'Node2') 2017-10-23 15:56:45,182 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 6, 'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 15:56:45,182 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 6, 'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 15:56:45,182 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'msg': {'txnSeqNo': 6, 'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE'}, 'Node2') 2017-10-23 15:56:45,182 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 6, 'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 15:56:45,182 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 15:56:45,182 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'op': 'MESSAGE_REQUEST'}, 'Node2') 2017-10-23 15:56:45,182 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 15:56:45,182 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 15:56:45,183 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'op': 'MESSAGE_REQUEST'}, 'Node2') 2017-10-23 15:56:45,183 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 15:56:45,183 | DEBUG | ledger_manager.py ( 244) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'txnSeqNo': 6} from Node2 2017-10-23 15:56:45,183 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 1 of size 6 with 6 2017-10-23 15:56:45,183 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 1 of size 6 with 6 2017-10-23 15:56:45,184 | DEBUG | ledger_manager.py ( 309) | processLedgerStatus | Node3 found out from {'Node1', 'Node2'} that its ledger of type 1 is latest 2017-10-23 15:56:45,184 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found from ledger status LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'txnSeqNo': 6} that it does not need catchup 2017-10-23 15:56:45,184 | DEBUG | node.py (1537) | preLedgerCatchUp | Node3 going to process any ordered requests before starting catchup. 2017-10-23 15:56:45,184 | DEBUG | replica.py (2130) | _remove_ordered_from_queue | Node3:0 going to remove 0 Ordered messages from outbox 2017-10-23 15:56:45,184 | DEBUG | node.py (1918) | force_process_ordered | Node3 processed 0 Ordered batches for instance 0 before starting catch up 2017-10-23 15:56:45,184 | DEBUG | replica.py (2130) | _remove_ordered_from_queue | Node3:1 going to remove 0 Ordered messages from outbox 2017-10-23 15:56:45,184 | DEBUG | node.py (1918) | force_process_ordered | Node3 processed 0 Ordered batches for instance 1 before starting catch up 2017-10-23 15:56:45,184 | DEBUG | node.py (2451) | processStashedOrderedReqs | Node3 processed 0 stashed ordered requests 2017-10-23 15:56:45,184 | DEBUG | monitor.py ( 183) | reset | Monitor being reset 2017-10-23 15:56:45,184 | DEBUG | node.py (1547) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 1 2017-10-23 15:56:45,184 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 1 of size 6 with 6 2017-10-23 15:56:45,184 | INFO | ledger_manager.py ( 831) | catchupCompleted | Node3 completed catching up ledger 1, caught up 0 in total 2017-10-23 15:56:45,185 | DEBUG | node.py (1653) | num_txns_caught_up_in_last_catchup | Node3 caught up to 0 txns in the last catchup 2017-10-23 15:56:45,185 | DEBUG | node.py (2451) | processStashedOrderedReqs | Node3 processed 0 stashed ordered requests 2017-10-23 15:56:45,185 | DEBUG | monitor.py ( 183) | reset | Monitor being reset 2017-10-23 15:56:45,185 | DEBUG | primary_selector.py ( 186) | _hasViewChangeQuorum | Node3 needs 1 ViewChangeDone messages 2017-10-23 15:56:45,185 | DEBUG | node.py (1625) | caught_up_for_current_view | Node3 does not have view change quorum for view 0 2017-10-23 15:56:45,185 | DEBUG | node.py (1608) | is_catchup_needed | Node3 is not caught up for the current view 0 2017-10-23 15:56:45,185 | DEBUG | node.py (1653) | num_txns_caught_up_in_last_catchup | Node3 caught up to 0 txns in the last catchup 2017-10-23 15:56:45,185 | DEBUG | node.py (1611) | is_catchup_needed | Node3 ordered till last prepared certificate 2017-10-23 15:56:45,185 | INFO | node.py (1593) | allLedgersCaughtUp | Node3 does not need any more catchups 2017-10-23 15:56:45,190 | DEBUG | primary_decider.py ( 131) | send | Node3's elector sending VIEW_CHANGE_DONE{'ledgerInfo': [(0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'), (1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'name': 'Node1', 'viewNo': 0} 2017-10-23 15:56:45,190 | INFO | primary_selector.py ( 192) | _hasViewChangeQuorum | Node3 got view change quorum (2 >= 2) 2017-10-23 15:56:45,191 | DEBUG | primary_selector.py ( 209) | has_view_change_from_primary | Node3 received ViewChangeDone from primary Node1 2017-10-23 15:56:45,192 | DEBUG | primary_selector.py ( 231) | has_sufficient_same_view_change_done_messages | Node3 found acceptable primary Node1 and ledger info ((0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'), (1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')) 2017-10-23 15:56:45,192 | DEBUG | primary_selector.py ( 272) | _startSelection | Node3 starting selection 2017-10-23 15:56:45,193 | DISPLAY | primary_selector.py ( 284) | _startSelection | Node3:0 selected primary Node1:0 for instance 0 (view 0) 2017-10-23 15:56:45,193 | INFO | node.py ( 482) | start_participating | Node3 started participating 2017-10-23 15:56:45,193 | INFO | replica.py ( 393) | primaryName | Node3:0 setting primaryName for view no 0 to: Node1:0 2017-10-23 15:56:45,193 | DEBUG | replica.py (1625) | _gc | Node3:0 cleaning up till (0, 0) 2017-10-23 15:56:45,193 | DEBUG | replica.py (1640) | _gc | Node3:0 found 0 3-phase keys to clean 2017-10-23 15:56:45,193 | DEBUG | replica.py (1642) | _gc | Node3:0 found 0 request keys to clean 2017-10-23 15:56:45,194 | INFO | replica.py ( 300) | h | Node3:0 set watermarks as 0 300 2017-10-23 15:56:45,194 | DISPLAY | primary_selector.py ( 307) | _startSelection | Node3:0 declares view change 0 as completed for instance 0, new primary is Node1:0, ledger info is [(0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'), (1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-10-23 15:56:45,194 | DISPLAY | primary_selector.py ( 284) | _startSelection | Node3:1 selected primary Node2:1 for instance 1 (view 0) 2017-10-23 15:56:45,195 | INFO | replica.py ( 393) | primaryName | Node3:1 setting primaryName for view no 0 to: Node2:1 2017-10-23 15:56:45,195 | DEBUG | replica.py (1625) | _gc | Node3:1 cleaning up till (0, 0) 2017-10-23 15:56:45,195 | DEBUG | replica.py (1640) | _gc | Node3:1 found 0 3-phase keys to clean 2017-10-23 15:56:45,195 | DEBUG | replica.py (1642) | _gc | Node3:1 found 0 request keys to clean 2017-10-23 15:56:45,195 | INFO | replica.py ( 300) | h | Node3:1 set watermarks as 0 300 2017-10-23 15:56:45,195 | DEBUG | replica.py ( 494) | _setup_for_non_master | Node3:1 Setting last ordered for non-master as (0, 0) 2017-10-23 15:56:45,195 | DEBUG | replica.py ( 310) | last_ordered_3pc | Node3:1 set last ordered as (0, 0) 2017-10-23 15:56:45,195 | DISPLAY | primary_selector.py ( 307) | _startSelection | Node3:1 declares view change 0 as completed for instance 1, new primary is Node2:1, ledger info is [(0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'), (1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-10-23 15:56:45,196 | DEBUG | ledger_manager.py ( 244) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'txnSeqNo': 6} from Node2 2017-10-23 15:56:45,196 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 1 of size 6 with 6 2017-10-23 15:56:45,196 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 1 of size 6 with 6 2017-10-23 15:56:45,196 | DEBUG | node.py (2593) | send | Node3 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'txnSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node2'] 2017-10-23 15:56:45,197 | DEBUG | node.py (2593) | send | Node3 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'txnSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node2'] 2017-10-23 15:56:45,198 | DEBUG | node.py (2593) | send | Node3 sending message VIEW_CHANGE_DONE{'ledgerInfo': [(0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'), (1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'name': 'Node1', 'viewNo': 0} to all recipients: ['Node1', 'Node2', 'Node4'] 2017-10-23 15:56:45,198 | DEBUG | zstack.py ( 699) | transmit | Node3 transmitting message b'{"ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,6,"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0,"op":"VIEW_CHANGE_DONE"}' to Node1 2017-10-23 15:56:45,199 | TRACE | batched.py ( 85) | flushOutBoxes | Node3 sending msg b'{"ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,6,"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0,"op":"VIEW_CHANGE_DONE"}' to Node1 2017-10-23 15:56:45,199 | DEBUG | batched.py ( 89) | flushOutBoxes | Node3 batching 3 msgs to Node2 into one transmission 2017-10-23 15:56:45,199 | TRACE | batched.py ( 90) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', b'{"msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', b'{"ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,6,"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0,"op":"VIEW_CHANGE_DONE"}']) 2017-10-23 15:56:45,199 | TRACE | batched.py ( 98) | flushOutBoxes | Node3 sending payload to Node2: b'{"messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":6,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":6,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"ledgerInfo\\":[[0,4,\\"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA\\"],[1,6,\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"name\\":\\"Node1\\",\\"viewNo\\":0,\\"op\\":\\"VIEW_CHANGE_DONE\\"}"],"op":"BATCH","signature":null}' 2017-10-23 15:56:45,210 | DEBUG | zstack.py ( 699) | transmit | Node3 transmitting message b'{"messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":6,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":6,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"ledgerInfo\\":[[0,4,\\"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA\\"],[1,6,\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"name\\":\\"Node1\\",\\"viewNo\\":0,\\"op\\":\\"VIEW_CHANGE_DONE\\"}"],"op":"BATCH","signature":null}' to Node2 2017-10-23 15:56:45,211 | DEBUG | zstack.py ( 699) | transmit | Node3 transmitting message b'{"ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,6,"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0,"op":"VIEW_CHANGE_DONE"}' to Node4 2017-10-23 15:56:45,211 | WARNING | zstack.py ( 704) | transmit | Remote Node4 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-23 15:56:45,211 | TRACE | batched.py ( 85) | flushOutBoxes | Node3 sending msg b'{"ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,6,"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0,"op":"VIEW_CHANGE_DONE"}' to Node4 2017-10-23 15:56:45,226 | TRACE | zstack.py ( 472) | _receiveFromListener | Node3 got 1 messages through listener 2017-10-23 15:56:45,227 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node2: VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-10-23 15:56:45,227 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0, 'op': 'VIEW_CHANGE_DONE'}, 'Node2') 2017-10-23 15:56:45,227 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-10-23 15:56:45,227 | DEBUG | node.py (1261) | sendToElector | Node3 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0}, 'Node2') 2017-10-23 15:56:45,229 | DEBUG | primary_selector.py ( 103) | _processViewChangeDoneMessage | Node3's primary selector started processing of ViewChangeDone msg from Node2 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-10-23 15:56:45,229 | DEBUG | message_processor.py ( 28) | discard | Node3 discarding message VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} because it already decided primary which is Node1:0 2017-10-23 15:56:45,589 | TRACE | zstack.py ( 472) | _receiveFromListener | Node3 got 1 messages through listener 2017-10-23 15:56:45,589 | DEBUG | zstack.py ( 652) | handlePingPong | Node3 got ping from Node4 2017-10-23 15:56:45,589 | DEBUG | zstack.py ( 643) | sendPingPong | Node3 will be sending in batch 2017-10-23 15:56:45,591 | DEBUG | zstack.py ( 699) | transmit | Node3 transmitting message b'po' to Node4 2017-10-23 15:56:45,591 | TRACE | batched.py ( 85) | flushOutBoxes | Node3 sending msg b'po' to Node4 2017-10-23 15:56:45,631 | TRACE | zstack.py ( 472) | _receiveFromListener | Node3 got 1 messages through listener 2017-10-23 15:56:45,632 | DEBUG | zstack.py ( 658) | handlePingPong | Node3 got pong from Node4 2017-10-23 15:56:45,632 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node4: BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}']} 2017-10-23 15:56:45,632 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}'], 'signature': None, 'op': 'BATCH'}, 'Node4') 2017-10-23 15:56:45,632 | DEBUG | node.py (1328) | unpackNodeMsg | Node3 processing a batch BATCH{'signature': None, 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', '{"msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}']} 2017-10-23 15:56:45,633 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-23 15:56:45,633 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE'}, 'Node4') 2017-10-23 15:56:45,633 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-23 15:56:45,633 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-23 15:56:45,633 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE'}, 'Node4') 2017-10-23 15:56:45,633 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 0, 'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-23 15:56:45,633 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 6, 'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 15:56:45,633 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'msg': {'txnSeqNo': 6, 'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE'}, 'Node4') 2017-10-23 15:56:45,634 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 6, 'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 15:56:45,634 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_RESPONSE{'msg': {'txnSeqNo': 6, 'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 15:56:45,634 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'msg': {'txnSeqNo': 6, 'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE'}, 'Node4') 2017-10-23 15:56:45,634 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'txnSeqNo': 6, 'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 15:56:45,634 | DEBUG | ledger_manager.py ( 244) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0} from Node4 2017-10-23 15:56:45,635 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-10-23 15:56:45,635 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-10-23 15:56:45,635 | DEBUG | ledger_manager.py ( 309) | processLedgerStatus | Node3 found out from {'Node2', 'Node4'} that its ledger of type 2 is latest 2017-10-23 15:56:45,635 | DEBUG | ledger_manager.py ( 244) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0} from Node4 2017-10-23 15:56:45,635 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-10-23 15:56:45,635 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 2 of size 0 with 0 2017-10-23 15:56:45,635 | DEBUG | ledger_manager.py ( 309) | processLedgerStatus | Node3 found out from {'Node2', 'Node4'} that its ledger of type 2 is latest 2017-10-23 15:56:45,636 | DEBUG | ledger_manager.py ( 244) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'txnSeqNo': 6} from Node4 2017-10-23 15:56:45,636 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 1 of size 6 with 6 2017-10-23 15:56:45,636 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 1 of size 6 with 6 2017-10-23 15:56:45,636 | DEBUG | ledger_manager.py ( 309) | processLedgerStatus | Node3 found out from {'Node2', 'Node4'} that its ledger of type 1 is latest 2017-10-23 15:56:45,636 | DEBUG | ledger_manager.py ( 244) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'txnSeqNo': 6} from Node4 2017-10-23 15:56:45,637 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 1 of size 6 with 6 2017-10-23 15:56:45,637 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 1 of size 6 with 6 2017-10-23 15:56:45,637 | DEBUG | ledger_manager.py ( 309) | processLedgerStatus | Node3 found out from {'Node2', 'Node4'} that its ledger of type 1 is latest 2017-10-23 15:56:45,638 | DEBUG | keep_in_touch.py ( 68) | conns | Node3's connections changed from {'Node1', 'Node2'} to {'Node1', 'Node2', 'Node4'} 2017-10-23 15:56:45,638 | INFO | keep_in_touch.py ( 96) | _connsChanged | Node3 now connected to Node4 2017-10-23 15:56:45,638 | DEBUG | motor.py ( 34) | set_status | Node3 changing status from started_hungry to started 2017-10-23 15:56:45,639 | DEBUG | node.py ( 918) | checkInstances | Node3 choosing to start election on the basis of count 4 and nodes {'Node1', 'Node2', 'Node4'} 2017-10-23 15:56:45,639 | DEBUG | node.py ( 879) | send_current_state_to_lagging_node | Node3 sending current state CURRENT_STATE{'primary': [VIEW_CHANGE_DONE{'ledgerInfo': ((0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'), (1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')), 'name': 'Node1', 'viewNo': 0}], 'viewNo': 0} to lagged node Node4 2017-10-23 15:56:45,639 | DEBUG | node.py (2593) | send | Node3 sending message CURRENT_STATE{'primary': [VIEW_CHANGE_DONE{'ledgerInfo': ((0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'), (1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')), 'name': 'Node1', 'viewNo': 0}], 'viewNo': 0} to 1 recipients: ['Node4'] 2017-10-23 15:56:45,639 | DEBUG | node.py (2593) | send | Node3 sending message LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'txnSeqNo': 4} to 1 recipients: ['Node4'] 2017-10-23 15:56:45,644 | DEBUG | node.py (2593) | send | Node3 sending message LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'txnSeqNo': 6} to 1 recipients: ['Node4'] 2017-10-23 15:56:45,644 | DEBUG | node.py (2593) | send | Node3 sending message LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0} to 1 recipients: ['Node4'] 2017-10-23 15:56:45,645 | TRACE | zstack.py ( 472) | _receiveFromListener | Node3 got 1 messages through listener 2017-10-23 15:56:45,646 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node4: BATCH{'signature': None, 'messages': ['{"op":"CURRENT_STATE","viewNo":0,"primary":[]}', '{"ppSeqNo":null,"ledgerId":0,"merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA","txnSeqNo":4,"op":"LEDGER_STATUS","viewNo":null}']} 2017-10-23 15:56:45,646 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'messages': ['{"op":"CURRENT_STATE","viewNo":0,"primary":[]}', '{"ppSeqNo":null,"ledgerId":0,"merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA","txnSeqNo":4,"op":"LEDGER_STATUS","viewNo":null}'], 'signature': None, 'op': 'BATCH'}, 'Node4') 2017-10-23 15:56:45,646 | DEBUG | node.py (1328) | unpackNodeMsg | Node3 processing a batch BATCH{'signature': None, 'messages': ['{"op":"CURRENT_STATE","viewNo":0,"primary":[]}', '{"ppSeqNo":null,"ledgerId":0,"merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA","txnSeqNo":4,"op":"LEDGER_STATUS","viewNo":null}']} 2017-10-23 15:56:45,646 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node4: CURRENT_STATE{'primary': [], 'viewNo': 0} 2017-10-23 15:56:45,646 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'primary': [], 'viewNo': 0, 'op': 'CURRENT_STATE'}, 'Node4') 2017-10-23 15:56:45,647 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox CURRENT_STATE{'primary': [], 'viewNo': 0} 2017-10-23 15:56:45,647 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node4: LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'txnSeqNo': 4} 2017-10-23 15:56:45,647 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'ledgerId': 0, 'ppSeqNo': None, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'viewNo': None, 'op': 'LEDGER_STATUS', 'txnSeqNo': 4}, 'Node4') 2017-10-23 15:56:45,647 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'txnSeqNo': 4} 2017-10-23 15:56:45,648 | DEBUG | node.py ( 884) | process_current_state_message | Node3 processing current state CURRENT_STATE{'primary': [], 'viewNo': 0} from Node4 2017-10-23 15:56:45,648 | DEBUG | ledger_manager.py ( 244) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 0, 'viewNo': None, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'txnSeqNo': 4} from Node4 2017-10-23 15:56:45,648 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-10-23 15:56:45,648 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-10-23 15:56:45,649 | DEBUG | batched.py ( 89) | flushOutBoxes | Node3 batching 4 msgs to Node4 into one transmission 2017-10-23 15:56:45,650 | TRACE | batched.py ( 90) | flushOutBoxes | messages: deque([b'{"primary":[{"ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,6,"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0}],"viewNo":0,"op":"CURRENT_STATE"}', b'{"ledgerId":0,"merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA","ppSeqNo":null,"op":"LEDGER_STATUS","viewNo":null,"txnSeqNo":4}', b'{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"op":"LEDGER_STATUS","viewNo":null,"txnSeqNo":6}', b'{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"op":"LEDGER_STATUS","viewNo":null,"txnSeqNo":0}']) 2017-10-23 15:56:45,650 | TRACE | batched.py ( 98) | flushOutBoxes | Node3 sending payload to Node4: b'{"messages":["{\\"primary\\":[{\\"ledgerInfo\\":[[0,4,\\"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA\\"],[1,6,\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"name\\":\\"Node1\\",\\"viewNo\\":0}],\\"viewNo\\":0,\\"op\\":\\"CURRENT_STATE\\"}","{\\"ledgerId\\":0,\\"merkleRoot\\":\\"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA\\",\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null,\\"txnSeqNo\\":4}","{\\"ledgerId\\":1,\\"merkleRoot\\":\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\",\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null,\\"txnSeqNo\\":6}","{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null,\\"txnSeqNo\\":0}"],"op":"BATCH","signature":null}' 2017-10-23 15:56:45,650 | DEBUG | zstack.py ( 699) | transmit | Node3 transmitting message b'{"messages":["{\\"primary\\":[{\\"ledgerInfo\\":[[0,4,\\"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA\\"],[1,6,\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"name\\":\\"Node1\\",\\"viewNo\\":0}],\\"viewNo\\":0,\\"op\\":\\"CURRENT_STATE\\"}","{\\"ledgerId\\":0,\\"merkleRoot\\":\\"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA\\",\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null,\\"txnSeqNo\\":4}","{\\"ledgerId\\":1,\\"merkleRoot\\":\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\",\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null,\\"txnSeqNo\\":6}","{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null,\\"txnSeqNo\\":0}"],"op":"BATCH","signature":null}' to Node4 2017-10-23 15:56:45,693 | TRACE | zstack.py ( 472) | _receiveFromListener | Node3 got 1 messages through listener 2017-10-23 15:56:45,693 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node4: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,6,"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":0,"name":"Node1"}']} 2017-10-23 15:56:45,693 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,6,"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":0,"name":"Node1"}'], 'signature': None, 'op': 'BATCH'}, 'Node4') 2017-10-23 15:56:45,694 | DEBUG | node.py (1328) | unpackNodeMsg | Node3 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,6,"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":0,"name":"Node1"}']} 2017-10-23 15:56:45,694 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-23 15:56:45,694 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'op': 'MESSAGE_REQUEST'}, 'Node4') 2017-10-23 15:56:45,694 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-23 15:56:45,694 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-23 15:56:45,694 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'op': 'MESSAGE_REQUEST'}, 'Node4') 2017-10-23 15:56:45,694 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-23 15:56:45,694 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 15:56:45,695 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'op': 'MESSAGE_REQUEST'}, 'Node4') 2017-10-23 15:56:45,695 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 15:56:45,695 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 15:56:45,695 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'op': 'MESSAGE_REQUEST'}, 'Node4') 2017-10-23 15:56:45,695 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 15:56:45,695 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node4: VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-10-23 15:56:45,695 | INFO | node.py (1275) | handleOneNodeMsg | Node3 msg validated ({'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0, 'op': 'VIEW_CHANGE_DONE'}, 'Node4') 2017-10-23 15:56:45,695 | DEBUG | node.py (1342) | postToNodeInBox | Node3 appending to nodeInbox VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-10-23 15:56:45,696 | DEBUG | node.py (2593) | send | Node3 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-10-23 15:56:45,696 | DEBUG | node.py (2593) | send | Node3 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-10-23 15:56:45,697 | DEBUG | node.py (2593) | send | Node3 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'txnSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-10-23 15:56:45,698 | DEBUG | node.py (2593) | send | Node3 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'ppSeqNo': None, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'txnSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-10-23 15:56:45,698 | DEBUG | node.py (1261) | sendToElector | Node3 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0}, 'Node4') 2017-10-23 15:56:45,699 | DEBUG | primary_selector.py ( 103) | _processViewChangeDoneMessage | Node3's primary selector started processing of ViewChangeDone msg from Node4 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-10-23 15:56:45,699 | DEBUG | message_processor.py ( 28) | discard | Node3 discarding message VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} because it already decided primary which is Node1:0 2017-10-23 15:56:45,700 | DEBUG | batched.py ( 89) | flushOutBoxes | Node3 batching 4 msgs to Node4 into one transmission 2017-10-23 15:56:45,700 | TRACE | batched.py ( 90) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', b'{"msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', b'{"msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}']) 2017-10-23 15:56:45,700 | TRACE | batched.py ( 98) | flushOutBoxes | Node3 sending payload to Node4: b'{"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":6,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":6,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}"],"op":"BATCH","signature":null}' 2017-10-23 15:56:45,700 | DEBUG | zstack.py ( 699) | transmit | Node3 transmitting message b'{"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":2},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":6,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":6,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"params\\":{\\"ledgerId\\":1},\\"op\\":\\"MESSAGE_RESPONSE\\"}"],"op":"BATCH","signature":null}' to Node4 2017-10-23 15:56:52,551 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node3 running action checkPerformance with id 1 2017-10-23 15:56:52,551 | TRACE | node.py (2005) | checkPerformance | Node3 checking its performance 2017-10-23 15:56:52,551 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 15:56:52,551 | DEBUG | monitor.py ( 290) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 15:56:52,551 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 15:56:52,551 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 15:56:52,551 | DEBUG | node.py (2022) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 15:56:52,552 | TRACE | has_action_queue.py ( 34) | _schedule | Node3 scheduling action checkPerformance with id 3 to run in 10 seconds 2017-10-23 15:56:57,611 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 15:56:57,612 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 15:56:57,612 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 15:56:57,612 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 15:57:02,564 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node3 running action checkPerformance with id 3 2017-10-23 15:57:02,564 | TRACE | node.py (2005) | checkPerformance | Node3 checking its performance 2017-10-23 15:57:02,565 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 15:57:02,565 | DEBUG | monitor.py ( 290) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 15:57:02,565 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 15:57:02,565 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 15:57:02,565 | DEBUG | node.py (2022) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 15:57:02,565 | TRACE | has_action_queue.py ( 34) | _schedule | Node3 scheduling action checkPerformance with id 4 to run in 10 seconds 2017-10-23 15:57:12,576 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node3 running action checkPerformance with id 4 2017-10-23 15:57:12,576 | TRACE | node.py (2005) | checkPerformance | Node3 checking its performance 2017-10-23 15:57:12,576 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 15:57:12,576 | DEBUG | monitor.py ( 290) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 15:57:12,576 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 15:57:12,576 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 15:57:12,576 | DEBUG | node.py (2022) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 15:57:12,577 | TRACE | has_action_queue.py ( 34) | _schedule | Node3 scheduling action checkPerformance with id 5 to run in 10 seconds 2017-10-23 15:57:12,622 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 15:57:12,622 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 15:57:12,622 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 15:57:12,622 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 15:57:22,583 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node3 running action checkPerformance with id 5 2017-10-23 15:57:22,584 | TRACE | node.py (2005) | checkPerformance | Node3 checking its performance 2017-10-23 15:57:22,584 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 15:57:22,584 | DEBUG | monitor.py ( 290) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 15:57:22,584 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 15:57:22,584 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 15:57:22,584 | DEBUG | node.py (2022) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 15:57:22,584 | TRACE | has_action_queue.py ( 34) | _schedule | Node3 scheduling action checkPerformance with id 6 to run in 10 seconds 2017-10-23 15:57:27,633 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 15:57:27,633 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 15:57:27,633 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 15:57:27,634 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 15:57:32,595 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node3 running action checkPerformance with id 6 2017-10-23 15:57:32,596 | TRACE | node.py (2005) | checkPerformance | Node3 checking its performance 2017-10-23 15:57:32,596 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 15:57:32,596 | DEBUG | monitor.py ( 290) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 15:57:32,596 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 15:57:32,596 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 15:57:32,596 | DEBUG | node.py (2022) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 15:57:32,596 | TRACE | has_action_queue.py ( 34) | _schedule | Node3 scheduling action checkPerformance with id 7 to run in 10 seconds 2017-10-23 15:57:42,550 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node3 running action checkNodeRequestSpike with id 2 2017-10-23 15:57:42,550 | DEBUG | node.py (2026) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 15:57:42,550 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 15:57:42,550 | TRACE | has_action_queue.py ( 34) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 8 to run in 60 seconds 2017-10-23 15:57:42,550 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node3 running action checkPerformance with id 1 2017-10-23 15:57:42,550 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 15:57:42,550 | TRACE | has_action_queue.py ( 34) | _schedule | Node3 scheduling action checkPerformance with id 2 to run in 60 seconds 2017-10-23 15:57:42,610 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node3 running action checkPerformance with id 7 2017-10-23 15:57:42,610 | TRACE | node.py (2005) | checkPerformance | Node3 checking its performance 2017-10-23 15:57:42,610 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 15:57:42,611 | DEBUG | monitor.py ( 290) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 15:57:42,611 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 15:57:42,611 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 15:57:42,611 | DEBUG | node.py (2022) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 15:57:42,611 | TRACE | has_action_queue.py ( 34) | _schedule | Node3 scheduling action checkPerformance with id 9 to run in 10 seconds 2017-10-23 15:57:42,647 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 15:57:42,647 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 15:57:42,647 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 15:57:42,647 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 15:57:52,620 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node3 running action checkPerformance with id 9 2017-10-23 15:57:52,621 | TRACE | node.py (2005) | checkPerformance | Node3 checking its performance 2017-10-23 15:57:52,621 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 15:57:52,621 | DEBUG | monitor.py ( 290) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 15:57:52,621 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 15:57:52,621 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 15:57:52,621 | DEBUG | node.py (2022) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 15:57:52,621 | TRACE | has_action_queue.py ( 34) | _schedule | Node3 scheduling action checkPerformance with id 10 to run in 10 seconds 2017-10-23 15:57:57,647 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 15:57:57,647 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 15:57:57,648 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 15:57:57,648 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 15:58:02,633 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node3 running action checkPerformance with id 10 2017-10-23 15:58:02,633 | TRACE | node.py (2005) | checkPerformance | Node3 checking its performance 2017-10-23 15:58:02,633 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 15:58:02,633 | DEBUG | monitor.py ( 290) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 15:58:02,633 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 15:58:02,633 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 15:58:02,633 | DEBUG | node.py (2022) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 15:58:02,633 | TRACE | has_action_queue.py ( 34) | _schedule | Node3 scheduling action checkPerformance with id 11 to run in 10 seconds 2017-10-23 15:58:12,637 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node3 running action checkPerformance with id 11 2017-10-23 15:58:12,638 | TRACE | node.py (2005) | checkPerformance | Node3 checking its performance 2017-10-23 15:58:12,638 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 15:58:12,638 | DEBUG | monitor.py ( 290) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 15:58:12,638 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 15:58:12,638 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 15:58:12,638 | DEBUG | node.py (2022) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 15:58:12,638 | TRACE | has_action_queue.py ( 34) | _schedule | Node3 scheduling action checkPerformance with id 12 to run in 10 seconds 2017-10-23 15:58:12,655 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 15:58:12,655 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 15:58:12,655 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 15:58:12,656 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 15:58:22,652 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node3 running action checkPerformance with id 12 2017-10-23 15:58:22,653 | TRACE | node.py (2005) | checkPerformance | Node3 checking its performance 2017-10-23 15:58:22,653 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 15:58:22,653 | DEBUG | monitor.py ( 290) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 15:58:22,653 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 15:58:22,653 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 15:58:22,653 | DEBUG | node.py (2022) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 15:58:22,653 | TRACE | has_action_queue.py ( 34) | _schedule | Node3 scheduling action checkPerformance with id 13 to run in 10 seconds 2017-10-23 15:58:27,657 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 15:58:27,657 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 15:58:27,657 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 15:58:27,657 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 15:58:32,662 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node3 running action checkPerformance with id 13 2017-10-23 15:58:32,662 | TRACE | node.py (2005) | checkPerformance | Node3 checking its performance 2017-10-23 15:58:32,662 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 15:58:32,662 | DEBUG | monitor.py ( 290) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 15:58:32,662 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 15:58:32,662 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 15:58:32,663 | DEBUG | node.py (2022) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 15:58:32,663 | TRACE | has_action_queue.py ( 34) | _schedule | Node3 scheduling action checkPerformance with id 14 to run in 10 seconds 2017-10-23 15:58:42,564 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node3 running action checkNodeRequestSpike with id 8 2017-10-23 15:58:42,564 | DEBUG | node.py (2026) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 15:58:42,564 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 15:58:42,564 | TRACE | has_action_queue.py ( 34) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 15 to run in 60 seconds 2017-10-23 15:58:42,564 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node3 running action checkPerformance with id 2 2017-10-23 15:58:42,564 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 15:58:42,564 | TRACE | has_action_queue.py ( 34) | _schedule | Node3 scheduling action checkPerformance with id 3 to run in 60 seconds 2017-10-23 15:58:42,667 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node3 running action checkPerformance with id 14 2017-10-23 15:58:42,667 | TRACE | node.py (2005) | checkPerformance | Node3 checking its performance 2017-10-23 15:58:42,667 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 15:58:42,667 | DEBUG | monitor.py ( 290) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 15:58:42,667 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 15:58:42,667 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 15:58:42,668 | DEBUG | node.py (2022) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 15:58:42,668 | TRACE | has_action_queue.py ( 34) | _schedule | Node3 scheduling action checkPerformance with id 16 to run in 10 seconds 2017-10-23 15:58:42,668 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 15:58:42,669 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 15:58:42,669 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 15:58:42,669 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 15:58:52,668 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node3 running action checkPerformance with id 16 2017-10-23 15:58:52,669 | TRACE | node.py (2005) | checkPerformance | Node3 checking its performance 2017-10-23 15:58:52,669 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 15:58:52,669 | DEBUG | monitor.py ( 290) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 15:58:52,669 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 15:58:52,669 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 15:58:52,669 | DEBUG | node.py (2022) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 15:58:52,669 | TRACE | has_action_queue.py ( 34) | _schedule | Node3 scheduling action checkPerformance with id 17 to run in 10 seconds 2017-10-23 15:58:57,678 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 15:58:57,678 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 15:58:57,679 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 15:58:57,679 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 15:59:02,674 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node3 running action checkPerformance with id 17 2017-10-23 15:59:02,674 | TRACE | node.py (2005) | checkPerformance | Node3 checking its performance 2017-10-23 15:59:02,674 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 15:59:02,675 | DEBUG | monitor.py ( 290) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 15:59:02,675 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 15:59:02,675 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 15:59:02,675 | DEBUG | node.py (2022) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 15:59:02,675 | TRACE | has_action_queue.py ( 34) | _schedule | Node3 scheduling action checkPerformance with id 18 to run in 10 seconds 2017-10-23 15:59:12,679 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node3 running action checkPerformance with id 18 2017-10-23 15:59:12,680 | TRACE | node.py (2005) | checkPerformance | Node3 checking its performance 2017-10-23 15:59:12,680 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 15:59:12,680 | DEBUG | monitor.py ( 290) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 15:59:12,680 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 15:59:12,680 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 15:59:12,680 | DEBUG | node.py (2022) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 15:59:12,680 | TRACE | has_action_queue.py ( 34) | _schedule | Node3 scheduling action checkPerformance with id 19 to run in 10 seconds 2017-10-23 15:59:12,681 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 15:59:12,681 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 15:59:12,681 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 15:59:12,681 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 15:59:22,689 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node3 running action checkPerformance with id 19 2017-10-23 15:59:22,689 | TRACE | node.py (2005) | checkPerformance | Node3 checking its performance 2017-10-23 15:59:22,689 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 15:59:22,689 | DEBUG | monitor.py ( 290) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 15:59:22,689 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 15:59:22,689 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 15:59:22,689 | DEBUG | node.py (2022) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 15:59:22,689 | TRACE | has_action_queue.py ( 34) | _schedule | Node3 scheduling action checkPerformance with id 20 to run in 10 seconds 2017-10-23 15:59:27,689 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 15:59:27,689 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 15:59:27,689 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 15:59:27,689 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 15:59:32,693 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node3 running action checkPerformance with id 20 2017-10-23 15:59:32,693 | TRACE | node.py (2005) | checkPerformance | Node3 checking its performance 2017-10-23 15:59:32,697 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 15:59:32,697 | DEBUG | monitor.py ( 290) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 15:59:32,697 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 15:59:32,697 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 15:59:32,697 | DEBUG | node.py (2022) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 15:59:32,697 | TRACE | has_action_queue.py ( 34) | _schedule | Node3 scheduling action checkPerformance with id 21 to run in 10 seconds 2017-10-23 15:59:41,614 | TRACE | zstack.py ( 472) | _receiveFromListener | Node3C got 1 messages through listener 2017-10-23 15:59:41,614 | DEBUG | zstack.py ( 652) | handlePingPong | Node3C got ping from b'xo5JUY.S$6PWbpmz5XzA to ('127.0.0.1', 30003) 2017-10-23 16:05:54,275 | DEBUG | base_events.py (1270) | _run_once | poll 3.414 ms took 0.012 ms: 1 events 2017-10-23 16:05:54,276 | DEBUG | base_events.py ( 719) | create_connection | connected to 127.0.0.1:'30003': (<_SelectorSocketTransport fd=75 read=polling write=>, ) 2017-10-23 16:05:54,284 | TRACE | has_action_queue.py ( 34) | _schedule | DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya scheduling action partial(_declareTimeoutExceeded) with id 1 to run in 600 seconds 2017-10-23 16:05:54,286 | TRACE | zstack.py ( 472) | _receiveFromListener | Node3 got 3 messages through listener 2017-10-23 16:05:54,286 | TRACE | signing.py ( 98) | serializeMsg | serialized msg {'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1508774754234674, 'operation': {'action': 'start', 'version': '1.1.37', 'timeout': 10, 'schedule': {'4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'justification': None, 'name': 'upgrade-1137', 'force': True, 'type': '109'}, 'signature': '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ'} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:action:start|force:True|justification:|name:upgrade-1137|schedule:4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA:2017-10-17T11:35:00.000000+00:00|4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc:2017-10-17T11:40:00.000000+00:00|8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb:2017-10-17T11:25:00.000000+00:00|DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya:2017-10-17T11:30:00.000000+00:00|Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv:2017-10-17T11:20:00.000000+00:00|sha256:f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3|timeout:10|type:109|version:1.1.37|reqId:1508774754234674 2017-10-23 16:05:54,287 | INFO | node.py (2250) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1508774754234674 2017-10-23 16:05:54,287 | DEBUG | node.py (1313) | validateNodeMsg | Node3 received node message from Node2: PROPAGATE{'request': {'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1508774754234674, 'operation': {'action': 'start', 'version': '1.1.37', 'timeout': 10, 'schedule': {'4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'justification': None, 'name': 'upgrade-1137', 'force': True, 'type': '109'}, 'signature': '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ'}, 'senderClient': 'xo5JUY.S$6PWbpmz5XzA wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.219 seconds 2017-10-23 16:05:54,603 | DEBUG | base_events.py ( 681) | create_connection | connect to ('127.0.0.1', 30003) 2017-10-23 16:05:54,604 | DEBUG | base_events.py (1270) | _run_once | poll 8.611 ms took 0.008 ms: 1 events 2017-10-23 16:05:54,605 | DEBUG | base_events.py ( 719) | create_connection | connected to 127.0.0.1:'30003': (<_SelectorSocketTransport fd=75 read=polling write=>, ) 2017-10-23 16:05:54,605 | TRACE | has_action_queue.py ( 34) | _schedule | DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya scheduling action partial(_declareTimeoutExceeded) with id 2 to run in 600 seconds 2017-10-23 16:05:57,914 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:05:57,915 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:05:57,915 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:05:57,919 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:06:02,989 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node3 running action checkPerformance with id 66 2017-10-23 16:06:02,989 | TRACE | node.py (2005) | checkPerformance | Node3 checking its performance 2017-10-23 16:06:02,989 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:06:02,990 | TRACE | monitor.py ( 299) | isMasterThroughputTooLow | Node3 master throughput ratio 0.9526391087544533 is acceptable. 2017-10-23 16:06:02,990 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:06:02,990 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:06:02,990 | DEBUG | node.py (2022) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:06:02,990 | TRACE | has_action_queue.py ( 34) | _schedule | Node3 scheduling action checkPerformance with id 67 to run in 10 seconds 2017-10-23 16:06:12,925 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:06:12,925 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:06:12,925 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:06:12,926 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:06:13,005 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node3 running action checkPerformance with id 67 2017-10-23 16:06:13,005 | TRACE | node.py (2005) | checkPerformance | Node3 checking its performance 2017-10-23 16:06:13,005 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:06:13,005 | TRACE | monitor.py ( 299) | isMasterThroughputTooLow | Node3 master throughput ratio 0.9526391087544533 is acceptable. 2017-10-23 16:06:13,005 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:06:13,005 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:06:13,005 | DEBUG | node.py (2022) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:06:13,006 | TRACE | has_action_queue.py ( 34) | _schedule | Node3 scheduling action checkPerformance with id 68 to run in 10 seconds 2017-10-23 16:06:23,011 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node3 running action checkPerformance with id 68 2017-10-23 16:06:23,011 | TRACE | node.py (2005) | checkPerformance | Node3 checking its performance 2017-10-23 16:06:23,011 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:06:23,012 | TRACE | monitor.py ( 299) | isMasterThroughputTooLow | Node3 master throughput ratio 0.9526391087544533 is acceptable. 2017-10-23 16:06:23,012 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:06:23,012 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:06:23,012 | DEBUG | node.py (2022) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:06:23,012 | TRACE | has_action_queue.py ( 34) | _schedule | Node3 scheduling action checkPerformance with id 69 to run in 10 seconds 2017-10-23 16:06:27,930 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:06:27,931 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:06:27,931 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:06:27,931 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:06:33,020 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node3 running action checkPerformance with id 69 2017-10-23 16:06:33,020 | TRACE | node.py (2005) | checkPerformance | Node3 checking its performance 2017-10-23 16:06:33,020 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:06:33,020 | TRACE | monitor.py ( 299) | isMasterThroughputTooLow | Node3 master throughput ratio 0.9526391087544533 is acceptable. 2017-10-23 16:06:33,020 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:06:33,020 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:06:33,020 | DEBUG | node.py (2022) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:06:33,020 | TRACE | has_action_queue.py ( 34) | _schedule | Node3 scheduling action checkPerformance with id 70 to run in 10 seconds 2017-10-23 16:06:45,964 | DEBUG | node_runner.py ( 18) | run_node | You can find logs in /home/sovrin/.sovrin/Node3.log 2017-10-23 16:06:45,965 | DEBUG | node_runner.py ( 21) | run_node | Sovrin related env vars: [] 2017-10-23 16:06:49,062 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: git 2017-10-23 16:06:49,123 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: hg 2017-10-23 16:06:49,251 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: svn 2017-10-23 16:06:49,252 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: bzr 2017-10-23 16:06:50,759 | DEBUG | selector_events.py ( 53) | __init__ | Using selector: EpollSelector 2017-10-23 16:06:50,759 | DEBUG | looper.py ( 125) | __init__ | Setting handler for SIGINT 2017-10-23 16:06:50,852 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-10-23 16:06:50,886 | DEBUG | ledger.py ( 72) | recoverTree | Recovering tree from hash store of size 7 2017-10-23 16:06:50,887 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.0004956529010087252 seconds 2017-10-23 16:06:50,982 | DEBUG | idr_cache.py ( 25) | __init__ | Initializing identity cache Node3 2017-10-23 16:06:51,004 | INFO | node.py (2420) | initStateFromLedger | Node3 found state to be empty, recreating from ledger 2017-10-23 16:06:51,198 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-10-23 16:06:51,229 | DEBUG | ledger.py ( 72) | recoverTree | Recovering tree from hash store of size 4 2017-10-23 16:06:51,230 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.0004594610072672367 seconds 2017-10-23 16:06:51,230 | INFO | node.py (2420) | initStateFromLedger | Node3 found state to be empty, recreating from ledger 2017-10-23 16:06:51,331 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-10-23 16:06:51,331 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-10-23 16:06:51,331 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: typing.Dict<~KT, ~VT>] because it does not have a 'pluginType' attribute 2017-10-23 16:06:51,332 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-10-23 16:06:51,332 | INFO | plugin_loader.py ( 117) | _load | plugin FirebaseStatsConsumer successfully loaded from module plugin_firebase_stats_consumer 2017-10-23 16:06:51,332 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-10-23 16:06:51,332 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkPerformance will be repeating every 60 seconds 2017-10-23 16:06:51,332 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 1 to run in 60 seconds 2017-10-23 16:06:51,332 | DEBUG | replica.py ( 313) | h | Node3:0 set watermarks as 0 300 2017-10-23 16:06:51,333 | DISPLAY | replicas.py ( 36) | grow | Node3 added replica Node3:0 to instance 0 (master) 2017-10-23 16:06:51,333 | DEBUG | replica.py ( 313) | h | Node3:1 set watermarks as 0 300 2017-10-23 16:06:51,333 | DISPLAY | replicas.py ( 36) | grow | Node3 added replica Node3:1 to instance 1 (backup) 2017-10-23 16:06:51,333 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkPerformance will be repeating every 10 seconds 2017-10-23 16:06:51,333 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 1 to run in 10 seconds 2017-10-23 16:06:51,333 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkNodeRequestSpike will be repeating every 60 seconds 2017-10-23 16:06:51,333 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 2 to run in 60 seconds 2017-10-23 16:06:51,334 | DEBUG | plugin_helper.py ( 24) | loadPlugins | Plugin loading started to load plugins from basedir: /home/sovrin/.sovrin 2017-10-23 16:06:51,334 | DEBUG | plugin_helper.py ( 68) | loadPlugins | Total plugins loaded from basedir /home/sovrin/.sovrin are : 0 2017-10-23 16:06:51,334 | DEBUG | node.py ( 340) | __init__ | total plugins loaded in node: 0 2017-10-23 16:06:51,733 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-10-23 16:06:51,766 | DEBUG | ledger.py ( 72) | recoverTree | Recovering tree from hash store of size 1 2017-10-23 16:06:51,766 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.000365355983376503 seconds 2017-10-23 16:06:51,813 | WARNING | upgrader.py ( 126) | check_upgrade_succeeded | Upgrade for node Node3 was not scheduled. Last event is scheduled:2017-10-17 11:30:00+00:00:1.1.37:None 2017-10-23 16:06:51,813 | INFO | node.py (2420) | initStateFromLedger | Node3 found state to be empty, recreating from ledger 2017-10-23 16:06:51,813 | DEBUG | motor.py ( 34) | set_status | Node3 changing status from stopped to starting 2017-10-23 16:06:51,813 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-10-23 16:06:51,813 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-10-23 16:06:51,813 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-10-23 16:06:51,813 | DEBUG | zstack.py ( 319) | start | Node3 starting with restricted as True and reSetupAuth as True 2017-10-23 16:06:51,814 | DEBUG | authenticator.py ( 31) | start | Starting ZAP at inproc://zeromq.zap.1 2017-10-23 16:06:51,814 | DEBUG | base.py ( 72) | allow | Allowing 0.0.0.0 2017-10-23 16:06:51,814 | DEBUG | base.py ( 112) | configure_curve | Configure curve: *[/home/sovrin/.sovrin/Node3/public_keys] 2017-10-23 16:06:51,823 | DEBUG | zstack.py ( 347) | open | Node3 will bind its listener at 9705 2017-10-23 16:06:51,823 | INFO | stacks.py ( 84) | start | CONNECTION: Node3 listening for other nodes at 0.0.0.0:9705 2017-10-23 16:06:51,823 | DEBUG | zstack.py ( 319) | start | Node3C starting with restricted as False and reSetupAuth as True 2017-10-23 16:06:51,823 | DEBUG | authenticator.py ( 31) | start | Starting ZAP at inproc://zeromq.zap.2 2017-10-23 16:06:51,823 | DEBUG | base.py ( 72) | allow | Allowing 0.0.0.0 2017-10-23 16:06:51,823 | DEBUG | base.py ( 112) | configure_curve | Configure curve: *[*] 2017-10-23 16:06:51,824 | DEBUG | zstack.py ( 347) | open | Node3C will bind its listener at 9706 2017-10-23 16:06:51,824 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action propose_view_change with id 3 to run in 60 seconds 2017-10-23 16:06:51,824 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 4 to run in 3 seconds 2017-10-23 16:06:51,824 | DEBUG | has_action_queue.py ( 79) | startRepeating | dump_json_file will be repeating every 60 seconds 2017-10-23 16:06:51,824 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 5 to run in 60 seconds 2017-10-23 16:06:51,824 | INFO | node.py ( 621) | start | Node3 first time running... 2017-10-23 16:06:51,834 | DEBUG | kit_zstack.py ( 97) | connectToMissing | CONNECTION: Node3 found the following missing connections: Node2, Node1, Node4 2017-10-23 16:06:51,835 | TRACE | remote.py ( 86) | connect | connecting socket 78 54743504 to remote Node2:HA(host='10.0.0.3', port=9703) 2017-10-23 16:06:51,835 | INFO | zstack.py ( 590) | connect | CONNECTION: Node3 looking for Node2 at 10.0.0.3:9703 2017-10-23 16:06:51,836 | DEBUG | zstack.py ( 645) | sendPingPong | Node3 pinged Node2 2017-10-23 16:06:51,836 | TRACE | remote.py ( 86) | connect | connecting socket 80 54847008 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-23 16:06:51,836 | INFO | zstack.py ( 590) | connect | CONNECTION: Node3 looking for Node1 at 10.0.0.2:9701 2017-10-23 16:06:51,836 | DEBUG | zstack.py ( 645) | sendPingPong | Node3 pinged Node1 2017-10-23 16:06:51,837 | TRACE | remote.py ( 86) | connect | connecting socket 81 54884288 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-23 16:06:51,837 | INFO | zstack.py ( 590) | connect | CONNECTION: Node3 looking for Node4 at 10.0.0.5:9707 2017-10-23 16:06:51,837 | DEBUG | zstack.py ( 645) | sendPingPong | Node3 pinged Node4 2017-10-23 16:06:51,837 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:06:51,853 | DEBUG | zstack.py ( 723) | transmit | Node3 transmitting message b'pi' to Node2 2017-10-23 16:06:51,854 | TRACE | batched.py ( 96) | flushOutBoxes | Node3 sending msg b'pi' to Node2 2017-10-23 16:06:51,854 | DEBUG | zstack.py ( 723) | transmit | Node3 transmitting message b'pi' to Node1 2017-10-23 16:06:51,855 | TRACE | batched.py ( 96) | flushOutBoxes | Node3 sending msg b'pi' to Node1 2017-10-23 16:06:51,855 | DEBUG | zstack.py ( 723) | transmit | Node3 transmitting message b'pi' to Node4 2017-10-23 16:06:51,855 | TRACE | batched.py ( 96) | flushOutBoxes | Node3 sending msg b'pi' to Node4 2017-10-23 16:06:51,876 | TRACE | zstack.py ( 479) | _receiveFromListener | Node3 got 2 messages through listener 2017-10-23 16:06:51,877 | DEBUG | zstack.py ( 663) | handlePingPong | Node3 got ping from Node2 2017-10-23 16:06:51,877 | DEBUG | zstack.py ( 645) | sendPingPong | Node3 ponged Node2 2017-10-23 16:06:51,877 | DEBUG | zstack.py ( 669) | handlePingPong | Node3 got pong from Node2 2017-10-23 16:06:51,877 | DEBUG | zstack.py ( 723) | transmit | Node3 transmitting message b'po' to Node2 2017-10-23 16:06:51,877 | TRACE | batched.py ( 96) | flushOutBoxes | Node3 sending msg b'po' to Node2 2017-10-23 16:06:51,878 | DEBUG | keep_in_touch.py ( 68) | conns | Node3's connections changed from set() to {'Node2'} 2017-10-23 16:06:51,878 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node3 now connected to Node2 2017-10-23 16:06:51,878 | DEBUG | node.py (2608) | send | Node3 sending message LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'txnSeqNo': 4} to 1 recipients: ['Node2'] 2017-10-23 16:06:51,879 | DEBUG | zstack.py ( 723) | transmit | Node3 transmitting message b'{"viewNo":null,"op":"LEDGER_STATUS","merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ledgerId":0,"txnSeqNo":4,"ppSeqNo":null}' to Node2 2017-10-23 16:06:51,880 | TRACE | batched.py ( 96) | flushOutBoxes | Node3 sending msg b'{"viewNo":null,"op":"LEDGER_STATUS","merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ledgerId":0,"txnSeqNo":4,"ppSeqNo":null}' to Node2 2017-10-23 16:06:51,902 | TRACE | zstack.py ( 479) | _receiveFromListener | Node3 got 1 messages through listener 2017-10-23 16:06:51,903 | DEBUG | node.py (1257) | validateNodeMsg | Node3 received node message from Node2: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'txnSeqNo': 4} 2017-10-23 16:06:51,903 | DEBUG | node.py (1219) | handleOneNodeMsg | Node3 msg validated ({'viewNo': None, 'op': 'LEDGER_STATUS', 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None}, 'Node2') 2017-10-23 16:06:51,903 | DEBUG | node.py (1286) | postToNodeInBox | Node3 appending to nodeInbox LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'txnSeqNo': 4} 2017-10-23 16:06:51,903 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'txnSeqNo': 4} from Node2 2017-10-23 16:06:51,903 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-10-23 16:06:51,903 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-10-23 16:06:52,030 | TRACE | zstack.py ( 479) | _receiveFromListener | Node3C got 2 messages through listener 2017-10-23 16:06:52,030 | DEBUG | zstack.py ( 663) | handlePingPong | Node3C got ping from b'xo5JUY.S$6PWbpmz5XzA processing config ledger for any POOL_CONFIGs 2017-10-23 16:06:52,480 | DEBUG | upgrader.py ( 208) | processLedger | DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya processing config ledger for any upgrades 2017-10-23 16:06:52,480 | INFO | upgrader.py ( 214) | processLedger | DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya found upgrade START txn OrderedDict([('action', 'start'), ('force', True), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade-1137'), ('reqId', 1508774754234674), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-10-17T11:35:00.000000+00:00'), ('4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', '2017-10-17T11:40:00.000000+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-10-17T11:25:00.000000+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-10-17T11:30:00.000000+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-10-17T11:20:00.000000+00:00')])), ('sha256', 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3'), ('signature', '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ'), ('timeout', 10), ('txnTime', 1508774754), ('type', '109'), ('version', '1.1.37'), ('seqNo', 1)]) 2017-10-23 16:06:52,480 | INFO | upgrader.py ( 292) | handleUpgradeTxn | Node 'Node3' handles upgrade txn OrderedDict([('action', 'start'), ('force', True), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade-1137'), ('reqId', 1508774754234674), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-10-17T11:35:00.000000+00:00'), ('4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', '2017-10-17T11:40:00.000000+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-10-17T11:25:00.000000+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-10-17T11:30:00.000000+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-10-17T11:20:00.000000+00:00')])), ('sha256', 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3'), ('signature', '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ'), ('timeout', 10), ('txnTime', 1508774754), ('type', '109'), ('version', '1.1.37'), ('seqNo', 1)]) 2017-10-23 16:06:52,480 | DEBUG | node.py (2608) | send | Node3 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node1'] 2017-10-23 16:06:52,480 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node3 asking Node1 for ledger status of ledger 1 2017-10-23 16:06:52,480 | DEBUG | node.py (2608) | send | Node3 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node2'] 2017-10-23 16:06:52,480 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node3 asking Node2 for ledger status of ledger 1 2017-10-23 16:06:52,481 | DEBUG | node.py (2608) | send | Node3 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to all recipients: ['Node2', 'Node1', 'Node4'] 2017-10-23 16:06:52,481 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node3 asking Node3 for ledger status of ledger 1 2017-10-23 16:06:52,481 | DEBUG | node.py (2608) | send | Node3 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-10-23 16:06:52,481 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node3 asking Node4 for ledger status of ledger 1 2017-10-23 16:06:52,481 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node3 going to process 0 stashed ledger statuses for ledger 1 2017-10-23 16:06:52,481 | INFO | upgrader.py ( 150) | should_notify_about_upgrade_result | Node's 'Node3' last upgrade txn is None 2017-10-23 16:06:52,481 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 2, caught up 0 in total 2017-10-23 16:06:52,481 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ledgerId': 2, 'txnSeqNo': 1} from Node2 2017-10-23 16:06:52,481 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 1 with 1 2017-10-23 16:06:52,482 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 1 with 1 2017-10-23 16:06:52,482 | DEBUG | batched.py ( 100) | flushOutBoxes | Node3 batching 4 msgs to Node2 into one transmission 2017-10-23 16:06:52,482 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}']) 2017-10-23 16:06:52,482 | TRACE | batched.py ( 110) | flushOutBoxes | Node3 sending payload to Node2: b'{"op":"BATCH","signature":null,"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":7,\\"viewNo\\":null}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":7,\\"viewNo\\":null}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}"]}' 2017-10-23 16:06:52,490 | DEBUG | zstack.py ( 723) | transmit | Node3 transmitting message b'{"op":"BATCH","signature":null,"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":7,\\"viewNo\\":null}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":7,\\"viewNo\\":null}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}"]}' to Node2 2017-10-23 16:06:52,490 | DEBUG | batched.py ( 100) | flushOutBoxes | Node3 batching 2 msgs to Node1 into one transmission 2017-10-23 16:06:52,490 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}']) 2017-10-23 16:06:52,491 | TRACE | batched.py ( 110) | flushOutBoxes | Node3 sending payload to Node1: b'{"op":"BATCH","signature":null,"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}"]}' 2017-10-23 16:06:52,491 | DEBUG | zstack.py ( 723) | transmit | Node3 transmitting message b'{"op":"BATCH","signature":null,"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}"]}' to Node1 2017-10-23 16:06:52,491 | DEBUG | batched.py ( 100) | flushOutBoxes | Node3 batching 2 msgs to Node4 into one transmission 2017-10-23 16:06:52,491 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}']) 2017-10-23 16:06:52,491 | TRACE | batched.py ( 110) | flushOutBoxes | Node3 sending payload to Node4: b'{"op":"BATCH","signature":null,"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}"]}' 2017-10-23 16:06:52,491 | DEBUG | zstack.py ( 723) | transmit | Node3 transmitting message b'{"op":"BATCH","signature":null,"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}"]}' to Node4 2017-10-23 16:06:52,491 | DEBUG | zstack.py ( 728) | transmit | Remote Node4 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-23 16:06:52,503 | TRACE | zstack.py ( 479) | _receiveFromListener | Node3 got 2 messages through listener 2017-10-23 16:06:52,504 | DEBUG | node.py (1257) | validateNodeMsg | Node3 received node message from Node1: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"viewNo":0,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1","op":"VIEW_CHANGE_DONE"}', '{"viewNo":0,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1","op":"VIEW_CHANGE_DONE"}']} 2017-10-23 16:06:52,504 | DEBUG | node.py (1219) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"viewNo":0,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1","op":"VIEW_CHANGE_DONE"}', '{"viewNo":0,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1","op":"VIEW_CHANGE_DONE"}']}, 'Node1') 2017-10-23 16:06:52,504 | DEBUG | node.py (1272) | unpackNodeMsg | Node3 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"viewNo":0,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1","op":"VIEW_CHANGE_DONE"}', '{"viewNo":0,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1","op":"VIEW_CHANGE_DONE"}']} 2017-10-23 16:06:52,504 | DEBUG | node.py (1257) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 16:06:52,504 | DEBUG | node.py (1219) | handleOneNodeMsg | Node3 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}}, 'Node1') 2017-10-23 16:06:52,505 | DEBUG | node.py (1286) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 16:06:52,505 | DEBUG | node.py (1257) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 16:06:52,505 | DEBUG | node.py (1219) | handleOneNodeMsg | Node3 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}}, 'Node1') 2017-10-23 16:06:52,505 | DEBUG | node.py (1286) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 16:06:52,505 | DEBUG | node.py (1257) | validateNodeMsg | Node3 received node message from Node1: VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'} 2017-10-23 16:06:52,505 | DEBUG | node.py (1219) | handleOneNodeMsg | Node3 msg validated ({'viewNo': 0, 'op': 'VIEW_CHANGE_DONE', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}, 'Node1') 2017-10-23 16:06:52,506 | DEBUG | node.py (1286) | postToNodeInBox | Node3 appending to nodeInbox VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'} 2017-10-23 16:06:52,506 | DEBUG | node.py (1257) | validateNodeMsg | Node3 received node message from Node1: VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'} 2017-10-23 16:06:52,506 | DEBUG | node.py (1219) | handleOneNodeMsg | Node3 msg validated ({'viewNo': 0, 'op': 'VIEW_CHANGE_DONE', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}, 'Node1') 2017-10-23 16:06:52,506 | DEBUG | node.py (1286) | postToNodeInBox | Node3 appending to nodeInbox VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'} 2017-10-23 16:06:52,506 | DEBUG | node.py (1257) | validateNodeMsg | Node3 received node message from Node1: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}']} 2017-10-23 16:06:52,506 | DEBUG | node.py (1219) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}']}, 'Node1') 2017-10-23 16:06:52,506 | DEBUG | node.py (1272) | unpackNodeMsg | Node3 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}']} 2017-10-23 16:06:52,507 | DEBUG | node.py (1257) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': None, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'txnSeqNo': 7, 'ledgerId': 1, 'ppSeqNo': None}} 2017-10-23 16:06:52,507 | DEBUG | node.py (1219) | handleOneNodeMsg | Node3 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}, 'msg': {'viewNo': None, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'txnSeqNo': 7, 'ledgerId': 1, 'ppSeqNo': None}}, 'Node1') 2017-10-23 16:06:52,507 | DEBUG | node.py (1286) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': None, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'txnSeqNo': 7, 'ledgerId': 1, 'ppSeqNo': None}} 2017-10-23 16:06:52,507 | DEBUG | node.py (1257) | validateNodeMsg | Node3 received node message from Node1: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': None, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'txnSeqNo': 7, 'ledgerId': 1, 'ppSeqNo': None}} 2017-10-23 16:06:52,507 | DEBUG | node.py (1219) | handleOneNodeMsg | Node3 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}, 'msg': {'viewNo': None, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'txnSeqNo': 7, 'ledgerId': 1, 'ppSeqNo': None}}, 'Node1') 2017-10-23 16:06:52,507 | DEBUG | node.py (1286) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': None, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'txnSeqNo': 7, 'ledgerId': 1, 'ppSeqNo': None}} 2017-10-23 16:06:52,508 | DEBUG | node.py (2608) | send | Node3 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ledgerId': 1, 'txnSeqNo': 7}} to 1 recipients: ['Node1'] 2017-10-23 16:06:52,508 | DEBUG | node.py (2608) | send | Node3 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ledgerId': 1, 'txnSeqNo': 7}} to 1 recipients: ['Node1'] 2017-10-23 16:06:52,508 | DEBUG | node.py (1205) | sendToElector | Node3 sending message to elector: (VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}, 'Node1') 2017-10-23 16:06:52,508 | DEBUG | node.py (1205) | sendToElector | Node3 sending message to elector: (VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}, 'Node1') 2017-10-23 16:06:52,509 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ledgerId': 1, 'txnSeqNo': 7} from Node1 2017-10-23 16:06:52,509 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 7 with 7 2017-10-23 16:06:52,509 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 7 with 7 2017-10-23 16:06:52,509 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ledgerId': 1, 'txnSeqNo': 7} from Node1 2017-10-23 16:06:52,512 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 7 with 7 2017-10-23 16:06:52,512 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 7 with 7 2017-10-23 16:06:52,513 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node3's primary selector started processing of ViewChangeDone msg from Node1 : VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'} 2017-10-23 16:06:52,513 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node3 needs 1 ViewChangeDone messages 2017-10-23 16:06:52,513 | DEBUG | primary_selector.py ( 266) | _startSelection | Node3 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-10-23 16:06:52,513 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node3's primary selector started processing of ViewChangeDone msg from Node1 : VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'} 2017-10-23 16:06:52,513 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node3 needs 1 ViewChangeDone messages 2017-10-23 16:06:52,513 | DEBUG | primary_selector.py ( 266) | _startSelection | Node3 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-10-23 16:06:52,513 | DEBUG | batched.py ( 100) | flushOutBoxes | Node3 batching 2 msgs to Node1 into one transmission 2017-10-23 16:06:52,513 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null}}']) 2017-10-23 16:06:52,514 | TRACE | batched.py ( 110) | flushOutBoxes | Node3 sending payload to Node1: b'{"op":"BATCH","signature":null,"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":7,\\"viewNo\\":null}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":7,\\"viewNo\\":null}}"]}' 2017-10-23 16:06:52,514 | DEBUG | zstack.py ( 723) | transmit | Node3 transmitting message b'{"op":"BATCH","signature":null,"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":7,\\"viewNo\\":null}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":7,\\"viewNo\\":null}}"]}' to Node1 2017-10-23 16:06:52,535 | TRACE | zstack.py ( 479) | _receiveFromListener | Node3 got 1 messages through listener 2017-10-23 16:06:52,535 | DEBUG | node.py (1257) | validateNodeMsg | Node3 received node message from Node2: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS"}', '{"name":"Node1","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"op":"VIEW_CHANGE_DONE","viewNo":0}']} 2017-10-23 16:06:52,535 | DEBUG | node.py (1219) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS"}', '{"name":"Node1","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"op":"VIEW_CHANGE_DONE","viewNo":0}']}, 'Node2') 2017-10-23 16:06:52,535 | DEBUG | node.py (1272) | unpackNodeMsg | Node3 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS"}', '{"name":"Node1","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"op":"VIEW_CHANGE_DONE","viewNo":0}']} 2017-10-23 16:06:52,536 | DEBUG | node.py (1257) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': None, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'txnSeqNo': 7, 'ledgerId': 1, 'ppSeqNo': None}} 2017-10-23 16:06:52,536 | DEBUG | node.py (1219) | handleOneNodeMsg | Node3 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}, 'msg': {'viewNo': None, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'txnSeqNo': 7, 'ledgerId': 1, 'ppSeqNo': None}}, 'Node2') 2017-10-23 16:06:52,536 | DEBUG | node.py (1286) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': None, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'txnSeqNo': 7, 'ledgerId': 1, 'ppSeqNo': None}} 2017-10-23 16:06:52,536 | DEBUG | node.py (1257) | validateNodeMsg | Node3 received node message from Node2: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': None, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'txnSeqNo': 7, 'ledgerId': 1, 'ppSeqNo': None}} 2017-10-23 16:06:52,536 | DEBUG | node.py (1219) | handleOneNodeMsg | Node3 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}, 'msg': {'viewNo': None, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'txnSeqNo': 7, 'ledgerId': 1, 'ppSeqNo': None}}, 'Node2') 2017-10-23 16:06:52,536 | DEBUG | node.py (1286) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': None, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'txnSeqNo': 7, 'ledgerId': 1, 'ppSeqNo': None}} 2017-10-23 16:06:52,537 | DEBUG | node.py (1257) | validateNodeMsg | Node3 received node message from Node2: VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'} 2017-10-23 16:06:52,537 | DEBUG | node.py (1219) | handleOneNodeMsg | Node3 msg validated ({'viewNo': 0, 'op': 'VIEW_CHANGE_DONE', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}, 'Node2') 2017-10-23 16:06:52,537 | DEBUG | node.py (1286) | postToNodeInBox | Node3 appending to nodeInbox VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'} 2017-10-23 16:06:52,537 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ledgerId': 1, 'txnSeqNo': 7} from Node2 2017-10-23 16:06:52,537 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 7 with 7 2017-10-23 16:06:52,537 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 7 with 7 2017-10-23 16:06:52,537 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found out from {'Node2', 'Node1'} that its ledger of type 1 is latest 2017-10-23 16:06:52,538 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node3 found from ledger status LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ledgerId': 1, 'txnSeqNo': 7} that it does not need catchup 2017-10-23 16:06:52,538 | DEBUG | node.py (1479) | preLedgerCatchUp | Node3 going to process any ordered requests before starting catchup. 2017-10-23 16:06:52,538 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node3:0 going to remove 0 Ordered messages from outbox 2017-10-23 16:06:52,538 | DEBUG | node.py (1888) | force_process_ordered | Node3 processed 0 Ordered batches for instance 0 before starting catch up 2017-10-23 16:06:52,538 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node3:1 going to remove 0 Ordered messages from outbox 2017-10-23 16:06:52,538 | DEBUG | node.py (1888) | force_process_ordered | Node3 processed 0 Ordered batches for instance 1 before starting catch up 2017-10-23 16:06:52,538 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node3 processed 0 stashed ordered requests 2017-10-23 16:06:52,538 | DEBUG | monitor.py ( 192) | reset | Node3's Monitor being reset 2017-10-23 16:06:52,539 | INFO | node.py (1489) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 1 2017-10-23 16:06:52,539 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 7 with 7 2017-10-23 16:06:52,539 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 1, caught up 0 in total 2017-10-23 16:06:52,539 | DEBUG | node.py (1609) | num_txns_caught_up_in_last_catchup | Node3 caught up to 0 txns in the last catchup 2017-10-23 16:06:52,539 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node3 processed 0 stashed ordered requests 2017-10-23 16:06:52,539 | DEBUG | monitor.py ( 192) | reset | Node3's Monitor being reset 2017-10-23 16:06:52,539 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node3 needs 1 ViewChangeDone messages 2017-10-23 16:06:52,539 | DEBUG | node.py (1568) | caught_up_for_current_view | Node3 does not have view change quorum for view 0 2017-10-23 16:06:52,539 | DEBUG | node.py (1552) | is_catchup_needed | Node3 is not caught up for the current view 0 2017-10-23 16:06:52,540 | DEBUG | node.py (1609) | num_txns_caught_up_in_last_catchup | Node3 caught up to 0 txns in the last catchup 2017-10-23 16:06:52,540 | DEBUG | node.py (1557) | is_catchup_needed | Node3 ordered till last prepared certificate 2017-10-23 16:06:52,540 | INFO | node.py (1537) | allLedgersCaughtUp | CATCH-UP: Node3 does not need any more catchups 2017-10-23 16:06:52,540 | DEBUG | primary_decider.py ( 134) | send | Node3's elector sending VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'), (2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF')], 'name': 'Node1'} 2017-10-23 16:06:52,540 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node3 got view change quorum (2 >= 2) 2017-10-23 16:06:52,540 | DEBUG | primary_selector.py ( 215) | has_view_change_from_primary | Node3 received ViewChangeDone from primary Node1 2017-10-23 16:06:52,541 | DEBUG | primary_selector.py ( 238) | has_sufficient_same_view_change_done_messages | Node3 found acceptable primary Node1 and ledger info ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'), (2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF')) 2017-10-23 16:06:52,541 | DEBUG | primary_selector.py ( 281) | _startSelection | Node3 starting selection 2017-10-23 16:06:52,541 | DISPLAY | primary_selector.py ( 291) | _startSelection | PRIMARY SELECTION: Node3:0 selected primary Node1:0 for instance 0 (view 0) 2017-10-23 16:06:52,541 | INFO | node.py ( 510) | start_participating | Node3 started participating 2017-10-23 16:06:52,541 | DEBUG | replica.py ( 408) | primaryName | Node3:0 setting primaryName for view no 0 to: Node1:0 2017-10-23 16:06:52,541 | DEBUG | replica.py (1694) | _gc | Node3:0 cleaning up till (0, 0) 2017-10-23 16:06:52,541 | DEBUG | replica.py (1709) | _gc | Node3:0 found 0 3-phase keys to clean 2017-10-23 16:06:52,541 | DEBUG | replica.py (1711) | _gc | Node3:0 found 0 request keys to clean 2017-10-23 16:06:52,542 | DEBUG | replica.py ( 313) | h | Node3:0 set watermarks as 0 300 2017-10-23 16:06:52,542 | DISPLAY | primary_selector.py ( 315) | _startSelection | VIEW CHANGE: Node3:0 declares view change 0 as completed for instance 0, new primary is Node1:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'), (2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF')] 2017-10-23 16:06:52,542 | DISPLAY | primary_selector.py ( 291) | _startSelection | PRIMARY SELECTION: Node3:1 selected primary Node2:1 for instance 1 (view 0) 2017-10-23 16:06:52,542 | DEBUG | replica.py ( 408) | primaryName | Node3:1 setting primaryName for view no 0 to: Node2:1 2017-10-23 16:06:52,542 | DEBUG | replica.py (1694) | _gc | Node3:1 cleaning up till (0, 0) 2017-10-23 16:06:52,545 | DEBUG | replica.py (1709) | _gc | Node3:1 found 0 3-phase keys to clean 2017-10-23 16:06:52,545 | DEBUG | replica.py (1711) | _gc | Node3:1 found 0 request keys to clean 2017-10-23 16:06:52,545 | DEBUG | replica.py ( 313) | h | Node3:1 set watermarks as 0 300 2017-10-23 16:06:52,546 | DEBUG | replica.py ( 514) | _setup_for_non_master | Node3:1 Setting last ordered for non-master as (0, 0) 2017-10-23 16:06:52,546 | DEBUG | replica.py ( 323) | last_ordered_3pc | Node3:1 set last ordered as (0, 0) 2017-10-23 16:06:52,550 | DISPLAY | primary_selector.py ( 315) | _startSelection | VIEW CHANGE: Node3:1 declares view change 0 as completed for instance 1, new primary is Node2:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'), (2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF')] 2017-10-23 16:06:52,551 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ledgerId': 1, 'txnSeqNo': 7} from Node2 2017-10-23 16:06:52,551 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 7 with 7 2017-10-23 16:06:52,551 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 7 with 7 2017-10-23 16:06:52,551 | DEBUG | node.py (1205) | sendToElector | Node3 sending message to elector: (VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}, 'Node2') 2017-10-23 16:06:52,552 | DEBUG | node.py (2608) | send | Node3 sending message VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'), (2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF')], 'name': 'Node1'} to all recipients: ['Node2', 'Node1', 'Node4'] 2017-10-23 16:06:52,552 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node3's primary selector started processing of ViewChangeDone msg from Node2 : VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'} 2017-10-23 16:06:52,552 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'} because it already decided primary which is Node1:0 2017-10-23 16:06:52,552 | DEBUG | zstack.py ( 723) | transmit | Node3 transmitting message b'{"viewNo":0,"op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1"}' to Node2 2017-10-23 16:06:52,552 | TRACE | batched.py ( 96) | flushOutBoxes | Node3 sending msg b'{"viewNo":0,"op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1"}' to Node2 2017-10-23 16:06:52,552 | DEBUG | zstack.py ( 723) | transmit | Node3 transmitting message b'{"viewNo":0,"op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1"}' to Node1 2017-10-23 16:06:52,553 | TRACE | batched.py ( 96) | flushOutBoxes | Node3 sending msg b'{"viewNo":0,"op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1"}' to Node1 2017-10-23 16:06:52,553 | DEBUG | zstack.py ( 723) | transmit | Node3 transmitting message b'{"viewNo":0,"op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1"}' to Node4 2017-10-23 16:06:52,553 | DEBUG | zstack.py ( 728) | transmit | Remote Node4 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-23 16:06:52,553 | TRACE | batched.py ( 96) | flushOutBoxes | Node3 sending msg b'{"viewNo":0,"op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1"}' to Node4 2017-10-23 16:06:53,766 | TRACE | zstack.py ( 479) | _receiveFromListener | Node3 got 2 messages through listener 2017-10-23 16:06:53,767 | DEBUG | zstack.py ( 663) | handlePingPong | Node3 got ping from Node4 2017-10-23 16:06:53,767 | DEBUG | zstack.py ( 645) | sendPingPong | Node3 ponged Node4 2017-10-23 16:06:53,767 | DEBUG | zstack.py ( 669) | handlePingPong | Node3 got pong from Node4 2017-10-23 16:06:53,767 | DEBUG | node.py (1257) | validateNodeMsg | Node3 received node message from Node4: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null}}']} 2017-10-23 16:06:53,767 | DEBUG | node.py (1219) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null}}']}, 'Node4') 2017-10-23 16:06:53,767 | DEBUG | node.py (1272) | unpackNodeMsg | Node3 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null}}', '{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null}}']} 2017-10-23 16:06:53,767 | DEBUG | node.py (1257) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'viewNo': None, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'txnSeqNo': 1, 'ledgerId': 2, 'ppSeqNo': None}} 2017-10-23 16:06:53,768 | DEBUG | node.py (1219) | handleOneNodeMsg | Node3 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}, 'msg': {'viewNo': None, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'txnSeqNo': 1, 'ledgerId': 2, 'ppSeqNo': None}}, 'Node4') 2017-10-23 16:06:53,768 | DEBUG | node.py (1286) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'viewNo': None, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'txnSeqNo': 1, 'ledgerId': 2, 'ppSeqNo': None}} 2017-10-23 16:06:53,769 | DEBUG | node.py (1257) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'viewNo': None, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'txnSeqNo': 1, 'ledgerId': 2, 'ppSeqNo': None}} 2017-10-23 16:06:53,769 | DEBUG | node.py (1219) | handleOneNodeMsg | Node3 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}, 'msg': {'viewNo': None, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'txnSeqNo': 1, 'ledgerId': 2, 'ppSeqNo': None}}, 'Node4') 2017-10-23 16:06:53,770 | DEBUG | node.py (1286) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': {'viewNo': None, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'txnSeqNo': 1, 'ledgerId': 2, 'ppSeqNo': None}} 2017-10-23 16:06:53,770 | DEBUG | node.py (1257) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': None, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'txnSeqNo': 7, 'ledgerId': 1, 'ppSeqNo': None}} 2017-10-23 16:06:53,770 | DEBUG | node.py (1219) | handleOneNodeMsg | Node3 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}, 'msg': {'viewNo': None, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'txnSeqNo': 7, 'ledgerId': 1, 'ppSeqNo': None}}, 'Node4') 2017-10-23 16:06:53,770 | DEBUG | node.py (1286) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': None, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'txnSeqNo': 7, 'ledgerId': 1, 'ppSeqNo': None}} 2017-10-23 16:06:53,770 | DEBUG | node.py (1257) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': None, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'txnSeqNo': 7, 'ledgerId': 1, 'ppSeqNo': None}} 2017-10-23 16:06:53,770 | DEBUG | node.py (1219) | handleOneNodeMsg | Node3 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}, 'msg': {'viewNo': None, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'txnSeqNo': 7, 'ledgerId': 1, 'ppSeqNo': None}}, 'Node4') 2017-10-23 16:06:53,770 | DEBUG | node.py (1286) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': {'viewNo': None, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'txnSeqNo': 7, 'ledgerId': 1, 'ppSeqNo': None}} 2017-10-23 16:06:53,771 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ledgerId': 2, 'txnSeqNo': 1} from Node4 2017-10-23 16:06:53,771 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 1 with 1 2017-10-23 16:06:53,771 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 1 with 1 2017-10-23 16:06:53,771 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found out from {'Node2', 'Node4'} that its ledger of type 2 is latest 2017-10-23 16:06:53,771 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ledgerId': 2, 'txnSeqNo': 1} from Node4 2017-10-23 16:06:53,771 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 1 with 1 2017-10-23 16:06:53,771 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 2 of size 1 with 1 2017-10-23 16:06:53,772 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found out from {'Node2', 'Node4'} that its ledger of type 2 is latest 2017-10-23 16:06:53,772 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ledgerId': 1, 'txnSeqNo': 7} from Node4 2017-10-23 16:06:53,772 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 7 with 7 2017-10-23 16:06:53,772 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 7 with 7 2017-10-23 16:06:53,772 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found out from {'Node2', 'Node4'} that its ledger of type 1 is latest 2017-10-23 16:06:53,772 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ledgerId': 1, 'txnSeqNo': 7} from Node4 2017-10-23 16:06:53,772 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 7 with 7 2017-10-23 16:06:53,772 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 1 of size 7 with 7 2017-10-23 16:06:53,773 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node3 found out from {'Node2', 'Node4'} that its ledger of type 1 is latest 2017-10-23 16:06:53,773 | DEBUG | zstack.py ( 723) | transmit | Node3 transmitting message b'po' to Node4 2017-10-23 16:06:53,774 | TRACE | batched.py ( 96) | flushOutBoxes | Node3 sending msg b'po' to Node4 2017-10-23 16:06:53,775 | DEBUG | keep_in_touch.py ( 68) | conns | Node3's connections changed from {'Node2', 'Node1'} to {'Node2', 'Node1', 'Node4'} 2017-10-23 16:06:53,775 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node3 now connected to Node4 2017-10-23 16:06:53,775 | DEBUG | motor.py ( 34) | set_status | Node3 changing status from started_hungry to started 2017-10-23 16:06:53,775 | DEBUG | node.py ( 941) | checkInstances | Node3 choosing to start election on the basis of count 4 and nodes {'Node2', 'Node1', 'Node4'} 2017-10-23 16:06:53,775 | DEBUG | node.py ( 902) | send_current_state_to_lagging_node | Node3 sending current state CURRENT_STATE{'primary': [VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'), (2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF')), 'name': 'Node1'}], 'viewNo': 0} to lagged node Node4 2017-10-23 16:06:53,775 | DEBUG | node.py (2608) | send | Node3 sending message CURRENT_STATE{'primary': [VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'), (2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF')), 'name': 'Node1'}], 'viewNo': 0} to 1 recipients: ['Node4'] 2017-10-23 16:06:53,776 | DEBUG | node.py (2608) | send | Node3 sending message LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'txnSeqNo': 4} to 1 recipients: ['Node4'] 2017-10-23 16:06:53,776 | DEBUG | node.py (2608) | send | Node3 sending message LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ledgerId': 1, 'txnSeqNo': 7} to 1 recipients: ['Node4'] 2017-10-23 16:06:53,776 | DEBUG | node.py (2608) | send | Node3 sending message LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ledgerId': 2, 'txnSeqNo': 1} to 1 recipients: ['Node4'] 2017-10-23 16:06:53,777 | DEBUG | batched.py ( 100) | flushOutBoxes | Node3 batching 4 msgs to Node4 into one transmission 2017-10-23 16:06:53,777 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1","viewNo":0}],"viewNo":0,"op":"CURRENT_STATE"}', b'{"viewNo":null,"op":"LEDGER_STATUS","merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ledgerId":0,"txnSeqNo":4,"ppSeqNo":null}', b'{"viewNo":null,"op":"LEDGER_STATUS","merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ledgerId":1,"txnSeqNo":7,"ppSeqNo":null}', b'{"viewNo":null,"op":"LEDGER_STATUS","merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ledgerId":2,"txnSeqNo":1,"ppSeqNo":null}']) 2017-10-23 16:06:53,777 | TRACE | batched.py ( 110) | flushOutBoxes | Node3 sending payload to Node4: b'{"op":"BATCH","signature":null,"messages":["{\\"primary\\":[{\\"ledgerInfo\\":[[0,4,\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\"],[1,7,\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\"],[2,1,\\"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF\\"]],\\"name\\":\\"Node1\\",\\"viewNo\\":0}],\\"viewNo\\":0,\\"op\\":\\"CURRENT_STATE\\"}","{\\"viewNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ledgerId\\":0,\\"txnSeqNo\\":4,\\"ppSeqNo\\":null}","{\\"viewNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\",\\"ledgerId\\":1,\\"txnSeqNo\\":7,\\"ppSeqNo\\":null}","{\\"viewNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF\\",\\"ledgerId\\":2,\\"txnSeqNo\\":1,\\"ppSeqNo\\":null}"]}' 2017-10-23 16:06:53,777 | DEBUG | zstack.py ( 723) | transmit | Node3 transmitting message b'{"op":"BATCH","signature":null,"messages":["{\\"primary\\":[{\\"ledgerInfo\\":[[0,4,\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\"],[1,7,\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\"],[2,1,\\"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF\\"]],\\"name\\":\\"Node1\\",\\"viewNo\\":0}],\\"viewNo\\":0,\\"op\\":\\"CURRENT_STATE\\"}","{\\"viewNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ledgerId\\":0,\\"txnSeqNo\\":4,\\"ppSeqNo\\":null}","{\\"viewNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\",\\"ledgerId\\":1,\\"txnSeqNo\\":7,\\"ppSeqNo\\":null}","{\\"viewNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF\\",\\"ledgerId\\":2,\\"txnSeqNo\\":1,\\"ppSeqNo\\":null}"]}' to Node4 2017-10-23 16:06:53,814 | TRACE | zstack.py ( 479) | _receiveFromListener | Node3 got 1 messages through listener 2017-10-23 16:06:53,815 | DEBUG | node.py (1257) | validateNodeMsg | Node3 received node message from Node4: BATCH{'signature': None, 'messages': ['{"viewNo":0,"primary":[],"op":"CURRENT_STATE"}', '{"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","txnSeqNo":4,"viewNo":null,"op":"LEDGER_STATUS","ledgerId":0,"ppSeqNo":null}']} 2017-10-23 16:06:53,815 | DEBUG | node.py (1219) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"viewNo":0,"primary":[],"op":"CURRENT_STATE"}', '{"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","txnSeqNo":4,"viewNo":null,"op":"LEDGER_STATUS","ledgerId":0,"ppSeqNo":null}']}, 'Node4') 2017-10-23 16:06:53,815 | DEBUG | node.py (1272) | unpackNodeMsg | Node3 processing a batch BATCH{'signature': None, 'messages': ['{"viewNo":0,"primary":[],"op":"CURRENT_STATE"}', '{"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","txnSeqNo":4,"viewNo":null,"op":"LEDGER_STATUS","ledgerId":0,"ppSeqNo":null}']} 2017-10-23 16:06:53,815 | DEBUG | node.py (1257) | validateNodeMsg | Node3 received node message from Node4: CURRENT_STATE{'primary': [], 'viewNo': 0} 2017-10-23 16:06:53,815 | DEBUG | node.py (1219) | handleOneNodeMsg | Node3 msg validated ({'primary': [], 'viewNo': 0, 'op': 'CURRENT_STATE'}, 'Node4') 2017-10-23 16:06:53,815 | DEBUG | node.py (1286) | postToNodeInBox | Node3 appending to nodeInbox CURRENT_STATE{'primary': [], 'viewNo': 0} 2017-10-23 16:06:53,815 | DEBUG | node.py (1257) | validateNodeMsg | Node3 received node message from Node4: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'txnSeqNo': 4} 2017-10-23 16:06:53,815 | DEBUG | node.py (1219) | handleOneNodeMsg | Node3 msg validated ({'viewNo': None, 'op': 'LEDGER_STATUS', 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'txnSeqNo': 4, 'ppSeqNo': None}, 'Node4') 2017-10-23 16:06:53,816 | DEBUG | node.py (1286) | postToNodeInBox | Node3 appending to nodeInbox LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'txnSeqNo': 4} 2017-10-23 16:06:53,816 | DEBUG | node.py ( 907) | process_current_state_message | Node3 processing current state CURRENT_STATE{'primary': [], 'viewNo': 0} from Node4 2017-10-23 16:06:53,816 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node3 received ledger status: LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'txnSeqNo': 4} from Node4 2017-10-23 16:06:53,816 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-10-23 16:06:53,816 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node3 comparing its ledger 0 of size 4 with 4 2017-10-23 16:06:53,852 | TRACE | zstack.py ( 479) | _receiveFromListener | Node3 got 1 messages through listener 2017-10-23 16:06:53,853 | DEBUG | node.py (1257) | validateNodeMsg | Node3 received node message from Node4: BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"viewNo":0,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1","op":"VIEW_CHANGE_DONE"}']} 2017-10-23 16:06:53,853 | DEBUG | node.py (1219) | handleOneNodeMsg | Node3 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"viewNo":0,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1","op":"VIEW_CHANGE_DONE"}']}, 'Node4') 2017-10-23 16:06:53,853 | DEBUG | node.py (1272) | unpackNodeMsg | Node3 processing a batch BATCH{'signature': None, 'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"viewNo":0,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1","op":"VIEW_CHANGE_DONE"}']} 2017-10-23 16:06:53,853 | DEBUG | node.py (1257) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-23 16:06:53,853 | DEBUG | node.py (1219) | handleOneNodeMsg | Node3 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}}, 'Node4') 2017-10-23 16:06:53,853 | DEBUG | node.py (1286) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-23 16:06:53,854 | DEBUG | node.py (1257) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-23 16:06:53,854 | DEBUG | node.py (1219) | handleOneNodeMsg | Node3 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}}, 'Node4') 2017-10-23 16:06:53,854 | DEBUG | node.py (1286) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-23 16:06:53,854 | DEBUG | node.py (1257) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 16:06:53,854 | DEBUG | node.py (1219) | handleOneNodeMsg | Node3 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}}, 'Node4') 2017-10-23 16:06:53,854 | DEBUG | node.py (1286) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 16:06:53,854 | DEBUG | node.py (1257) | validateNodeMsg | Node3 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 16:06:53,854 | DEBUG | node.py (1219) | handleOneNodeMsg | Node3 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}}, 'Node4') 2017-10-23 16:06:53,855 | DEBUG | node.py (1286) | postToNodeInBox | Node3 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-23 16:06:53,855 | DEBUG | node.py (1257) | validateNodeMsg | Node3 received node message from Node4: VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'} 2017-10-23 16:06:53,855 | DEBUG | node.py (1219) | handleOneNodeMsg | Node3 msg validated ({'viewNo': 0, 'op': 'VIEW_CHANGE_DONE', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}, 'Node4') 2017-10-23 16:06:53,855 | DEBUG | node.py (1286) | postToNodeInBox | Node3 appending to nodeInbox VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'} 2017-10-23 16:06:53,855 | DEBUG | node.py (2608) | send | Node3 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ledgerId': 2, 'txnSeqNo': 1}} to 1 recipients: ['Node4'] 2017-10-23 16:06:53,856 | DEBUG | node.py (2608) | send | Node3 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}, 'msg': LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ledgerId': 2, 'txnSeqNo': 1}} to 1 recipients: ['Node4'] 2017-10-23 16:06:53,856 | DEBUG | node.py (2608) | send | Node3 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ledgerId': 1, 'txnSeqNo': 7}} to 1 recipients: ['Node4'] 2017-10-23 16:06:53,856 | DEBUG | node.py (2608) | send | Node3 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}, 'msg': LEDGER_STATUS{'viewNo': None, 'ppSeqNo': None, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ledgerId': 1, 'txnSeqNo': 7}} to 1 recipients: ['Node4'] 2017-10-23 16:06:53,858 | DEBUG | node.py (1205) | sendToElector | Node3 sending message to elector: (VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}, 'Node4') 2017-10-23 16:06:53,858 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node3's primary selector started processing of ViewChangeDone msg from Node4 : VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'} 2017-10-23 16:06:53,858 | DEBUG | message_processor.py ( 29) | discard | Node3 discarding message VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'} because it already decided primary which is Node1:0 2017-10-23 16:06:53,858 | DEBUG | batched.py ( 100) | flushOutBoxes | Node3 batching 4 msgs to Node4 into one transmission 2017-10-23 16:06:53,858 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null}}']) 2017-10-23 16:06:53,858 | TRACE | batched.py ( 110) | flushOutBoxes | Node3 sending payload to Node4: b'{"op":"BATCH","signature":null,"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":1,\\"viewNo\\":null}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":1,\\"viewNo\\":null}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":7,\\"viewNo\\":null}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":7,\\"viewNo\\":null}}"]}' 2017-10-23 16:06:53,859 | DEBUG | zstack.py ( 723) | transmit | Node3 transmitting message b'{"op":"BATCH","signature":null,"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":1,\\"viewNo\\":null}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":1,\\"viewNo\\":null}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":7,\\"viewNo\\":null}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":7,\\"viewNo\\":null}}"]}' to Node4 2017-10-23 16:06:54,827 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 4 2017-10-23 16:07:01,339 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 1 2017-10-23 16:07:01,339 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:07:01,339 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:07:01,339 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:07:01,340 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:07:01,340 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:07:01,340 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:07:01,340 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 6 to run in 10 seconds 2017-10-23 16:07:06,836 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:07:06,836 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:07:06,836 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:07:06,837 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:07:11,351 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 6 2017-10-23 16:07:11,351 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:07:11,351 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:07:11,352 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:07:11,352 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:07:11,352 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:07:11,352 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:07:11,352 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 7 to run in 10 seconds 2017-10-23 16:07:21,365 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 7 2017-10-23 16:07:21,365 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:07:21,365 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:07:21,365 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:07:21,365 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:07:21,366 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:07:21,366 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:07:21,366 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 8 to run in 10 seconds 2017-10-23 16:07:21,836 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:07:21,837 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:07:21,837 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:07:21,837 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:07:31,371 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 8 2017-10-23 16:07:31,371 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:07:31,371 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:07:31,371 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:07:31,371 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:07:31,372 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:07:31,372 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:07:31,372 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 9 to run in 10 seconds 2017-10-23 16:07:36,844 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:07:36,845 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:07:36,845 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:07:36,845 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:07:41,382 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 9 2017-10-23 16:07:41,382 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:07:41,383 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:07:41,383 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:07:41,383 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:07:41,383 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:07:41,383 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:07:41,383 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 10 to run in 10 seconds 2017-10-23 16:07:51,345 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 2 2017-10-23 16:07:51,345 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:07:51,345 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:07:51,345 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 11 to run in 60 seconds 2017-10-23 16:07:51,345 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 1 2017-10-23 16:07:51,346 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:07:51,346 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 2 to run in 60 seconds 2017-10-23 16:07:51,394 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 10 2017-10-23 16:07:51,394 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:07:51,394 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:07:51,394 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:07:51,394 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:07:51,394 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:07:51,394 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:07:51,394 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 12 to run in 10 seconds 2017-10-23 16:07:51,831 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 5 2017-10-23 16:07:51,831 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 13 to run in 60 seconds 2017-10-23 16:07:51,832 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action propose_view_change with id 3 2017-10-23 16:07:51,832 | TRACE | node.py (2097) | propose_view_change | The primary is already connected so view change will not be proposed 2017-10-23 16:07:51,856 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:07:51,856 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:07:51,856 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:07:51,856 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:07:52,216 | TRACE | zstack.py ( 479) | _receiveFromListener | Node3 got 1 messages through listener 2017-10-23 16:07:52,217 | DEBUG | node.py (1257) | validateNodeMsg | Node3 received node message from Node1: INSTANCE_CHANGE{'viewNo': 1, 'reason': 26} 2017-10-23 16:07:52,217 | DEBUG | node.py (1219) | handleOneNodeMsg | Node3 msg validated ({'viewNo': 1, 'op': 'INSTANCE_CHANGE', 'reason': 26}, 'Node1') 2017-10-23 16:07:52,217 | DEBUG | node.py (1286) | postToNodeInBox | Node3 appending to nodeInbox INSTANCE_CHANGE{'viewNo': 1, 'reason': 26} 2017-10-23 16:07:52,217 | DEBUG | node.py (1915) | processInstanceChange | Node3 received instance change request: INSTANCE_CHANGE{'viewNo': 1, 'reason': 26} from Node1 2017-10-23 16:07:52,217 | DEBUG | node.py (1958) | do_view_change_if_possible | Node3 has no quorum for view 1 2017-10-23 16:07:52,217 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:07:52,217 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:07:52,217 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:07:52,218 | DEBUG | node.py (1946) | processInstanceChange | Node3 received instance change message INSTANCE_CHANGE{'viewNo': 1, 'reason': 26} but did not find the master to be slow or has already sent an instance change message 2017-10-23 16:08:01,406 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 12 2017-10-23 16:08:01,407 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:08:01,407 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:08:01,407 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:08:01,407 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:08:01,407 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:08:01,407 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:08:01,407 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 14 to run in 10 seconds 2017-10-23 16:08:06,866 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:08:06,867 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:08:06,867 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:08:06,867 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:08:11,419 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 14 2017-10-23 16:08:11,419 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:08:11,419 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:08:11,419 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:08:11,419 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:08:11,419 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:08:11,420 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:08:11,420 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 15 to run in 10 seconds 2017-10-23 16:08:21,429 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 15 2017-10-23 16:08:21,430 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:08:21,430 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:08:21,430 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:08:21,430 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:08:21,430 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:08:21,430 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:08:21,430 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 16 to run in 10 seconds 2017-10-23 16:08:21,871 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:08:21,871 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:08:21,871 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:08:21,871 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:08:31,432 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 16 2017-10-23 16:08:31,432 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:08:31,432 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:08:31,432 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:08:31,432 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:08:31,432 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:08:31,432 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:08:31,432 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 17 to run in 10 seconds 2017-10-23 16:08:36,876 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:08:36,876 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:08:36,877 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:08:36,877 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:08:41,433 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 17 2017-10-23 16:08:41,434 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:08:41,434 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:08:41,434 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:08:41,434 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:08:41,434 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:08:41,434 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:08:41,435 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 18 to run in 10 seconds 2017-10-23 16:08:51,351 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 11 2017-10-23 16:08:51,351 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:08:51,352 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:08:51,352 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 19 to run in 60 seconds 2017-10-23 16:08:51,352 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 2 2017-10-23 16:08:51,352 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:08:51,352 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 3 to run in 60 seconds 2017-10-23 16:08:51,436 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 18 2017-10-23 16:08:51,437 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:08:51,437 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:08:51,437 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:08:51,437 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:08:51,437 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:08:51,437 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:08:51,437 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 20 to run in 10 seconds 2017-10-23 16:08:51,842 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 13 2017-10-23 16:08:51,842 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 21 to run in 60 seconds 2017-10-23 16:08:51,882 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:08:51,883 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:08:51,883 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:08:51,883 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:09:01,442 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 20 2017-10-23 16:09:01,443 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:09:01,443 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:09:01,443 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:09:01,443 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:09:01,443 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:09:01,444 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:09:01,444 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 22 to run in 10 seconds 2017-10-23 16:09:06,894 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:09:06,894 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:09:06,894 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:09:06,895 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:09:11,454 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 22 2017-10-23 16:09:11,454 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:09:11,454 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:09:11,454 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:09:11,454 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:09:11,455 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:09:11,455 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:09:11,455 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 23 to run in 10 seconds 2017-10-23 16:09:21,463 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 23 2017-10-23 16:09:21,463 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:09:21,463 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:09:21,463 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:09:21,463 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:09:21,463 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:09:21,463 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:09:21,463 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 24 to run in 10 seconds 2017-10-23 16:09:21,899 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:09:21,900 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:09:21,900 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:09:21,901 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:09:31,464 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 24 2017-10-23 16:09:31,464 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:09:31,465 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:09:31,465 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:09:31,465 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:09:31,465 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:09:31,466 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:09:31,466 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 25 to run in 10 seconds 2017-10-23 16:09:36,906 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:09:36,907 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:09:36,909 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:09:36,909 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:09:41,468 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 25 2017-10-23 16:09:41,468 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:09:41,468 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:09:41,468 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:09:41,468 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:09:41,469 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:09:41,469 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:09:41,469 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 26 to run in 10 seconds 2017-10-23 16:09:51,356 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 19 2017-10-23 16:09:51,356 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:09:51,356 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:09:51,356 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 27 to run in 60 seconds 2017-10-23 16:09:51,357 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 3 2017-10-23 16:09:51,357 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:09:51,357 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 4 to run in 60 seconds 2017-10-23 16:09:51,469 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 26 2017-10-23 16:09:51,469 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:09:51,469 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:09:51,470 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:09:51,470 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:09:51,470 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:09:51,470 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:09:51,470 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 28 to run in 10 seconds 2017-10-23 16:09:51,845 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 21 2017-10-23 16:09:51,846 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 29 to run in 60 seconds 2017-10-23 16:09:51,916 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:09:51,916 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:09:51,916 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:09:51,916 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:10:01,481 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 28 2017-10-23 16:10:01,481 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:10:01,481 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:10:01,481 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:10:01,481 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:10:01,481 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:10:01,481 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:10:01,481 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 30 to run in 10 seconds 2017-10-23 16:10:06,922 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:10:06,923 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:10:06,923 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:10:06,923 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:10:11,492 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 30 2017-10-23 16:10:11,493 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:10:11,493 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:10:11,493 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:10:11,493 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:10:11,493 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:10:11,493 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:10:11,493 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 31 to run in 10 seconds 2017-10-23 16:10:21,501 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 31 2017-10-23 16:10:21,501 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:10:21,501 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:10:21,501 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:10:21,501 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:10:21,501 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:10:21,501 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:10:21,501 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 32 to run in 10 seconds 2017-10-23 16:10:21,936 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:10:21,936 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:10:21,936 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:10:21,937 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:10:31,508 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 32 2017-10-23 16:10:31,508 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:10:31,509 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:10:31,509 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:10:31,509 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:10:31,509 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:10:31,509 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:10:31,509 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 33 to run in 10 seconds 2017-10-23 16:10:36,943 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:10:36,943 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:10:36,943 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:10:36,944 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:10:41,510 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 33 2017-10-23 16:10:41,510 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:10:41,510 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:10:41,510 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:10:41,510 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:10:41,510 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:10:41,510 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:10:41,510 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 34 to run in 10 seconds 2017-10-23 16:10:51,364 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 27 2017-10-23 16:10:51,364 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:10:51,364 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:10:51,366 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 35 to run in 60 seconds 2017-10-23 16:10:51,366 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 4 2017-10-23 16:10:51,366 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:10:51,367 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 5 to run in 60 seconds 2017-10-23 16:10:51,522 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 34 2017-10-23 16:10:51,522 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:10:51,522 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:10:51,522 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:10:51,523 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:10:51,523 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:10:51,523 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:10:51,523 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 36 to run in 10 seconds 2017-10-23 16:10:51,851 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 29 2017-10-23 16:10:51,852 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 37 to run in 60 seconds 2017-10-23 16:10:51,954 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:10:51,954 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:10:51,954 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:10:51,954 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:11:01,524 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 36 2017-10-23 16:11:01,524 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:11:01,524 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:11:01,524 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:11:01,524 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:11:01,526 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:11:01,526 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:11:01,527 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 38 to run in 10 seconds 2017-10-23 16:11:06,955 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:11:06,956 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:11:06,956 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:11:06,957 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:11:11,536 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 38 2017-10-23 16:11:11,536 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:11:11,536 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:11:11,536 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:11:11,536 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:11:11,536 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:11:11,536 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:11:11,537 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 39 to run in 10 seconds 2017-10-23 16:11:21,545 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 39 2017-10-23 16:11:21,546 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:11:21,546 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:11:21,546 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:11:21,546 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:11:21,546 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:11:21,546 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:11:21,546 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 40 to run in 10 seconds 2017-10-23 16:11:21,962 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:11:21,962 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:11:21,962 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:11:21,963 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:11:31,554 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 40 2017-10-23 16:11:31,554 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:11:31,554 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:11:31,554 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:11:31,554 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:11:31,554 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:11:31,554 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:11:31,554 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 41 to run in 10 seconds 2017-10-23 16:11:36,974 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:11:36,974 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:11:36,974 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:11:36,975 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:11:41,561 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 41 2017-10-23 16:11:41,561 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:11:41,561 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:11:41,561 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:11:41,561 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:11:41,562 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:11:41,562 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:11:41,562 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 42 to run in 10 seconds 2017-10-23 16:11:51,367 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 35 2017-10-23 16:11:51,368 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:11:51,368 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:11:51,368 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 43 to run in 60 seconds 2017-10-23 16:11:51,369 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 5 2017-10-23 16:11:51,369 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:11:51,369 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 6 to run in 60 seconds 2017-10-23 16:11:51,568 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 42 2017-10-23 16:11:51,568 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:11:51,568 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:11:51,568 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:11:51,568 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:11:51,568 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:11:51,568 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:11:51,568 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 44 to run in 10 seconds 2017-10-23 16:11:51,853 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 37 2017-10-23 16:11:51,854 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 45 to run in 60 seconds 2017-10-23 16:11:51,979 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:11:51,979 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:11:51,979 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:11:51,979 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:12:01,576 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 44 2017-10-23 16:12:01,576 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:12:01,576 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:12:01,576 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:12:01,577 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:12:01,577 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:12:01,577 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:12:01,577 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 46 to run in 10 seconds 2017-10-23 16:12:06,985 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:12:06,986 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:12:06,986 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:12:06,987 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:12:11,582 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 46 2017-10-23 16:12:11,582 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:12:11,582 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:12:11,582 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:12:11,583 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:12:11,583 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:12:11,583 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:12:11,583 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 47 to run in 10 seconds 2017-10-23 16:12:21,584 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 47 2017-10-23 16:12:21,584 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:12:21,585 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:12:21,585 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:12:21,585 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:12:21,585 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:12:21,585 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:12:21,585 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 48 to run in 10 seconds 2017-10-23 16:12:21,989 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:12:21,990 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:12:21,990 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:12:21,990 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:12:31,588 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 48 2017-10-23 16:12:31,589 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:12:31,589 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:12:31,589 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:12:31,589 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:12:31,589 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:12:31,589 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:12:31,589 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 49 to run in 10 seconds 2017-10-23 16:12:37,000 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:12:37,000 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:12:37,000 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:12:37,002 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:12:41,591 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 49 2017-10-23 16:12:41,591 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:12:41,591 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:12:41,591 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:12:41,591 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:12:41,591 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:12:41,592 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:12:41,592 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 50 to run in 10 seconds 2017-10-23 16:12:51,371 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 43 2017-10-23 16:12:51,371 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:12:51,371 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:12:51,371 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 51 to run in 60 seconds 2017-10-23 16:12:51,372 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 6 2017-10-23 16:12:51,372 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:12:51,372 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 7 to run in 60 seconds 2017-10-23 16:12:51,593 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 50 2017-10-23 16:12:51,593 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:12:51,593 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:12:51,593 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:12:51,594 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:12:51,594 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:12:51,594 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:12:51,594 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 52 to run in 10 seconds 2017-10-23 16:12:51,861 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 45 2017-10-23 16:12:51,867 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 53 to run in 60 seconds 2017-10-23 16:12:52,007 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:12:52,007 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:12:52,008 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:12:52,008 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:13:01,596 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 52 2017-10-23 16:13:01,597 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:13:01,597 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:13:01,597 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:13:01,597 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:13:01,597 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:13:01,597 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:13:01,598 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 54 to run in 10 seconds 2017-10-23 16:13:07,017 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:13:07,017 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:13:07,017 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:13:07,018 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:13:11,599 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 54 2017-10-23 16:13:11,599 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:13:11,599 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:13:11,599 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:13:11,600 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:13:11,600 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:13:11,600 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:13:11,600 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 55 to run in 10 seconds 2017-10-23 16:13:21,606 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 55 2017-10-23 16:13:21,607 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:13:21,607 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:13:21,607 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:13:21,607 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:13:21,607 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:13:21,607 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:13:21,608 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 56 to run in 10 seconds 2017-10-23 16:13:22,027 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:13:22,027 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:13:22,028 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:13:22,028 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:13:31,619 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 56 2017-10-23 16:13:31,620 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:13:31,620 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:13:31,620 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:13:31,621 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:13:31,621 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:13:31,621 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:13:31,621 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 57 to run in 10 seconds 2017-10-23 16:13:37,032 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:13:37,032 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:13:37,033 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:13:37,033 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:13:41,629 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 57 2017-10-23 16:13:41,629 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:13:41,629 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:13:41,629 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:13:41,629 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:13:41,630 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:13:41,630 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:13:41,630 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 58 to run in 10 seconds 2017-10-23 16:13:51,380 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 51 2017-10-23 16:13:51,380 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:13:51,381 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:13:51,381 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 59 to run in 60 seconds 2017-10-23 16:13:51,381 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 7 2017-10-23 16:13:51,381 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:13:51,381 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 8 to run in 60 seconds 2017-10-23 16:13:51,634 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 58 2017-10-23 16:13:51,634 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:13:51,634 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:13:51,634 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:13:51,634 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:13:51,634 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:13:51,634 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:13:51,634 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 60 to run in 10 seconds 2017-10-23 16:13:51,875 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 53 2017-10-23 16:13:51,876 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 61 to run in 60 seconds 2017-10-23 16:13:52,041 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:13:52,041 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:13:52,041 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:13:52,042 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:14:01,647 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 60 2017-10-23 16:14:01,647 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:14:01,647 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:14:01,647 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:14:01,648 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:14:01,648 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:14:01,648 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:14:01,648 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 62 to run in 10 seconds 2017-10-23 16:14:07,051 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:14:07,052 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:14:07,052 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:14:07,052 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:14:11,660 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 62 2017-10-23 16:14:11,660 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:14:11,660 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:14:11,660 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:14:11,660 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:14:11,661 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:14:11,661 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:14:11,661 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 63 to run in 10 seconds 2017-10-23 16:14:21,668 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 63 2017-10-23 16:14:21,668 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:14:21,668 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:14:21,668 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:14:21,668 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:14:21,668 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:14:21,669 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:14:21,669 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 64 to run in 10 seconds 2017-10-23 16:14:22,054 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:14:22,054 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:14:22,054 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:14:22,055 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:14:31,683 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 64 2017-10-23 16:14:31,683 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:14:31,683 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:14:31,683 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:14:31,683 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:14:31,683 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:14:31,683 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:14:31,684 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 65 to run in 10 seconds 2017-10-23 16:14:37,055 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:14:37,056 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:14:37,056 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:14:37,056 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:14:41,696 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 65 2017-10-23 16:14:41,696 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:14:41,696 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:14:41,697 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:14:41,697 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:14:41,697 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:14:41,697 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:14:41,697 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 66 to run in 10 seconds 2017-10-23 16:14:51,381 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 59 2017-10-23 16:14:51,381 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:14:51,382 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:14:51,382 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 67 to run in 60 seconds 2017-10-23 16:14:51,382 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 8 2017-10-23 16:14:51,382 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:14:51,382 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 9 to run in 60 seconds 2017-10-23 16:14:51,704 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 66 2017-10-23 16:14:51,704 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:14:51,704 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:14:51,705 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:14:51,705 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:14:51,705 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:14:51,705 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:14:51,705 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 68 to run in 10 seconds 2017-10-23 16:14:51,884 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 61 2017-10-23 16:14:51,887 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 69 to run in 60 seconds 2017-10-23 16:14:52,063 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:14:52,064 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:14:52,064 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:14:52,064 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:15:01,708 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 68 2017-10-23 16:15:01,708 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:15:01,708 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:15:01,708 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:15:01,709 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:15:01,709 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:15:01,709 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:15:01,709 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 70 to run in 10 seconds 2017-10-23 16:15:07,069 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:15:07,069 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:15:07,069 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:15:07,069 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:15:11,713 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 70 2017-10-23 16:15:11,714 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:15:11,714 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:15:11,714 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:15:11,714 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:15:11,715 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:15:11,715 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:15:11,715 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 71 to run in 10 seconds 2017-10-23 16:15:21,726 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 71 2017-10-23 16:15:21,726 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:15:21,731 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:15:21,732 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:15:21,732 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:15:21,732 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:15:21,733 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:15:21,733 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 72 to run in 10 seconds 2017-10-23 16:15:22,072 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:15:22,072 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:15:22,073 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:15:22,073 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:15:31,736 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 72 2017-10-23 16:15:31,736 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:15:31,736 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:15:31,736 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:15:31,736 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:15:31,737 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:15:31,737 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:15:31,737 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 73 to run in 10 seconds 2017-10-23 16:15:37,073 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:15:37,073 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:15:37,073 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:15:37,074 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:15:41,740 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 73 2017-10-23 16:15:41,740 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:15:41,741 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:15:41,741 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:15:41,741 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:15:41,741 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:15:41,741 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:15:41,741 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 74 to run in 10 seconds 2017-10-23 16:15:51,392 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 67 2017-10-23 16:15:51,393 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:15:51,393 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:15:51,393 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 75 to run in 60 seconds 2017-10-23 16:15:51,393 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 9 2017-10-23 16:15:51,393 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:15:51,393 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 10 to run in 60 seconds 2017-10-23 16:15:51,751 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 74 2017-10-23 16:15:51,751 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:15:51,751 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:15:51,751 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:15:51,752 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:15:51,752 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:15:51,752 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:15:51,752 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 76 to run in 10 seconds 2017-10-23 16:15:51,893 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 69 2017-10-23 16:15:51,894 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 77 to run in 60 seconds 2017-10-23 16:15:52,081 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:15:52,081 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:15:52,082 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:15:52,083 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:16:01,753 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 76 2017-10-23 16:16:01,753 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:16:01,753 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:16:01,753 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:16:01,753 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:16:01,753 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:16:01,753 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:16:01,753 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 78 to run in 10 seconds 2017-10-23 16:16:07,084 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:16:07,084 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:16:07,085 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:16:07,085 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:16:11,756 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 78 2017-10-23 16:16:11,756 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:16:11,756 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:16:11,756 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:16:11,757 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:16:11,757 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:16:11,757 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:16:11,757 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 79 to run in 10 seconds 2017-10-23 16:16:21,758 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 79 2017-10-23 16:16:21,759 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:16:21,759 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:16:21,759 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:16:21,759 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:16:21,759 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:16:21,761 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:16:21,761 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 80 to run in 10 seconds 2017-10-23 16:16:22,090 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:16:22,090 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:16:22,090 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:16:22,090 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:16:31,769 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 80 2017-10-23 16:16:31,769 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:16:31,770 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:16:31,770 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:16:31,770 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:16:31,770 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:16:31,770 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:16:31,770 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 81 to run in 10 seconds 2017-10-23 16:16:37,098 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:16:37,098 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:16:37,098 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:16:37,099 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:16:41,778 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 81 2017-10-23 16:16:41,778 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:16:41,778 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:16:41,778 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:16:41,779 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:16:41,779 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:16:41,779 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:16:41,779 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 82 to run in 10 seconds 2017-10-23 16:16:51,402 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 75 2017-10-23 16:16:51,403 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:16:51,403 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:16:51,403 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 83 to run in 60 seconds 2017-10-23 16:16:51,403 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 10 2017-10-23 16:16:51,403 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:16:51,404 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 11 to run in 60 seconds 2017-10-23 16:16:51,781 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 82 2017-10-23 16:16:51,782 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:16:51,782 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:16:51,782 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:16:51,782 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:16:51,782 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:16:51,782 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:16:51,782 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 84 to run in 10 seconds 2017-10-23 16:16:51,900 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 77 2017-10-23 16:16:51,902 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 85 to run in 60 seconds 2017-10-23 16:16:52,110 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:16:52,111 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:16:52,111 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:16:52,112 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:17:01,790 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 84 2017-10-23 16:17:01,790 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:17:01,791 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:17:01,791 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:17:01,791 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:17:01,791 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:17:01,792 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:17:01,792 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 86 to run in 10 seconds 2017-10-23 16:17:07,119 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:17:07,119 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:17:07,119 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:17:07,120 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:17:11,796 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 86 2017-10-23 16:17:11,797 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:17:11,797 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:17:11,797 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:17:11,797 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:17:11,797 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:17:11,797 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:17:11,797 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 87 to run in 10 seconds 2017-10-23 16:17:21,799 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 87 2017-10-23 16:17:21,799 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:17:21,799 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:17:21,799 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:17:21,800 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:17:21,800 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:17:21,800 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:17:21,800 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 88 to run in 10 seconds 2017-10-23 16:17:22,125 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:17:22,126 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:17:22,126 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:17:22,127 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:17:31,801 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 88 2017-10-23 16:17:31,801 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:17:31,801 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:17:31,802 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:17:31,802 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:17:31,802 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:17:31,802 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:17:31,802 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 89 to run in 10 seconds 2017-10-23 16:17:37,133 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:17:37,133 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:17:37,133 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:17:37,134 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:17:41,810 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 89 2017-10-23 16:17:41,810 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:17:41,810 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:17:41,810 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:17:41,810 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:17:41,811 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:17:41,811 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:17:41,811 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 90 to run in 10 seconds 2017-10-23 16:17:51,405 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 83 2017-10-23 16:17:51,406 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:17:51,406 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:17:51,406 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 91 to run in 60 seconds 2017-10-23 16:17:51,406 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 11 2017-10-23 16:17:51,406 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:17:51,406 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 12 to run in 60 seconds 2017-10-23 16:17:51,824 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 90 2017-10-23 16:17:51,824 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:17:51,824 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:17:51,825 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:17:51,825 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:17:51,825 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:17:51,825 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:17:51,825 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 92 to run in 10 seconds 2017-10-23 16:17:51,915 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 85 2017-10-23 16:17:51,916 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 93 to run in 60 seconds 2017-10-23 16:17:52,138 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:17:52,138 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:17:52,138 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:17:52,139 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:18:01,830 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 92 2017-10-23 16:18:01,830 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:18:01,830 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:18:01,830 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:18:01,831 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:18:01,831 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:18:01,831 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:18:01,831 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 94 to run in 10 seconds 2017-10-23 16:18:07,145 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:18:07,145 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:18:07,145 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:18:07,145 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:18:11,846 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 94 2017-10-23 16:18:11,847 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:18:11,847 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:18:11,848 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:18:11,848 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:18:11,849 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:18:11,849 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:18:11,850 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 95 to run in 10 seconds 2017-10-23 16:18:21,852 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 95 2017-10-23 16:18:21,852 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:18:21,853 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:18:21,853 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:18:21,853 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:18:21,853 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:18:21,853 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:18:21,853 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 96 to run in 10 seconds 2017-10-23 16:18:22,147 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:18:22,147 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:18:22,147 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:18:22,148 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:18:31,856 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 96 2017-10-23 16:18:31,856 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:18:31,856 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:18:31,856 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:18:31,856 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:18:31,857 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:18:31,857 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:18:31,857 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 97 to run in 10 seconds 2017-10-23 16:18:37,148 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:18:37,148 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:18:37,148 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:18:37,149 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:18:41,865 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 97 2017-10-23 16:18:41,865 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:18:41,865 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:18:41,865 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:18:41,865 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:18:41,866 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:18:41,866 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:18:41,866 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 98 to run in 10 seconds 2017-10-23 16:18:51,416 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 91 2017-10-23 16:18:51,416 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:18:51,416 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:18:51,417 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 99 to run in 60 seconds 2017-10-23 16:18:51,417 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 12 2017-10-23 16:18:51,417 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:18:51,417 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 13 to run in 60 seconds 2017-10-23 16:18:51,876 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 98 2017-10-23 16:18:51,876 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:18:51,877 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:18:51,877 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:18:51,877 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:18:51,877 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:18:51,877 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:18:51,877 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 100 to run in 10 seconds 2017-10-23 16:18:51,919 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 93 2017-10-23 16:18:51,920 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 101 to run in 60 seconds 2017-10-23 16:18:52,150 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:18:52,150 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:18:52,150 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:18:52,151 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:19:01,883 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 100 2017-10-23 16:19:01,883 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:19:01,883 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:19:01,884 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:19:01,884 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:19:01,884 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:19:01,884 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:19:01,885 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 102 to run in 10 seconds 2017-10-23 16:19:07,162 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:19:07,162 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:19:07,162 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:19:07,163 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:19:11,888 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 102 2017-10-23 16:19:11,888 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:19:11,888 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:19:11,888 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:19:11,888 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:19:11,889 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:19:11,889 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:19:11,889 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 103 to run in 10 seconds 2017-10-23 16:19:21,892 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 103 2017-10-23 16:19:21,892 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:19:21,892 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:19:21,892 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:19:21,892 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:19:21,892 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:19:21,892 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:19:21,892 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 104 to run in 10 seconds 2017-10-23 16:19:22,166 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:19:22,167 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:19:22,167 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:19:22,168 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:19:31,903 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 104 2017-10-23 16:19:31,903 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:19:31,903 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:19:31,903 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:19:31,903 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:19:31,903 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:19:31,903 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:19:31,903 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 105 to run in 10 seconds 2017-10-23 16:19:37,167 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:19:37,168 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:19:37,168 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:19:37,169 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:19:41,905 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 105 2017-10-23 16:19:41,905 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:19:41,905 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:19:41,905 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:19:41,905 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:19:41,905 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:19:41,905 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:19:41,905 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 106 to run in 10 seconds 2017-10-23 16:19:51,417 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 99 2017-10-23 16:19:51,417 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:19:51,417 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:19:51,417 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 107 to run in 60 seconds 2017-10-23 16:19:51,417 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 13 2017-10-23 16:19:51,417 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:19:51,417 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 14 to run in 60 seconds 2017-10-23 16:19:51,918 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 106 2017-10-23 16:19:51,918 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:19:51,918 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:19:51,918 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:19:51,918 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:19:51,918 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:19:51,918 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:19:51,918 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 108 to run in 10 seconds 2017-10-23 16:19:51,933 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 101 2017-10-23 16:19:51,935 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 109 to run in 60 seconds 2017-10-23 16:19:52,169 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:19:52,169 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:19:52,169 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:19:52,170 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:20:01,921 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 108 2017-10-23 16:20:01,921 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:20:01,921 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:20:01,921 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:20:01,921 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:20:01,921 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:20:01,921 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:20:01,921 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 110 to run in 10 seconds 2017-10-23 16:20:07,177 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:20:07,178 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:20:07,178 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:20:07,178 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:20:11,936 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 110 2017-10-23 16:20:11,937 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:20:11,937 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:20:11,937 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:20:11,937 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:20:11,938 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:20:11,938 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:20:11,938 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 111 to run in 10 seconds 2017-10-23 16:20:21,939 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 111 2017-10-23 16:20:21,939 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:20:21,939 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:20:21,940 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:20:21,940 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:20:21,940 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:20:21,940 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:20:21,940 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 112 to run in 10 seconds 2017-10-23 16:20:22,190 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:20:22,190 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:20:22,190 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:20:22,190 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:20:31,943 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 112 2017-10-23 16:20:31,943 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:20:31,943 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:20:31,944 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:20:31,944 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:20:31,944 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:20:31,944 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:20:31,944 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 113 to run in 10 seconds 2017-10-23 16:20:37,193 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:20:37,193 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:20:37,193 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:20:37,193 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:20:41,952 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 113 2017-10-23 16:20:41,952 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:20:41,953 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:20:41,953 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:20:41,953 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:20:41,953 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:20:41,953 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:20:41,953 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 114 to run in 10 seconds 2017-10-23 16:20:51,422 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 107 2017-10-23 16:20:51,422 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:20:51,422 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:20:51,422 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 115 to run in 60 seconds 2017-10-23 16:20:51,422 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 14 2017-10-23 16:20:51,423 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:20:51,423 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 15 to run in 60 seconds 2017-10-23 16:20:51,936 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 109 2017-10-23 16:20:51,937 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 116 to run in 60 seconds 2017-10-23 16:20:51,961 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 114 2017-10-23 16:20:51,961 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:20:51,961 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:20:51,961 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:20:51,962 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:20:51,962 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:20:51,962 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:20:51,962 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 117 to run in 10 seconds 2017-10-23 16:20:52,202 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:20:52,202 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:20:52,202 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:20:52,202 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:21:01,966 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 117 2017-10-23 16:21:01,966 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:21:01,966 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:21:01,966 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:21:01,966 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:21:01,966 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:21:01,966 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:21:01,966 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 118 to run in 10 seconds 2017-10-23 16:21:07,205 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:21:07,206 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:21:07,206 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:21:07,207 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:21:11,968 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 118 2017-10-23 16:21:11,968 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:21:11,968 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-23 16:21:11,968 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:21:11,968 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:21:11,969 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:21:11,969 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:21:11,969 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 119 to run in 10 seconds 2017-10-23 16:21:21,978 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 119 2017-10-23 16:21:21,978 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:21:21,978 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:21:21,979 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:21:21,979 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:21:21,979 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:21:21,979 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:21:21,980 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 120 to run in 10 seconds 2017-10-23 16:21:22,210 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:21:22,210 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:21:22,210 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:21:22,210 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:21:31,988 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 120 2017-10-23 16:21:31,989 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:21:31,989 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:21:31,989 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:21:31,989 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:21:31,989 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:21:31,989 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:21:31,989 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 121 to run in 10 seconds 2017-10-23 16:21:37,220 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:21:37,221 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:21:37,221 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:21:37,221 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:21:41,993 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 121 2017-10-23 16:21:41,994 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:21:41,994 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:21:41,994 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:21:41,995 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:21:41,995 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:21:41,995 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:21:41,995 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 122 to run in 10 seconds 2017-10-23 16:21:51,431 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 115 2017-10-23 16:21:51,431 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:21:51,432 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:21:51,432 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 123 to run in 60 seconds 2017-10-23 16:21:51,432 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 15 2017-10-23 16:21:51,432 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:21:51,432 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 16 to run in 60 seconds 2017-10-23 16:21:51,941 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 116 2017-10-23 16:21:51,945 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 124 to run in 60 seconds 2017-10-23 16:21:51,997 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 122 2017-10-23 16:21:51,998 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:21:51,998 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:21:51,998 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:21:51,998 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:21:51,998 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:21:51,999 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:21:51,999 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 125 to run in 10 seconds 2017-10-23 16:21:52,221 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:21:52,221 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:21:52,221 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:21:52,221 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:22:02,004 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 125 2017-10-23 16:22:02,004 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:22:02,004 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:22:02,004 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:22:02,004 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:22:02,005 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:22:02,005 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:22:02,005 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 126 to run in 10 seconds 2017-10-23 16:22:07,233 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:22:07,234 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:22:07,234 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:22:07,235 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:22:12,010 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 126 2017-10-23 16:22:12,010 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:22:12,010 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:22:12,010 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:22:12,010 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:22:12,011 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:22:12,011 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:22:12,011 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 127 to run in 10 seconds 2017-10-23 16:22:22,020 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 127 2017-10-23 16:22:22,020 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:22:22,021 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:22:22,021 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:22:22,021 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:22:22,021 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:22:22,021 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:22:22,021 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 128 to run in 10 seconds 2017-10-23 16:22:22,242 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:22:22,242 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:22:22,242 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:22:22,243 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:22:32,032 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 128 2017-10-23 16:22:32,032 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:22:32,032 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:22:32,032 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:22:32,032 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:22:32,032 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:22:32,032 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:22:32,033 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 129 to run in 10 seconds 2017-10-23 16:22:37,246 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:22:37,248 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:22:37,248 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:22:37,248 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:22:42,047 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 129 2017-10-23 16:22:42,047 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:22:42,047 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:22:42,047 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:22:42,047 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:22:42,047 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:22:42,048 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:22:42,048 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 130 to run in 10 seconds 2017-10-23 16:22:51,434 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 123 2017-10-23 16:22:51,435 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:22:51,435 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:22:51,435 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 131 to run in 60 seconds 2017-10-23 16:22:51,435 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 16 2017-10-23 16:22:51,435 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:22:51,435 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 17 to run in 60 seconds 2017-10-23 16:22:51,955 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 124 2017-10-23 16:22:51,956 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 132 to run in 60 seconds 2017-10-23 16:22:52,053 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 130 2017-10-23 16:22:52,054 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:22:52,054 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:22:52,054 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:22:52,054 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:22:52,054 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:22:52,054 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:22:52,054 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 133 to run in 10 seconds 2017-10-23 16:22:52,252 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:22:52,252 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:22:52,252 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:22:52,252 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:23:02,055 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 133 2017-10-23 16:23:02,055 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:23:02,056 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:23:02,056 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:23:02,056 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:23:02,056 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:23:02,056 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:23:02,057 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 134 to run in 10 seconds 2017-10-23 16:23:07,262 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:23:07,263 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:23:07,263 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:23:07,263 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:23:12,060 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 134 2017-10-23 16:23:12,060 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:23:12,060 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:23:12,061 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:23:12,061 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:23:12,061 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:23:12,061 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:23:12,061 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 135 to run in 10 seconds 2017-10-23 16:23:22,067 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 135 2017-10-23 16:23:22,067 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:23:22,067 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:23:22,067 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:23:22,067 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:23:22,068 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:23:22,068 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:23:22,068 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 136 to run in 10 seconds 2017-10-23 16:23:22,270 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:23:22,270 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:23:22,270 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:23:22,271 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:23:32,076 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 136 2017-10-23 16:23:32,076 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:23:32,076 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:23:32,076 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:23:32,076 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:23:32,076 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:23:32,077 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:23:32,077 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 137 to run in 10 seconds 2017-10-23 16:23:37,273 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:23:37,274 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:23:37,274 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:23:37,275 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:23:42,081 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 137 2017-10-23 16:23:42,081 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:23:42,081 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:23:42,081 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:23:42,081 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:23:42,081 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:23:42,081 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:23:42,081 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 138 to run in 10 seconds 2017-10-23 16:23:51,440 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 131 2017-10-23 16:23:51,440 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:23:51,440 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:23:51,440 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 139 to run in 60 seconds 2017-10-23 16:23:51,441 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 17 2017-10-23 16:23:51,441 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:23:51,441 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 18 to run in 60 seconds 2017-10-23 16:23:51,963 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 132 2017-10-23 16:23:51,964 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 140 to run in 60 seconds 2017-10-23 16:23:52,085 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 138 2017-10-23 16:23:52,085 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:23:52,085 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:23:52,085 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:23:52,086 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:23:52,086 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:23:52,086 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:23:52,086 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 141 to run in 10 seconds 2017-10-23 16:23:52,281 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:23:52,281 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:23:52,281 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:23:52,281 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:24:02,091 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 141 2017-10-23 16:24:02,091 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:24:02,091 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:24:02,092 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:24:02,092 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:24:02,092 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:24:02,092 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:24:02,092 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 142 to run in 10 seconds 2017-10-23 16:24:07,290 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:24:07,290 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:24:07,290 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:24:07,291 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:24:12,103 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 142 2017-10-23 16:24:12,103 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:24:12,103 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:24:12,103 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:24:12,103 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:24:12,103 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:24:12,103 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:24:12,104 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 143 to run in 10 seconds 2017-10-23 16:24:22,112 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 143 2017-10-23 16:24:22,113 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:24:22,113 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:24:22,113 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:24:22,113 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:24:22,113 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:24:22,113 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:24:22,114 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 144 to run in 10 seconds 2017-10-23 16:24:22,299 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:24:22,299 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:24:22,300 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:24:22,300 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:24:32,122 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 144 2017-10-23 16:24:32,122 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:24:32,122 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:24:32,122 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:24:32,122 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:24:32,122 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:24:32,123 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:24:32,123 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 145 to run in 10 seconds 2017-10-23 16:24:37,306 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:24:37,306 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:24:37,306 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:24:37,307 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:24:42,128 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 145 2017-10-23 16:24:42,128 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:24:42,128 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:24:42,128 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:24:42,128 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:24:42,128 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:24:42,128 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:24:42,128 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 146 to run in 10 seconds 2017-10-23 16:24:51,444 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 139 2017-10-23 16:24:51,444 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:24:51,444 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:24:51,444 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 147 to run in 60 seconds 2017-10-23 16:24:51,444 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 18 2017-10-23 16:24:51,444 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:24:51,445 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 19 to run in 60 seconds 2017-10-23 16:24:51,976 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 140 2017-10-23 16:24:51,978 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 148 to run in 60 seconds 2017-10-23 16:24:52,134 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 146 2017-10-23 16:24:52,135 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:24:52,135 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:24:52,135 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:24:52,135 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:24:52,135 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:24:52,136 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:24:52,136 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 149 to run in 10 seconds 2017-10-23 16:24:52,307 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:24:52,307 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:24:52,307 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:24:52,308 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:25:02,140 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 149 2017-10-23 16:25:02,140 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:25:02,140 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:25:02,141 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:25:02,141 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:25:02,141 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:25:02,141 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:25:02,141 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 150 to run in 10 seconds 2017-10-23 16:25:07,309 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:25:07,310 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:25:07,310 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:25:07,313 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:25:12,148 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 150 2017-10-23 16:25:12,148 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:25:12,148 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:25:12,149 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:25:12,149 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:25:12,149 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:25:12,149 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:25:12,149 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 151 to run in 10 seconds 2017-10-23 16:25:22,157 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 151 2017-10-23 16:25:22,157 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:25:22,157 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:25:22,157 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:25:22,157 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:25:22,157 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:25:22,157 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:25:22,157 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 152 to run in 10 seconds 2017-10-23 16:25:22,312 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:25:22,312 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:25:22,313 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:25:22,313 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:25:32,169 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 152 2017-10-23 16:25:32,170 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:25:32,170 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:25:32,170 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:25:32,170 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:25:32,170 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:25:32,170 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:25:32,170 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 153 to run in 10 seconds 2017-10-23 16:25:37,325 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:25:37,326 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:25:37,326 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:25:37,328 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:25:42,174 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 153 2017-10-23 16:25:42,175 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:25:42,175 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:25:42,175 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:25:42,175 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:25:42,175 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:25:42,176 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:25:42,176 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 154 to run in 10 seconds 2017-10-23 16:25:51,451 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 147 2017-10-23 16:25:51,451 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:25:51,452 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:25:51,452 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 155 to run in 60 seconds 2017-10-23 16:25:51,452 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 19 2017-10-23 16:25:51,452 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:25:51,452 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 20 to run in 60 seconds 2017-10-23 16:25:51,989 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 148 2017-10-23 16:25:51,990 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 156 to run in 60 seconds 2017-10-23 16:25:52,185 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 154 2017-10-23 16:25:52,185 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:25:52,185 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:25:52,185 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:25:52,185 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:25:52,185 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:25:52,185 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:25:52,186 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 157 to run in 10 seconds 2017-10-23 16:25:52,332 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:25:52,332 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:25:52,332 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:25:52,333 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:26:02,191 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 157 2017-10-23 16:26:02,192 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:26:02,192 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:26:02,192 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:26:02,192 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:26:02,192 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:26:02,192 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:26:02,192 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 158 to run in 10 seconds 2017-10-23 16:26:07,334 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:26:07,334 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:26:07,334 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:26:07,335 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:26:12,194 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 158 2017-10-23 16:26:12,194 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:26:12,195 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:26:12,195 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:26:12,195 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:26:12,195 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:26:12,195 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:26:12,195 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 159 to run in 10 seconds 2017-10-23 16:26:22,200 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 159 2017-10-23 16:26:22,200 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:26:22,200 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:26:22,200 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:26:22,201 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:26:22,201 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:26:22,201 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:26:22,201 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 160 to run in 10 seconds 2017-10-23 16:26:22,341 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:26:22,342 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:26:22,342 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:26:22,343 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:26:32,209 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 160 2017-10-23 16:26:32,209 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:26:32,209 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:26:32,210 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:26:32,210 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:26:32,210 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:26:32,211 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:26:32,211 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 161 to run in 10 seconds 2017-10-23 16:26:37,344 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:26:37,344 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:26:37,345 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:26:37,345 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:26:42,222 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 161 2017-10-23 16:26:42,223 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:26:42,223 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:26:42,223 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:26:42,223 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:26:42,223 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:26:42,223 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:26:42,223 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 162 to run in 10 seconds 2017-10-23 16:26:51,456 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 155 2017-10-23 16:26:51,456 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:26:51,456 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:26:51,456 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 163 to run in 60 seconds 2017-10-23 16:26:51,456 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 20 2017-10-23 16:26:51,456 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:26:51,456 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 21 to run in 60 seconds 2017-10-23 16:26:51,994 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 156 2017-10-23 16:26:51,995 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 164 to run in 60 seconds 2017-10-23 16:26:52,232 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 162 2017-10-23 16:26:52,233 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:26:52,233 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:26:52,233 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:26:52,233 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:26:52,233 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:26:52,233 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:26:52,233 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 165 to run in 10 seconds 2017-10-23 16:26:52,354 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:26:52,354 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:26:52,354 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:26:52,356 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:27:02,240 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 165 2017-10-23 16:27:02,240 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:27:02,240 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:27:02,240 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:27:02,240 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:27:02,241 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:27:02,241 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:27:02,241 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 166 to run in 10 seconds 2017-10-23 16:27:07,356 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:27:07,356 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:27:07,357 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:27:07,357 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:27:12,248 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 166 2017-10-23 16:27:12,252 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:27:12,253 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:27:12,253 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:27:12,253 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:27:12,253 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:27:12,254 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:27:12,254 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 167 to run in 10 seconds 2017-10-23 16:27:22,266 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 167 2017-10-23 16:27:22,267 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:27:22,267 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:27:22,267 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:27:22,267 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:27:22,268 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:27:22,268 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:27:22,268 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 168 to run in 10 seconds 2017-10-23 16:27:22,368 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:27:22,369 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:27:22,369 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:27:22,370 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:27:32,269 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 168 2017-10-23 16:27:32,269 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:27:32,270 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:27:32,270 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:27:32,271 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:27:32,271 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:27:32,271 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:27:32,271 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 169 to run in 10 seconds 2017-10-23 16:27:37,378 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:27:37,378 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:27:37,378 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:27:37,378 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:27:42,281 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 169 2017-10-23 16:27:42,281 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:27:42,281 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:27:42,281 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:27:42,281 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:27:42,281 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:27:42,281 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:27:42,282 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 170 to run in 10 seconds 2017-10-23 16:27:51,462 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 163 2017-10-23 16:27:51,462 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:27:51,462 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:27:51,462 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 171 to run in 60 seconds 2017-10-23 16:27:51,462 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 21 2017-10-23 16:27:51,463 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:27:51,463 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 22 to run in 60 seconds 2017-10-23 16:27:51,999 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 164 2017-10-23 16:27:51,999 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 172 to run in 60 seconds 2017-10-23 16:27:52,282 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 170 2017-10-23 16:27:52,283 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:27:52,286 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:27:52,287 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:27:52,287 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:27:52,288 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:27:52,288 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:27:52,288 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 173 to run in 10 seconds 2017-10-23 16:27:52,380 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:27:52,380 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:27:52,380 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:27:52,381 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:28:02,289 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 173 2017-10-23 16:28:02,289 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:28:02,290 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:28:02,290 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:28:02,290 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:28:02,290 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:28:02,290 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:28:02,290 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 174 to run in 10 seconds 2017-10-23 16:28:07,383 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:28:07,383 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:28:07,383 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:28:07,384 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:28:12,292 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 174 2017-10-23 16:28:12,292 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:28:12,293 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:28:12,293 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:28:12,293 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:28:12,293 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:28:12,293 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:28:12,293 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 175 to run in 10 seconds 2017-10-23 16:28:22,303 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 175 2017-10-23 16:28:22,303 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:28:22,303 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:28:22,303 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:28:22,303 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:28:22,303 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:28:22,303 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:28:22,303 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 176 to run in 10 seconds 2017-10-23 16:28:22,386 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:28:22,386 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:28:22,387 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:28:22,387 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:28:32,308 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 176 2017-10-23 16:28:32,308 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:28:32,308 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:28:32,308 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:28:32,309 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:28:32,309 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:28:32,309 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:28:32,309 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 177 to run in 10 seconds 2017-10-23 16:28:37,390 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:28:37,390 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:28:37,391 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:28:37,391 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:28:42,318 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 177 2017-10-23 16:28:42,318 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:28:42,318 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:28:42,318 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:28:42,318 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:28:42,319 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:28:42,319 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:28:42,319 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 178 to run in 10 seconds 2017-10-23 16:28:51,476 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 171 2017-10-23 16:28:51,476 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:28:51,476 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:28:51,476 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 179 to run in 60 seconds 2017-10-23 16:28:51,476 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 22 2017-10-23 16:28:51,476 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:28:51,476 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 23 to run in 60 seconds 2017-10-23 16:28:52,009 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 172 2017-10-23 16:28:52,011 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 180 to run in 60 seconds 2017-10-23 16:28:52,326 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 178 2017-10-23 16:28:52,326 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:28:52,326 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:28:52,326 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:28:52,326 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:28:52,326 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:28:52,326 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:28:52,326 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 181 to run in 10 seconds 2017-10-23 16:28:52,393 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:28:52,394 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:28:52,394 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:28:52,394 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:29:02,333 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 181 2017-10-23 16:29:02,334 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:29:02,334 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:29:02,334 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:29:02,334 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:29:02,334 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:29:02,334 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:29:02,334 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 182 to run in 10 seconds 2017-10-23 16:29:07,404 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:29:07,404 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:29:07,404 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:29:07,405 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:29:12,342 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 182 2017-10-23 16:29:12,342 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:29:12,342 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:29:12,342 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:29:12,342 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:29:12,342 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:29:12,342 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:29:12,342 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 183 to run in 10 seconds 2017-10-23 16:29:22,353 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 183 2017-10-23 16:29:22,354 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:29:22,354 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:29:22,354 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:29:22,354 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:29:22,355 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:29:22,355 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:29:22,355 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 184 to run in 10 seconds 2017-10-23 16:29:22,406 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:29:22,406 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:29:22,406 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:29:22,407 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:29:32,359 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 184 2017-10-23 16:29:32,359 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:29:32,359 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:29:32,360 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:29:32,360 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:29:32,360 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:29:32,360 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:29:32,360 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 185 to run in 10 seconds 2017-10-23 16:29:37,406 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:29:37,408 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:29:37,408 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:29:37,410 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:29:42,362 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 185 2017-10-23 16:29:42,362 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:29:42,362 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:29:42,362 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:29:42,362 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:29:42,362 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:29:42,363 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:29:42,363 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 186 to run in 10 seconds 2017-10-23 16:29:51,479 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 179 2017-10-23 16:29:51,479 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:29:51,479 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:29:51,480 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 187 to run in 60 seconds 2017-10-23 16:29:51,480 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 23 2017-10-23 16:29:51,480 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:29:51,481 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 24 to run in 60 seconds 2017-10-23 16:29:52,021 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 180 2017-10-23 16:29:52,021 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 188 to run in 60 seconds 2017-10-23 16:29:52,367 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 186 2017-10-23 16:29:52,367 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:29:52,367 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:29:52,368 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:29:52,368 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:29:52,368 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:29:52,368 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:29:52,368 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 189 to run in 10 seconds 2017-10-23 16:29:52,416 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:29:52,416 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:29:52,416 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:29:52,416 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:30:02,375 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 189 2017-10-23 16:30:02,375 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:30:02,375 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:30:02,376 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:30:02,376 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:30:02,376 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:30:02,376 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:30:02,376 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 190 to run in 10 seconds 2017-10-23 16:30:07,417 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:30:07,417 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:30:07,418 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:30:07,418 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:30:12,380 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 190 2017-10-23 16:30:12,380 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:30:12,380 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:30:12,380 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:30:12,381 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:30:12,381 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:30:12,381 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:30:12,381 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 191 to run in 10 seconds 2017-10-23 16:30:22,391 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 191 2017-10-23 16:30:22,391 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:30:22,391 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:30:22,391 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:30:22,391 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:30:22,392 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:30:22,392 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:30:22,392 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 192 to run in 10 seconds 2017-10-23 16:30:22,429 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:30:22,429 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:30:22,429 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:30:22,430 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:30:32,394 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 192 2017-10-23 16:30:32,394 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:30:32,395 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:30:32,397 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:30:32,398 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:30:32,398 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:30:32,398 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:30:32,398 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 193 to run in 10 seconds 2017-10-23 16:30:37,438 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:30:37,439 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:30:37,439 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:30:37,439 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:30:42,405 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 193 2017-10-23 16:30:42,406 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:30:42,406 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:30:42,406 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:30:42,406 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:30:42,407 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:30:42,407 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:30:42,407 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 194 to run in 10 seconds 2017-10-23 16:30:51,486 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 187 2017-10-23 16:30:51,486 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:30:51,486 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:30:51,486 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 195 to run in 60 seconds 2017-10-23 16:30:51,486 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 24 2017-10-23 16:30:51,486 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:30:51,486 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 25 to run in 60 seconds 2017-10-23 16:30:52,031 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 188 2017-10-23 16:30:52,032 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 196 to run in 60 seconds 2017-10-23 16:30:52,408 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 194 2017-10-23 16:30:52,408 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:30:52,408 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:30:52,408 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:30:52,408 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:30:52,409 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:30:52,409 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:30:52,409 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 197 to run in 10 seconds 2017-10-23 16:30:52,447 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:30:52,447 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:30:52,448 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:30:52,448 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:31:02,418 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 197 2017-10-23 16:31:02,418 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:31:02,418 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:31:02,418 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:31:02,418 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:31:02,419 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:31:02,419 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:31:02,419 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 198 to run in 10 seconds 2017-10-23 16:31:07,447 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:31:07,448 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:31:07,448 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:31:07,448 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:31:12,427 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 198 2017-10-23 16:31:12,427 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:31:12,427 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:31:12,427 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:31:12,427 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:31:12,427 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:31:12,428 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:31:12,428 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 199 to run in 10 seconds 2017-10-23 16:31:22,431 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 199 2017-10-23 16:31:22,431 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:31:22,431 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:31:22,431 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:31:22,431 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:31:22,432 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:31:22,432 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:31:22,432 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 200 to run in 10 seconds 2017-10-23 16:31:22,458 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:31:22,458 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:31:22,459 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:31:22,461 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:31:32,438 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 200 2017-10-23 16:31:32,438 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:31:32,438 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:31:32,438 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:31:32,439 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:31:32,439 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:31:32,439 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:31:32,439 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 201 to run in 10 seconds 2017-10-23 16:31:37,458 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:31:37,459 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:31:37,459 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:31:37,459 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:31:42,447 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 201 2017-10-23 16:31:42,447 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:31:42,447 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:31:42,447 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:31:42,448 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:31:42,448 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:31:42,448 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:31:42,448 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 202 to run in 10 seconds 2017-10-23 16:31:51,489 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 195 2017-10-23 16:31:51,489 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:31:51,489 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:31:51,489 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 203 to run in 60 seconds 2017-10-23 16:31:51,490 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 25 2017-10-23 16:31:51,490 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:31:51,490 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 26 to run in 60 seconds 2017-10-23 16:31:52,038 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 196 2017-10-23 16:31:52,039 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 204 to run in 60 seconds 2017-10-23 16:31:52,453 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 202 2017-10-23 16:31:52,453 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:31:52,453 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:31:52,454 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:31:52,454 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:31:52,454 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:31:52,454 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:31:52,454 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 205 to run in 10 seconds 2017-10-23 16:31:52,467 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:31:52,468 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:31:52,468 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:31:52,468 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:32:02,456 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 205 2017-10-23 16:32:02,456 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:32:02,456 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:32:02,456 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:32:02,456 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:32:02,456 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:32:02,457 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:32:02,457 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 206 to run in 10 seconds 2017-10-23 16:32:07,475 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:32:07,475 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:32:07,475 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:32:07,476 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:32:12,462 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 206 2017-10-23 16:32:12,462 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:32:12,462 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:32:12,463 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:32:12,463 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:32:12,463 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:32:12,463 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:32:12,463 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 207 to run in 10 seconds 2017-10-23 16:32:22,468 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 207 2017-10-23 16:32:22,468 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:32:22,468 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:32:22,469 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:32:22,469 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:32:22,469 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:32:22,469 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:32:22,469 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 208 to run in 10 seconds 2017-10-23 16:32:22,481 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:32:22,482 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:32:22,482 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:32:22,482 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:32:32,470 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 208 2017-10-23 16:32:32,470 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:32:32,470 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:32:32,470 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:32:32,470 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:32:32,470 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:32:32,471 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:32:32,471 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 209 to run in 10 seconds 2017-10-23 16:32:37,484 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:32:37,487 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:32:37,487 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:32:37,488 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:32:42,475 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 209 2017-10-23 16:32:42,476 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:32:42,476 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:32:42,476 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:32:42,476 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:32:42,476 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:32:42,477 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:32:42,477 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 210 to run in 10 seconds 2017-10-23 16:32:51,492 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 203 2017-10-23 16:32:51,492 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:32:51,492 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:32:51,492 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 211 to run in 60 seconds 2017-10-23 16:32:51,493 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 26 2017-10-23 16:32:51,493 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:32:51,493 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 27 to run in 60 seconds 2017-10-23 16:32:52,039 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 204 2017-10-23 16:32:52,040 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 212 to run in 60 seconds 2017-10-23 16:32:52,486 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 210 2017-10-23 16:32:52,486 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:32:52,486 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:32:52,487 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:32:52,487 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:32:52,487 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:32:52,487 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:32:52,487 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 213 to run in 10 seconds 2017-10-23 16:32:52,488 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:32:52,488 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:32:52,488 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:32:52,488 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:33:02,492 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 213 2017-10-23 16:33:02,492 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:33:02,492 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:33:02,493 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:33:02,493 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:33:02,493 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:33:02,493 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:33:02,493 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 214 to run in 10 seconds 2017-10-23 16:33:07,488 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:33:07,489 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:33:07,489 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:33:07,490 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:33:12,498 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 214 2017-10-23 16:33:12,498 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:33:12,498 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:33:12,498 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:33:12,498 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:33:12,499 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:33:12,499 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:33:12,499 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 215 to run in 10 seconds 2017-10-23 16:33:22,496 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:33:22,497 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:33:22,497 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:33:22,497 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:33:22,509 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 215 2017-10-23 16:33:22,509 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:33:22,509 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:33:22,510 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:33:22,510 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:33:22,510 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:33:22,510 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:33:22,510 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 216 to run in 10 seconds 2017-10-23 16:33:32,511 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 216 2017-10-23 16:33:32,512 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:33:32,512 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:33:32,512 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:33:32,512 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:33:32,512 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:33:32,512 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:33:32,512 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 217 to run in 10 seconds 2017-10-23 16:33:37,501 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:33:37,501 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:33:37,501 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:33:37,502 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:33:42,523 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 217 2017-10-23 16:33:42,524 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:33:42,524 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:33:42,524 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:33:42,524 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:33:42,524 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:33:42,524 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:33:42,524 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 218 to run in 10 seconds 2017-10-23 16:33:51,500 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 211 2017-10-23 16:33:51,500 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:33:51,500 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:33:51,500 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 219 to run in 60 seconds 2017-10-23 16:33:51,500 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 27 2017-10-23 16:33:51,500 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:33:51,500 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 28 to run in 60 seconds 2017-10-23 16:33:52,045 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 212 2017-10-23 16:33:52,046 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 220 to run in 60 seconds 2017-10-23 16:33:52,502 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:33:52,502 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:33:52,502 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:33:52,503 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:33:52,526 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 218 2017-10-23 16:33:52,527 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:33:52,527 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:33:52,527 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:33:52,527 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:33:52,527 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:33:52,527 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:33:52,528 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 221 to run in 10 seconds 2017-10-23 16:34:02,530 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 221 2017-10-23 16:34:02,531 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:34:02,531 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:34:02,531 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:34:02,531 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:34:02,531 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:34:02,531 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:34:02,531 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 222 to run in 10 seconds 2017-10-23 16:34:07,503 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:34:07,503 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:34:07,503 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:34:07,503 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:34:12,533 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 222 2017-10-23 16:34:12,533 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:34:12,534 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:34:12,534 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:34:12,534 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:34:12,534 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:34:12,534 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:34:12,534 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 223 to run in 10 seconds 2017-10-23 16:34:22,508 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:34:22,509 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:34:22,509 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:34:22,509 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:34:22,545 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 223 2017-10-23 16:34:22,546 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:34:22,546 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:34:22,546 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:34:22,546 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:34:22,546 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:34:22,547 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:34:22,547 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 224 to run in 10 seconds 2017-10-23 16:34:32,557 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 224 2017-10-23 16:34:32,558 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:34:32,558 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:34:32,558 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:34:32,558 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:34:32,558 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:34:32,558 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:34:32,558 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 225 to run in 10 seconds 2017-10-23 16:34:37,510 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:34:37,510 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:34:37,510 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:34:37,511 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:34:42,561 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 225 2017-10-23 16:34:42,562 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:34:42,562 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:34:42,562 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:34:42,563 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:34:42,563 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:34:42,563 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:34:42,563 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 226 to run in 10 seconds 2017-10-23 16:34:51,507 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 219 2017-10-23 16:34:51,508 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:34:51,508 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:34:51,508 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 227 to run in 60 seconds 2017-10-23 16:34:51,508 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 28 2017-10-23 16:34:51,508 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:34:51,508 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 29 to run in 60 seconds 2017-10-23 16:34:52,047 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 220 2017-10-23 16:34:52,048 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 228 to run in 60 seconds 2017-10-23 16:34:52,514 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:34:52,514 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:34:52,514 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:34:52,515 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:34:52,566 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 226 2017-10-23 16:34:52,566 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:34:52,566 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:34:52,566 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:34:52,566 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:34:52,566 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:34:52,566 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:34:52,566 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 229 to run in 10 seconds 2017-10-23 16:35:02,567 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 229 2017-10-23 16:35:02,568 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:35:02,568 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:35:02,568 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:35:02,569 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:35:02,569 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:35:02,569 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:35:02,569 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 230 to run in 10 seconds 2017-10-23 16:35:07,523 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:35:07,523 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:35:07,523 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:35:07,524 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:35:12,580 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 230 2017-10-23 16:35:12,580 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:35:12,581 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:35:12,581 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:35:12,581 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:35:12,581 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:35:12,581 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:35:12,582 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 231 to run in 10 seconds 2017-10-23 16:35:22,523 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:35:22,524 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:35:22,524 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:35:22,524 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:35:22,589 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 231 2017-10-23 16:35:22,589 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:35:22,589 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:35:22,590 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:35:22,590 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:35:22,590 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:35:22,590 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:35:22,590 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 232 to run in 10 seconds 2017-10-23 16:35:32,601 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 232 2017-10-23 16:35:32,601 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:35:32,601 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:35:32,601 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:35:32,601 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:35:32,601 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:35:32,602 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:35:32,602 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 233 to run in 10 seconds 2017-10-23 16:35:37,535 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:35:37,535 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:35:37,535 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:35:37,536 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:35:42,611 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 233 2017-10-23 16:35:42,612 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:35:42,612 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:35:42,612 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:35:42,612 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:35:42,612 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:35:42,613 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:35:42,613 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 234 to run in 10 seconds 2017-10-23 16:35:51,511 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 227 2017-10-23 16:35:51,511 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:35:51,511 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:35:51,511 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 235 to run in 60 seconds 2017-10-23 16:35:51,511 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 29 2017-10-23 16:35:51,511 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:35:51,511 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 30 to run in 60 seconds 2017-10-23 16:35:52,054 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 228 2017-10-23 16:35:52,055 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 236 to run in 60 seconds 2017-10-23 16:35:52,545 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:35:52,545 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:35:52,545 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:35:52,546 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:35:52,624 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 234 2017-10-23 16:35:52,624 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:35:52,625 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:35:52,625 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:35:52,625 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:35:52,625 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:35:52,625 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:35:52,625 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 237 to run in 10 seconds 2017-10-23 16:36:02,630 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 237 2017-10-23 16:36:02,631 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:36:02,631 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:36:02,632 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:36:02,632 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:36:02,632 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:36:02,632 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:36:02,632 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 238 to run in 10 seconds 2017-10-23 16:36:07,549 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:36:07,550 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:36:07,550 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:36:07,550 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:36:12,638 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 238 2017-10-23 16:36:12,638 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:36:12,638 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:36:12,639 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:36:12,639 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:36:12,639 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:36:12,639 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:36:12,639 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 239 to run in 10 seconds 2017-10-23 16:36:22,556 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:36:22,556 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:36:22,557 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:36:22,558 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:36:22,650 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 239 2017-10-23 16:36:22,650 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:36:22,651 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:36:22,651 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:36:22,651 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:36:22,651 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:36:22,651 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:36:22,651 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 240 to run in 10 seconds 2017-10-23 16:36:32,658 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 240 2017-10-23 16:36:32,658 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:36:32,659 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:36:32,659 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:36:32,659 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:36:32,659 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:36:32,659 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:36:32,659 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 241 to run in 10 seconds 2017-10-23 16:36:37,561 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:36:37,561 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:36:37,561 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:36:37,561 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:36:42,662 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 241 2017-10-23 16:36:42,662 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:36:42,664 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:36:42,664 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:36:42,664 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:36:42,664 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:36:42,664 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:36:42,664 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 242 to run in 10 seconds 2017-10-23 16:36:51,518 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 235 2017-10-23 16:36:51,518 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:36:51,518 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:36:51,519 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 243 to run in 60 seconds 2017-10-23 16:36:51,519 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 30 2017-10-23 16:36:51,519 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:36:51,519 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 31 to run in 60 seconds 2017-10-23 16:36:52,058 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 236 2017-10-23 16:36:52,059 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 244 to run in 60 seconds 2017-10-23 16:36:52,567 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:36:52,567 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:36:52,568 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:36:52,568 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:36:52,668 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 242 2017-10-23 16:36:52,668 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:36:52,668 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:36:52,669 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:36:52,669 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:36:52,669 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:36:52,669 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:36:52,669 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 245 to run in 10 seconds 2017-10-23 16:37:02,670 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 245 2017-10-23 16:37:02,670 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:37:02,670 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:37:02,670 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:37:02,671 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:37:02,671 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:37:02,671 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:37:02,671 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 246 to run in 10 seconds 2017-10-23 16:37:07,578 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:37:07,578 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:37:07,579 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:37:07,579 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:37:12,679 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 246 2017-10-23 16:37:12,679 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:37:12,680 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:37:12,680 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:37:12,680 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:37:12,680 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:37:12,680 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:37:12,680 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 247 to run in 10 seconds 2017-10-23 16:37:22,583 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:37:22,583 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:37:22,583 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:37:22,583 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:37:22,686 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 247 2017-10-23 16:37:22,686 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:37:22,686 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:37:22,686 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:37:22,686 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:37:22,686 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:37:22,686 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:37:22,687 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 248 to run in 10 seconds 2017-10-23 16:37:32,695 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 248 2017-10-23 16:37:32,695 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:37:32,695 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:37:32,695 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:37:32,696 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:37:32,696 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:37:32,696 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:37:32,696 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 249 to run in 10 seconds 2017-10-23 16:37:37,590 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:37:37,590 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:37:37,590 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:37:37,591 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:37:42,699 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 249 2017-10-23 16:37:42,699 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:37:42,699 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:37:42,699 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:37:42,700 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:37:42,700 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:37:42,700 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:37:42,700 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 250 to run in 10 seconds 2017-10-23 16:37:51,523 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 243 2017-10-23 16:37:51,523 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:37:51,523 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:37:51,524 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 251 to run in 60 seconds 2017-10-23 16:37:51,524 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 31 2017-10-23 16:37:51,524 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:37:51,524 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 32 to run in 60 seconds 2017-10-23 16:37:52,060 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 244 2017-10-23 16:37:52,060 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 252 to run in 60 seconds 2017-10-23 16:37:52,597 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:37:52,597 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:37:52,597 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:37:52,598 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:37:52,710 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 250 2017-10-23 16:37:52,711 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:37:52,711 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:37:52,711 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:37:52,711 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:37:52,711 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:37:52,711 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:37:52,711 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 253 to run in 10 seconds 2017-10-23 16:38:02,721 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 253 2017-10-23 16:38:02,721 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:38:02,721 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:38:02,721 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:38:02,721 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:38:02,721 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:38:02,722 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:38:02,722 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 254 to run in 10 seconds 2017-10-23 16:38:07,611 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:38:07,612 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:38:07,612 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:38:07,612 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:38:12,733 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 254 2017-10-23 16:38:12,734 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:38:12,735 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:38:12,735 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:38:12,735 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:38:12,736 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:38:12,736 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:38:12,737 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 255 to run in 10 seconds 2017-10-23 16:38:22,617 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:38:22,617 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:38:22,617 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:38:22,618 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:38:22,741 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 255 2017-10-23 16:38:22,741 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:38:22,742 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:38:22,742 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:38:22,742 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:38:22,742 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:38:22,742 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:38:22,742 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 256 to run in 10 seconds 2017-10-23 16:38:32,747 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 256 2017-10-23 16:38:32,747 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:38:32,747 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:38:32,747 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:38:32,748 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:38:32,748 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:38:32,748 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:38:32,748 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 257 to run in 10 seconds 2017-10-23 16:38:37,623 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:38:37,623 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:38:37,623 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:38:37,624 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:38:42,756 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 257 2017-10-23 16:38:42,756 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:38:42,756 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:38:42,756 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:38:42,756 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:38:42,756 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:38:42,757 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:38:42,757 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 258 to run in 10 seconds 2017-10-23 16:38:51,534 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkNodeRequestSpike with id 251 2017-10-23 16:38:51,534 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node3 checking its request amount 2017-10-23 16:38:51,534 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:38:51,534 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkNodeRequestSpike with id 259 to run in 60 seconds 2017-10-23 16:38:51,534 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 32 2017-10-23 16:38:51,534 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-23 16:38:51,535 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 33 to run in 60 seconds 2017-10-23 16:38:52,062 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action dump_json_file with id 252 2017-10-23 16:38:52,063 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action dump_json_file with id 260 to run in 60 seconds 2017-10-23 16:38:52,625 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:38:52,625 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:38:52,625 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:38:52,625 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:38:52,757 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 258 2017-10-23 16:38:52,757 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:38:52,757 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:38:52,757 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:38:52,757 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:38:52,758 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:38:52,758 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:38:52,758 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 261 to run in 10 seconds 2017-10-23 16:39:02,759 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 261 2017-10-23 16:39:02,759 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:39:02,759 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:39:02,760 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:39:02,760 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:39:02,760 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:39:02,760 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:39:02,760 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 262 to run in 10 seconds 2017-10-23 16:39:07,629 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:39:07,629 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:39:07,630 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:39:07,630 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:39:12,770 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 262 2017-10-23 16:39:12,770 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:39:12,770 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:39:12,770 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:39:12,770 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:39:12,770 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:39:12,770 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:39:12,770 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 263 to run in 10 seconds 2017-10-23 16:39:22,632 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:39:22,632 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:39:22,632 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:39:22,633 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:39:22,781 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 263 2017-10-23 16:39:22,782 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:39:22,782 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:39:22,783 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:39:22,783 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:39:22,783 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:39:22,784 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:39:22,784 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 264 to run in 10 seconds 2017-10-23 16:39:32,785 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 264 2017-10-23 16:39:32,785 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:39:32,785 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:39:32,785 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:39:32,785 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:39:32,785 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:39:32,785 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:39:32,785 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 265 to run in 10 seconds 2017-10-23 16:39:37,632 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node2 HA(host='10.0.0.3', port=9703) 2017-10-23 16:39:37,633 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-23 16:39:37,633 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node3 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-23 16:39:37,633 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node3 next check for retries in 15.00 seconds 2017-10-23 16:39:42,795 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node3 running action checkPerformance with id 265 2017-10-23 16:39:42,795 | TRACE | node.py (1978) | checkPerformance | Node3 checking its performance 2017-10-23 16:39:42,795 | DEBUG | notifier_plugin_manager.py ( 80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0 2017-10-23 16:39:42,796 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node3 master throughput is not measurable. 2017-10-23 16:39:42,796 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node3 found master's latency to be lower than the threshold for all requests. 2017-10-23 16:39:42,796 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node3 found difference between master and backups avg latencies to be acceptable 2017-10-23 16:39:42,796 | DEBUG | node.py (1995) | checkPerformance | Node3's master has higher performance than backups 2017-10-23 16:39:42,796 | TRACE | has_action_queue.py ( 36) | _schedule | Node3 scheduling action checkPerformance with id 266 to run in 10 seconds 2017-10-23 16:39:43,265 | TRACE | zstack.py ( 479) | _receiveFromListener | Node3C got 1 messages through listener 2017-10-23 16:39:43,277 | DEBUG | node.py (2236) | verifySignature | Node3 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1508776783249274 2017-10-23 16:39:43,277 | TRACE | node.py (1388) | validateClientMsg | Node3C received CLIENT message: SafeRequest: {'reqId': 1508776783249274, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'timeout': 10, 'action': 'start', 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'type': '109', 'schedule': {'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00'}, 'version': '1.1.43', 'force': True, 'justification': None, 'name': 'upgrade-1143'}, 'signature': 'EejcE22puEWaVFYPWeEEm89YhBeBTvcwStJbP7Wsdenfxc9n5FCew1GFYwUnJsFSPW7YatjmJPYuamAYfagBQrN'} 2017-10-23 16:39:43,277 | DEBUG | node.py (1434) | processClientInBox | Node3C processing b'xo5JUY.S$6PWbpmz5XzA