2017-10-23 15:56:43,179 | INFO     | log.py               (  79) | setupRaet | Setting RAET log level 2
2017-10-23 15:56:43,190 | DEBUG    | start_sovrin_node    (  39) | <module> | You can find logs in /home/sovrin/.sovrin/Node4.log
2017-10-23 15:56:43,191 | DEBUG    | start_sovrin_node    (  42) | <module> | Sovrin related env vars: []
2017-10-23 15:56:44,383 | DEBUG    | __init__.py          (  60) | register | Registered VCS backend: git
2017-10-23 15:56:44,432 | DEBUG    | __init__.py          (  60) | register | Registered VCS backend: hg
2017-10-23 15:56:44,561 | DEBUG    | __init__.py          (  60) | register | Registered VCS backend: svn
2017-10-23 15:56:44,563 | DEBUG    | __init__.py          (  60) | register | Registered VCS backend: bzr
2017-10-23 15:56:45,081 | DEBUG    | selector_events.py   (  53) | __init__ | Using selector: EpollSelector
2017-10-23 15:56:45,083 | DEBUG    | looper.py            ( 123) | __init__ | Setting handler for SIGINT
2017-10-23 15:56:45,152 | DEBUG    | ledger.py            ( 206) | start | Starting ledger...
2017-10-23 15:56:45,152 | DEBUG    | file_store.py        ( 190) | appendNewLineIfReq | new line check for file: /home/sovrin/.sovrin/data/nodes/Node4/transactions_sandbox/1
2017-10-23 15:56:45,153 | DEBUG    | ledger.py            (  78) | recoverTree | Recovering tree from transaction log
2017-10-23 15:56:45,222 | DEBUG    | ledger.py            (  93) | recoverTree | Recovered tree in 0.06964874302502722 seconds
2017-10-23 15:56:45,270 | DEBUG    | idr_cache.py         (  25) | __init__ | Initializing identity cache Node4
2017-10-23 15:56:45,298 | INFO     | node.py              (2408) | initStateFromLedger | Node4 found state to be empty, recreating from ledger
2017-10-23 15:56:45,326 | DEBUG    | ledger.py            ( 206) | start | Starting ledger...
2017-10-23 15:56:45,327 | DEBUG    | file_store.py        ( 190) | appendNewLineIfReq | new line check for file: /home/sovrin/.sovrin/data/nodes/Node4/pool_transactions_sandbox/1
2017-10-23 15:56:45,327 | DEBUG    | ledger.py            (  78) | recoverTree | Recovering tree from transaction log
2017-10-23 15:56:45,386 | DEBUG    | ledger.py            (  93) | recoverTree | Recovered tree in 0.059028003946878016 seconds
2017-10-23 15:56:45,386 | INFO     | node.py              (2408) | initStateFromLedger | Node4 found state to be empty, recreating from ledger
2017-10-23 15:56:45,440 | DEBUG    | plugin_loader.py     (  95) | _load | skipping plugin plugin_firebase_stats_consumer[class: <class 'plenum.server.stats_consumer.StatsConsumer'>] because it does not have a 'pluginType' attribute
2017-10-23 15:56:45,441 | DEBUG    | plugin_loader.py     (  95) | _load | skipping plugin plugin_firebase_stats_consumer[class: <class 'plenum.server.plugin_loader.HasDynamicallyImportedModules'>] because it does not have a 'pluginType' attribute
2017-10-23 15:56:45,441 | DEBUG    | plugin_loader.py     (  95) | _load | skipping plugin plugin_firebase_stats_consumer[class: typing.Dict<~KT, ~VT>] because it does not have a 'pluginType' attribute
2017-10-23 15:56:45,441 | INFO     | plugin_loader.py     ( 116) | _load | plugin FirebaseStatsConsumer successfully loaded from module plugin_firebase_stats_consumer
2017-10-23 15:56:45,441 | DEBUG    | plugin_loader.py     (  95) | _load | skipping plugin plugin_firebase_stats_consumer[class: <class 'plenum.server.plugin.stats_consumer.stats_publisher.StatsPublisher'>] because it does not have a 'pluginType' attribute
2017-10-23 15:56:45,441 | DEBUG    | plugin_loader.py     (  95) | _load | skipping plugin plugin_firebase_stats_consumer[class: <enum 'Topic'>] because it does not have a 'pluginType' attribute
2017-10-23 15:56:45,441 | DEBUG    | has_action_queue.py  (  77) | startRepeating | checkPerformance will be repeating every 60 seconds
2017-10-23 15:56:45,441 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 1 to run in 60 seconds
2017-10-23 15:56:45,442 | INFO     | replica.py           ( 300) | h | Node4:0 set watermarks as 0 300
2017-10-23 15:56:45,442 | DISPLAY  | node.py              (1034) | addReplica | Node4 added replica Node4:0 to instance 0 (master)
2017-10-23 15:56:45,442 | INFO     | replica.py           ( 300) | h | Node4:1 set watermarks as 0 300
2017-10-23 15:56:45,442 | DISPLAY  | node.py              (1034) | addReplica | Node4 added replica Node4:1 to instance 1 (backup)
2017-10-23 15:56:45,442 | DEBUG    | has_action_queue.py  (  77) | startRepeating | checkPerformance will be repeating every 10 seconds
2017-10-23 15:56:45,442 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 1 to run in 10 seconds
2017-10-23 15:56:45,443 | DEBUG    | has_action_queue.py  (  77) | startRepeating | checkNodeRequestSpike will be repeating every 60 seconds
2017-10-23 15:56:45,443 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 2 to run in 60 seconds
2017-10-23 15:56:45,443 | DEBUG    | plugin_helper.py     (  24) | loadPlugins | Plugin loading started to load plugins from basedir: /home/sovrin/.sovrin
2017-10-23 15:56:45,443 | DEBUG    | plugin_helper.py     (  33) | loadPlugins | Plugin directory created at: /home/sovrin/.sovrin/plugins
2017-10-23 15:56:45,443 | DEBUG    | plugin_helper.py     (  67) | loadPlugins | Total plugins loaded from basedir /home/sovrin/.sovrin are : 0
2017-10-23 15:56:45,443 | DEBUG    | node.py              ( 325) | __init__ | total plugins loaded in node: 0
2017-10-23 15:56:45,474 | DEBUG    | ledger.py            ( 206) | start | Starting ledger...
2017-10-23 15:56:45,474 | DEBUG    | file_store.py        ( 190) | appendNewLineIfReq | new line check for file: /home/sovrin/.sovrin/data/nodes/Node4/config_transactions/1
2017-10-23 15:56:45,474 | DEBUG    | ledger.py            (  78) | recoverTree | Recovering tree from transaction log
2017-10-23 15:56:45,475 | DEBUG    | ledger.py            (  93) | recoverTree | Recovered tree in 0.0001921430230140686 seconds
2017-10-23 15:56:45,501 | INFO     | node.py              (2408) | initStateFromLedger | Node4 found state to be empty, recreating from ledger
2017-10-23 15:56:45,502 | DEBUG    | motor.py             (  34) | set_status | Node4 changing status from stopped to starting
2017-10-23 15:56:45,502 | DEBUG    | ledger.py            ( 204) | start | Ledger already started.
2017-10-23 15:56:45,502 | INFO     | zstack.py            ( 312) | start | Node4 starting with restricted as True and reSetupAuth as True
2017-10-23 15:56:45,502 | DEBUG    | authenticator.py     (  31) | start | Starting ZAP at inproc://zeromq.zap.1
2017-10-23 15:56:45,503 | DEBUG    | base.py              (  72) | allow | Allowing 0.0.0.0
2017-10-23 15:56:45,503 | DEBUG    | base.py              ( 112) | configure_curve | Configure curve: *[/home/sovrin/.sovrin/Node4/public_keys]
2017-10-23 15:56:45,503 | DEBUG    | zstack.py            ( 339) | open | Node4 will bind its listener at 9707
2017-10-23 15:56:45,503 | INFO     | stacks.py            (  76) | start | Node4 listening for other nodes at 0.0.0.0:9707
2017-10-23 15:56:45,503 | INFO     | zstack.py            ( 312) | start | Node4C starting with restricted as False and reSetupAuth as True
2017-10-23 15:56:45,504 | DEBUG    | authenticator.py     (  31) | start | Starting ZAP at inproc://zeromq.zap.2
2017-10-23 15:56:45,504 | DEBUG    | base.py              (  72) | allow | Allowing 0.0.0.0
2017-10-23 15:56:45,504 | DEBUG    | base.py              ( 112) | configure_curve | Configure curve: *[*]
2017-10-23 15:56:45,504 | DEBUG    | zstack.py            ( 339) | open | Node4C will bind its listener at 9708
2017-10-23 15:56:45,504 | INFO     | node.py              ( 594) | start | Node4 first time running...
2017-10-23 15:56:45,507 | DEBUG    | kit_zstack.py        (  96) | connectToMissing | Node4 found the following missing connections: Node2, Node3, Node1
2017-10-23 15:56:45,507 | TRACE    | remote.py            (  84) | connect | connecting socket 58 50848992 to remote Node2:HA(host='10.0.0.3', port=9703)
2017-10-23 15:56:45,508 | INFO     | zstack.py            ( 580) | connect | Node4 looking for Node2 at 10.0.0.3:9703
2017-10-23 15:56:45,508 | DEBUG    | zstack.py            ( 643) | sendPingPong | Node4 will be sending in batch
2017-10-23 15:56:45,509 | TRACE    | remote.py            (  84) | connect | connecting socket 59 50953040 to remote Node3:HA(host='10.0.0.4', port=9705)
2017-10-23 15:56:45,509 | INFO     | zstack.py            ( 580) | connect | Node4 looking for Node3 at 10.0.0.4:9705
2017-10-23 15:56:45,509 | DEBUG    | zstack.py            ( 643) | sendPingPong | Node4 will be sending in batch
2017-10-23 15:56:45,509 | TRACE    | remote.py            (  84) | connect | connecting socket 60 50990224 to remote Node1:HA(host='10.0.0.2', port=9701)
2017-10-23 15:56:45,510 | INFO     | zstack.py            ( 580) | connect | Node4 looking for Node1 at 10.0.0.2:9701
2017-10-23 15:56:45,510 | DEBUG    | zstack.py            ( 643) | sendPingPong | Node4 will be sending in batch
2017-10-23 15:56:45,510 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 15:56:45,529 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'pi' to Node2
2017-10-23 15:56:45,529 | TRACE    | batched.py           (  85) | flushOutBoxes | Node4 sending msg b'pi' to Node2
2017-10-23 15:56:45,529 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'pi' to Node3
2017-10-23 15:56:45,532 | TRACE    | batched.py           (  85) | flushOutBoxes | Node4 sending msg b'pi' to Node3
2017-10-23 15:56:45,532 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'pi' to Node1
2017-10-23 15:56:45,532 | TRACE    | batched.py           (  85) | flushOutBoxes | Node4 sending msg b'pi' to Node1
2017-10-23 15:56:45,585 | TRACE    | zstack.py            ( 472) | _receiveFromListener | Node4 got 5 messages through listener
2017-10-23 15:56:45,585 | DEBUG    | zstack.py            ( 652) | handlePingPong | Node4 got ping from Node1
2017-10-23 15:56:45,585 | DEBUG    | zstack.py            ( 643) | sendPingPong | Node4 will be sending in batch
2017-10-23 15:56:45,586 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node1: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}'], 'signature': None}
2017-10-23 15:56:45,586 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}'], 'signature': None}, 'Node1')
2017-10-23 15:56:45,586 | DEBUG    | node.py              (1328) | unpackNodeMsg | Node4 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}'], 'signature': None}
2017-10-23 15:56:45,593 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}
2017-10-23 15:56:45,593 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}}, 'Node1')
2017-10-23 15:56:45,594 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}
2017-10-23 15:56:45,594 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}
2017-10-23 15:56:45,594 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}}, 'Node1')
2017-10-23 15:56:45,594 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}
2017-10-23 15:56:45,594 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node1: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}'], 'signature': None}
2017-10-23 15:56:45,594 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}'], 'signature': None}, 'Node1')
2017-10-23 15:56:45,594 | DEBUG    | node.py              (1328) | unpackNodeMsg | Node4 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}'], 'signature': None}
2017-10-23 15:56:45,594 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}
2017-10-23 15:56:45,594 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}}, 'Node1')
2017-10-23 15:56:45,595 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}
2017-10-23 15:56:45,595 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}
2017-10-23 15:56:45,595 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}}, 'Node1')
2017-10-23 15:56:45,595 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}
2017-10-23 15:56:45,595 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node1: VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}
2017-10-23 15:56:45,595 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'op': 'VIEW_CHANGE_DONE', 'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}, 'Node1')
2017-10-23 15:56:45,595 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}
2017-10-23 15:56:45,596 | DEBUG    | zstack.py            ( 658) | handlePingPong | Node4 got pong from Node1
2017-10-23 15:56:45,596 | DEBUG    | node.py              (2593) | send | Node4 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None, 'txnSeqNo': 0}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node1']
2017-10-23 15:56:45,600 | DEBUG    | node.py              (2593) | send | Node4 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None, 'txnSeqNo': 0}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node1']
2017-10-23 15:56:45,600 | DEBUG    | node.py              (2593) | send | Node4 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'ledgerId': 1, 'viewNo': None, 'txnSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node1']
2017-10-23 15:56:45,600 | DEBUG    | node.py              (2593) | send | Node4 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'ledgerId': 1, 'viewNo': None, 'txnSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node1']
2017-10-23 15:56:45,601 | DEBUG    | node.py              (1261) | sendToElector | Node4 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}, 'Node1')
2017-10-23 15:56:45,602 | DEBUG    | batched.py           (  89) | flushOutBoxes | Node4 batching 5 msgs to Node1 into one transmission
2017-10-23 15:56:45,602 | TRACE    | batched.py           (  90) | flushOutBoxes |     messages: deque([b'po', b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', b'{"msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}'])
2017-10-23 15:56:45,602 | TRACE    | batched.py           (  98) | flushOutBoxes | Node4 sending payload to Node1: b'{"op":"BATCH","messages":["po","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":6,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":6,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}'
2017-10-23 15:56:45,602 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"op":"BATCH","messages":["po","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":6,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":6,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' to Node1
2017-10-23 15:56:45,603 | DEBUG    | keep_in_touch.py     (  68) | conns | Node4's connections changed from set() to {'Node1'}
2017-10-23 15:56:45,603 | INFO     | keep_in_touch.py     (  96) | _connsChanged | Node4 now connected to Node1
2017-10-23 15:56:45,603 | DEBUG    | node.py              (2593) | send | Node4 sending message LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'ledgerId': 0, 'viewNo': None, 'txnSeqNo': 4} to 1 recipients: ['Node1']
2017-10-23 15:56:45,604 | TRACE    | zstack.py            ( 472) | _receiveFromListener | Node4 got 10 messages through listener
2017-10-23 15:56:45,605 | DEBUG    | zstack.py            ( 652) | handlePingPong | Node4 got ping from Node2
2017-10-23 15:56:45,605 | DEBUG    | zstack.py            ( 643) | sendPingPong | Node4 will be sending in batch
2017-10-23 15:56:45,605 | DEBUG    | zstack.py            ( 652) | handlePingPong | Node4 got ping from Node3
2017-10-23 15:56:45,605 | DEBUG    | zstack.py            ( 643) | sendPingPong | Node4 will be sending in batch
2017-10-23 15:56:45,605 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node2: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}'], 'signature': None}
2017-10-23 15:56:45,605 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}'], 'signature': None}, 'Node2')
2017-10-23 15:56:45,605 | DEBUG    | node.py              (1328) | unpackNodeMsg | Node4 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}'], 'signature': None}
2017-10-23 15:56:45,606 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}
2017-10-23 15:56:45,606 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}}, 'Node2')
2017-10-23 15:56:45,606 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}
2017-10-23 15:56:45,606 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}
2017-10-23 15:56:45,606 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}}, 'Node2')
2017-10-23 15:56:45,606 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}
2017-10-23 15:56:45,606 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node3: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}'], 'signature': None}
2017-10-23 15:56:45,606 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}'], 'signature': None}, 'Node3')
2017-10-23 15:56:45,606 | DEBUG    | node.py              (1328) | unpackNodeMsg | Node4 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}'], 'signature': None}
2017-10-23 15:56:45,607 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}
2017-10-23 15:56:45,607 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}}, 'Node3')
2017-10-23 15:56:45,607 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}
2017-10-23 15:56:45,607 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}
2017-10-23 15:56:45,607 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}}, 'Node3')
2017-10-23 15:56:45,607 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}
2017-10-23 15:56:45,607 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node2: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}'], 'signature': None}
2017-10-23 15:56:45,607 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}'], 'signature': None}, 'Node2')
2017-10-23 15:56:45,607 | DEBUG    | node.py              (1328) | unpackNodeMsg | Node4 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}'], 'signature': None}
2017-10-23 15:56:45,607 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}
2017-10-23 15:56:45,608 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}}, 'Node2')
2017-10-23 15:56:45,608 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}
2017-10-23 15:56:45,608 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node2: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}
2017-10-23 15:56:45,608 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}}, 'Node2')
2017-10-23 15:56:45,608 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}
2017-10-23 15:56:45,608 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node3: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}'], 'signature': None}
2017-10-23 15:56:45,608 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}'], 'signature': None}, 'Node3')
2017-10-23 15:56:45,608 | DEBUG    | node.py              (1328) | unpackNodeMsg | Node4 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}'], 'signature': None}
2017-10-23 15:56:45,608 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}
2017-10-23 15:56:45,608 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}}, 'Node3')
2017-10-23 15:56:45,609 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}
2017-10-23 15:56:45,609 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}
2017-10-23 15:56:45,609 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}}, 'Node3')
2017-10-23 15:56:45,609 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}
2017-10-23 15:56:45,609 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node2: VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}
2017-10-23 15:56:45,609 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'op': 'VIEW_CHANGE_DONE', 'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}, 'Node2')
2017-10-23 15:56:45,609 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}
2017-10-23 15:56:45,616 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node3: VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}
2017-10-23 15:56:45,616 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'op': 'VIEW_CHANGE_DONE', 'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}, 'Node3')
2017-10-23 15:56:45,616 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}
2017-10-23 15:56:45,616 | DEBUG    | zstack.py            ( 658) | handlePingPong | Node4 got pong from Node2
2017-10-23 15:56:45,616 | DEBUG    | zstack.py            ( 658) | handlePingPong | Node4 got pong from Node3
2017-10-23 15:56:45,617 | DEBUG    | node.py              (2593) | send | Node4 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None, 'txnSeqNo': 0}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node2']
2017-10-23 15:56:45,617 | DEBUG    | node.py              (2593) | send | Node4 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None, 'txnSeqNo': 0}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node2']
2017-10-23 15:56:45,618 | DEBUG    | node.py              (2593) | send | Node4 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None, 'txnSeqNo': 0}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3']
2017-10-23 15:56:45,618 | DEBUG    | node.py              (2593) | send | Node4 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None, 'txnSeqNo': 0}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3']
2017-10-23 15:56:45,618 | DEBUG    | node.py              (2593) | send | Node4 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'ledgerId': 1, 'viewNo': None, 'txnSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node2']
2017-10-23 15:56:45,619 | DEBUG    | node.py              (2593) | send | Node4 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'ledgerId': 1, 'viewNo': None, 'txnSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node2']
2017-10-23 15:56:45,619 | DEBUG    | node.py              (2593) | send | Node4 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'ledgerId': 1, 'viewNo': None, 'txnSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3']
2017-10-23 15:56:45,620 | DEBUG    | node.py              (2593) | send | Node4 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'ledgerId': 1, 'viewNo': None, 'txnSeqNo': 6}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3']
2017-10-23 15:56:45,620 | DEBUG    | node.py              (1261) | sendToElector | Node4 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}, 'Node2')
2017-10-23 15:56:45,621 | DEBUG    | node.py              (1261) | sendToElector | Node4 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}, 'Node3')
2017-10-23 15:56:45,622 | DEBUG    | batched.py           (  89) | flushOutBoxes | Node4 batching 5 msgs to Node2 into one transmission
2017-10-23 15:56:45,622 | TRACE    | batched.py           (  90) | flushOutBoxes |     messages: deque([b'po', b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', b'{"msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}'])
2017-10-23 15:56:45,622 | TRACE    | batched.py           (  98) | flushOutBoxes | Node4 sending payload to Node2: b'{"op":"BATCH","messages":["po","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":6,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":6,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}'
2017-10-23 15:56:45,622 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"op":"BATCH","messages":["po","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":6,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":6,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' to Node2
2017-10-23 15:56:45,622 | DEBUG    | batched.py           (  89) | flushOutBoxes | Node4 batching 5 msgs to Node3 into one transmission
2017-10-23 15:56:45,622 | TRACE    | batched.py           (  90) | flushOutBoxes |     messages: deque([b'po', b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', b'{"msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}'])
2017-10-23 15:56:45,622 | TRACE    | batched.py           (  98) | flushOutBoxes | Node4 sending payload to Node3: b'{"op":"BATCH","messages":["po","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":6,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":6,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}'
2017-10-23 15:56:45,623 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"op":"BATCH","messages":["po","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":6,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":6,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null}' to Node3
2017-10-23 15:56:45,623 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"ppSeqNo":null,"ledgerId":0,"merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA","txnSeqNo":4,"op":"LEDGER_STATUS","viewNo":null}' to Node1
2017-10-23 15:56:45,623 | TRACE    | batched.py           (  85) | flushOutBoxes | Node4 sending msg b'{"ppSeqNo":null,"ledgerId":0,"merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA","txnSeqNo":4,"op":"LEDGER_STATUS","viewNo":null}' to Node1
2017-10-23 15:56:45,623 | DEBUG    | keep_in_touch.py     (  68) | conns | Node4's connections changed from {'Node1'} to {'Node2', 'Node3', 'Node1'}
2017-10-23 15:56:45,623 | INFO     | keep_in_touch.py     (  96) | _connsChanged | Node4 now connected to Node2
2017-10-23 15:56:45,623 | INFO     | keep_in_touch.py     (  96) | _connsChanged | Node4 now connected to Node3
2017-10-23 15:56:45,624 | DEBUG    | motor.py             (  34) | set_status | Node4 changing status from starting to started
2017-10-23 15:56:45,624 | DEBUG    | node.py              ( 918) | checkInstances | Node4 choosing to start election on the basis of count 4 and nodes {'Node2', 'Node3', 'Node1'}
2017-10-23 15:56:45,624 | DEBUG    | primary_selector.py  (  74) | get_msgs_for_lagged_nodes | Node4 has no ViewChangeDone message to send for view 0
2017-10-23 15:56:45,624 | DEBUG    | node.py              ( 879) | send_current_state_to_lagging_node | Node4 sending current state CURRENT_STATE{'viewNo': 0, 'primary': []} to lagged node Node2
2017-10-23 15:56:45,624 | DEBUG    | node.py              (2593) | send | Node4 sending message CURRENT_STATE{'viewNo': 0, 'primary': []} to 1 recipients: ['Node2']
2017-10-23 15:56:45,624 | DEBUG    | primary_selector.py  (  74) | get_msgs_for_lagged_nodes | Node4 has no ViewChangeDone message to send for view 0
2017-10-23 15:56:45,624 | DEBUG    | node.py              ( 879) | send_current_state_to_lagging_node | Node4 sending current state CURRENT_STATE{'viewNo': 0, 'primary': []} to lagged node Node3
2017-10-23 15:56:45,624 | DEBUG    | node.py              (2593) | send | Node4 sending message CURRENT_STATE{'viewNo': 0, 'primary': []} to 1 recipients: ['Node3']
2017-10-23 15:56:45,624 | DEBUG    | node.py              (2593) | send | Node4 sending message LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'ledgerId': 0, 'viewNo': None, 'txnSeqNo': 4} to 1 recipients: ['Node2']
2017-10-23 15:56:45,625 | DEBUG    | node.py              (2593) | send | Node4 sending message LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'ledgerId': 0, 'viewNo': None, 'txnSeqNo': 4} to 1 recipients: ['Node3']
2017-10-23 15:56:45,627 | DEBUG    | primary_selector.py  ( 103) | _processViewChangeDoneMessage | Node4's primary selector started processing of ViewChangeDone msg from Node1 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}
2017-10-23 15:56:45,627 | DEBUG    | primary_selector.py  ( 186) | _hasViewChangeQuorum | Node4 needs 1 ViewChangeDone messages
2017-10-23 15:56:45,628 | DEBUG    | primary_selector.py  ( 258) | _startSelection | Node4 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages
2017-10-23 15:56:45,628 | DEBUG    | primary_selector.py  ( 103) | _processViewChangeDoneMessage | Node4's primary selector started processing of ViewChangeDone msg from Node2 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}
2017-10-23 15:56:45,628 | INFO     | primary_selector.py  ( 192) | _hasViewChangeQuorum | Node4 got view change quorum (2 >= 2)
2017-10-23 15:56:45,629 | DEBUG    | primary_selector.py  ( 209) | has_view_change_from_primary | Node4 received ViewChangeDone from primary Node1
2017-10-23 15:56:45,630 | DEBUG    | primary_selector.py  ( 231) | has_sufficient_same_view_change_done_messages | Node4 found acceptable primary Node1 and ledger info ((0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'), (1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'))
2017-10-23 15:56:45,630 | INFO     | primary_selector.py  ( 263) | _startSelection | Node4 cannot start primary selection since mode is 100
2017-10-23 15:56:45,631 | DEBUG    | primary_selector.py  ( 103) | _processViewChangeDoneMessage | Node4's primary selector started processing of ViewChangeDone msg from Node3 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}
2017-10-23 15:56:45,631 | INFO     | primary_selector.py  ( 263) | _startSelection | Node4 cannot start primary selection since mode is 100
2017-10-23 15:56:45,632 | DEBUG    | batched.py           (  89) | flushOutBoxes | Node4 batching 2 msgs to Node2 into one transmission
2017-10-23 15:56:45,632 | TRACE    | batched.py           (  90) | flushOutBoxes |     messages: deque([b'{"op":"CURRENT_STATE","viewNo":0,"primary":[]}', b'{"ppSeqNo":null,"ledgerId":0,"merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA","txnSeqNo":4,"op":"LEDGER_STATUS","viewNo":null}'])
2017-10-23 15:56:45,632 | TRACE    | batched.py           (  98) | flushOutBoxes | Node4 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"viewNo\\":0,\\"primary\\":[]}","{\\"ppSeqNo\\":null,\\"ledgerId\\":0,\\"merkleRoot\\":\\"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA\\",\\"txnSeqNo\\":4,\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null}"],"signature":null}'
2017-10-23 15:56:45,632 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"viewNo\\":0,\\"primary\\":[]}","{\\"ppSeqNo\\":null,\\"ledgerId\\":0,\\"merkleRoot\\":\\"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA\\",\\"txnSeqNo\\":4,\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null}"],"signature":null}' to Node2
2017-10-23 15:56:45,632 | DEBUG    | batched.py           (  89) | flushOutBoxes | Node4 batching 2 msgs to Node3 into one transmission
2017-10-23 15:56:45,632 | TRACE    | batched.py           (  90) | flushOutBoxes |     messages: deque([b'{"op":"CURRENT_STATE","viewNo":0,"primary":[]}', b'{"ppSeqNo":null,"ledgerId":0,"merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA","txnSeqNo":4,"op":"LEDGER_STATUS","viewNo":null}'])
2017-10-23 15:56:45,632 | TRACE    | batched.py           (  98) | flushOutBoxes | Node4 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"viewNo\\":0,\\"primary\\":[]}","{\\"ppSeqNo\\":null,\\"ledgerId\\":0,\\"merkleRoot\\":\\"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA\\",\\"txnSeqNo\\":4,\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null}"],"signature":null}'
2017-10-23 15:56:45,632 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"op":"BATCH","messages":["{\\"op\\":\\"CURRENT_STATE\\",\\"viewNo\\":0,\\"primary\\":[]}","{\\"ppSeqNo\\":null,\\"ledgerId\\":0,\\"merkleRoot\\":\\"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA\\",\\"txnSeqNo\\":4,\\"op\\":\\"LEDGER_STATUS\\",\\"viewNo\\":null}"],"signature":null}' to Node3
2017-10-23 15:56:45,634 | TRACE    | zstack.py            ( 472) | _receiveFromListener | Node4 got 1 messages through listener
2017-10-23 15:56:45,634 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node1: BATCH{'messages': ['{"primary":[{"ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,6,"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0}],"op":"CURRENT_STATE","viewNo":0}', '{"op":"LEDGER_STATUS","ppSeqNo":null,"viewNo":null,"merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA","txnSeqNo":4,"ledgerId":0}', '{"op":"LEDGER_STATUS","ppSeqNo":null,"viewNo":null,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","txnSeqNo":6,"ledgerId":1}', '{"op":"LEDGER_STATUS","ppSeqNo":null,"viewNo":null,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","txnSeqNo":0,"ledgerId":2}'], 'signature': None}
2017-10-23 15:56:45,635 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'messages': ['{"primary":[{"ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,6,"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0}],"op":"CURRENT_STATE","viewNo":0}', '{"op":"LEDGER_STATUS","ppSeqNo":null,"viewNo":null,"merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA","txnSeqNo":4,"ledgerId":0}', '{"op":"LEDGER_STATUS","ppSeqNo":null,"viewNo":null,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","txnSeqNo":6,"ledgerId":1}', '{"op":"LEDGER_STATUS","ppSeqNo":null,"viewNo":null,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","txnSeqNo":0,"ledgerId":2}'], 'signature': None}, 'Node1')
2017-10-23 15:56:45,635 | DEBUG    | node.py              (1328) | unpackNodeMsg | Node4 processing a batch BATCH{'messages': ['{"primary":[{"ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,6,"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0}],"op":"CURRENT_STATE","viewNo":0}', '{"op":"LEDGER_STATUS","ppSeqNo":null,"viewNo":null,"merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA","txnSeqNo":4,"ledgerId":0}', '{"op":"LEDGER_STATUS","ppSeqNo":null,"viewNo":null,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","txnSeqNo":6,"ledgerId":1}', '{"op":"LEDGER_STATUS","ppSeqNo":null,"viewNo":null,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","txnSeqNo":0,"ledgerId":2}'], 'signature': None}
2017-10-23 15:56:45,635 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node1: CURRENT_STATE{'viewNo': 0, 'primary': [{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}]}
2017-10-23 15:56:45,635 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'op': 'CURRENT_STATE', 'primary': [{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}], 'viewNo': 0}, 'Node1')
2017-10-23 15:56:45,635 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox CURRENT_STATE{'viewNo': 0, 'primary': [{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}]}
2017-10-23 15:56:45,635 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node1: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'ledgerId': 0, 'viewNo': None, 'txnSeqNo': 4}
2017-10-23 15:56:45,636 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': None, 'ledgerId': 0, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'txnSeqNo': 4, 'op': 'LEDGER_STATUS', 'viewNo': None}, 'Node1')
2017-10-23 15:56:45,636 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'ledgerId': 0, 'viewNo': None, 'txnSeqNo': 4}
2017-10-23 15:56:45,636 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node1: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'ledgerId': 1, 'viewNo': None, 'txnSeqNo': 6}
2017-10-23 15:56:45,636 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': None, 'ledgerId': 1, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'txnSeqNo': 6, 'op': 'LEDGER_STATUS', 'viewNo': None}, 'Node1')
2017-10-23 15:56:45,636 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'ledgerId': 1, 'viewNo': None, 'txnSeqNo': 6}
2017-10-23 15:56:45,636 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node1: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None, 'txnSeqNo': 0}
2017-10-23 15:56:45,636 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'op': 'LEDGER_STATUS', 'viewNo': None}, 'Node1')
2017-10-23 15:56:45,636 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None, 'txnSeqNo': 0}
2017-10-23 15:56:45,637 | DEBUG    | node.py              ( 884) | process_current_state_message | Node4 processing current state CURRENT_STATE{'viewNo': 0, 'primary': [{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}]} from Node1
2017-10-23 15:56:45,637 | DEBUG    | node.py              (1261) | sendToElector | Node4 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}, 'Node1')
2017-10-23 15:56:45,637 | DEBUG    | ledger_manager.py    ( 244) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'ledgerId': 0, 'viewNo': None, 'txnSeqNo': 4} from Node1
2017-10-23 15:56:45,637 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 0 of size 4 with 4
2017-10-23 15:56:45,638 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 0 of size 4 with 4
2017-10-23 15:56:45,638 | DEBUG    | ledger_manager.py    ( 244) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'ledgerId': 1, 'viewNo': None, 'txnSeqNo': 6} from Node1
2017-10-23 15:56:45,638 | DEBUG    | ledger_manager.py    ( 997) | stashLedgerStatus | Node4 stashing ledger status LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'ledgerId': 1, 'viewNo': None, 'txnSeqNo': 6} from Node1
2017-10-23 15:56:45,638 | DEBUG    | ledger_manager.py    ( 244) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None, 'txnSeqNo': 0} from Node1
2017-10-23 15:56:45,639 | DEBUG    | ledger_manager.py    ( 997) | stashLedgerStatus | Node4 stashing ledger status LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None, 'txnSeqNo': 0} from Node1
2017-10-23 15:56:45,640 | DEBUG    | primary_selector.py  ( 103) | _processViewChangeDoneMessage | Node4's primary selector started processing of ViewChangeDone msg from Node1 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}
2017-10-23 15:56:45,641 | INFO     | primary_selector.py  ( 263) | _startSelection | Node4 cannot start primary selection since mode is 100
2017-10-23 15:56:45,655 | TRACE    | zstack.py            ( 472) | _receiveFromListener | Node4 got 1 messages through listener
2017-10-23 15:56:45,656 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node3: BATCH{'messages': ['{"primary":[{"ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,6,"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0}],"viewNo":0,"op":"CURRENT_STATE"}', '{"ledgerId":0,"merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA","ppSeqNo":null,"op":"LEDGER_STATUS","viewNo":null,"txnSeqNo":4}', '{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"op":"LEDGER_STATUS","viewNo":null,"txnSeqNo":6}', '{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"op":"LEDGER_STATUS","viewNo":null,"txnSeqNo":0}'], 'signature': None}
2017-10-23 15:56:45,656 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'messages': ['{"primary":[{"ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,6,"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0}],"viewNo":0,"op":"CURRENT_STATE"}', '{"ledgerId":0,"merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA","ppSeqNo":null,"op":"LEDGER_STATUS","viewNo":null,"txnSeqNo":4}', '{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"op":"LEDGER_STATUS","viewNo":null,"txnSeqNo":6}', '{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"op":"LEDGER_STATUS","viewNo":null,"txnSeqNo":0}'], 'signature': None}, 'Node3')
2017-10-23 15:56:45,656 | DEBUG    | node.py              (1328) | unpackNodeMsg | Node4 processing a batch BATCH{'messages': ['{"primary":[{"ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,6,"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0}],"viewNo":0,"op":"CURRENT_STATE"}', '{"ledgerId":0,"merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA","ppSeqNo":null,"op":"LEDGER_STATUS","viewNo":null,"txnSeqNo":4}', '{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"op":"LEDGER_STATUS","viewNo":null,"txnSeqNo":6}', '{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"op":"LEDGER_STATUS","viewNo":null,"txnSeqNo":0}'], 'signature': None}
2017-10-23 15:56:45,656 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node3: CURRENT_STATE{'viewNo': 0, 'primary': [{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}]}
2017-10-23 15:56:45,656 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'op': 'CURRENT_STATE', 'primary': [{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}], 'viewNo': 0}, 'Node3')
2017-10-23 15:56:45,656 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox CURRENT_STATE{'viewNo': 0, 'primary': [{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}]}
2017-10-23 15:56:45,657 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node3: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'ledgerId': 0, 'viewNo': None, 'txnSeqNo': 4}
2017-10-23 15:56:45,657 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': None, 'txnSeqNo': 4, 'ledgerId': 0, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'op': 'LEDGER_STATUS', 'viewNo': None}, 'Node3')
2017-10-23 15:56:45,657 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'ledgerId': 0, 'viewNo': None, 'txnSeqNo': 4}
2017-10-23 15:56:45,657 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node3: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'ledgerId': 1, 'viewNo': None, 'txnSeqNo': 6}
2017-10-23 15:56:45,657 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': None, 'txnSeqNo': 6, 'ledgerId': 1, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'op': 'LEDGER_STATUS', 'viewNo': None}, 'Node3')
2017-10-23 15:56:45,657 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'ledgerId': 1, 'viewNo': None, 'txnSeqNo': 6}
2017-10-23 15:56:45,657 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node3: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None, 'txnSeqNo': 0}
2017-10-23 15:56:45,658 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'op': 'LEDGER_STATUS', 'viewNo': None}, 'Node3')
2017-10-23 15:56:45,658 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None, 'txnSeqNo': 0}
2017-10-23 15:56:45,658 | DEBUG    | node.py              ( 884) | process_current_state_message | Node4 processing current state CURRENT_STATE{'viewNo': 0, 'primary': [{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}]} from Node3
2017-10-23 15:56:45,658 | DEBUG    | node.py              (1261) | sendToElector | Node4 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}, 'Node3')
2017-10-23 15:56:45,659 | DEBUG    | ledger_manager.py    ( 244) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'ledgerId': 0, 'viewNo': None, 'txnSeqNo': 4} from Node3
2017-10-23 15:56:45,659 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 0 of size 4 with 4
2017-10-23 15:56:45,659 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 0 of size 4 with 4
2017-10-23 15:56:45,659 | DEBUG    | ledger_manager.py    ( 309) | processLedgerStatus | Node4 found out from {'Node3', 'Node1'} that its ledger of type 0 is latest
2017-10-23 15:56:45,659 | DEBUG    | ledger_manager.py    ( 312) | processLedgerStatus | Node4 found from ledger status LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'ledgerId': 0, 'viewNo': None, 'txnSeqNo': 4} that it does not need catchup
2017-10-23 15:56:45,659 | DEBUG    | node.py              (1537) | preLedgerCatchUp | Node4 going to process any ordered requests before starting catchup.
2017-10-23 15:56:45,659 | DEBUG    | replica.py           (2130) | _remove_ordered_from_queue | Node4:0 going to remove 0 Ordered messages from outbox
2017-10-23 15:56:45,659 | DEBUG    | node.py              (1918) | force_process_ordered | Node4 processed 0 Ordered batches for instance 0 before starting catch up
2017-10-23 15:56:45,659 | DEBUG    | replica.py           (2130) | _remove_ordered_from_queue | Node4:1 going to remove 0 Ordered messages from outbox
2017-10-23 15:56:45,659 | DEBUG    | node.py              (1918) | force_process_ordered | Node4 processed 0 Ordered batches for instance 1 before starting catch up
2017-10-23 15:56:45,660 | DEBUG    | node.py              (2451) | processStashedOrderedReqs | Node4 processed 0 stashed ordered requests
2017-10-23 15:56:45,660 | DEBUG    | monitor.py           ( 183) | reset | Monitor being reset
2017-10-23 15:56:45,660 | DEBUG    | node.py              (1547) | preLedgerCatchUp | Node4 reverted 0 batches before starting catch up for ledger 0
2017-10-23 15:56:45,660 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 0 of size 4 with 4
2017-10-23 15:56:45,660 | DEBUG    | node.py              ( 918) | checkInstances | Node4 choosing to start election on the basis of count 4 and nodes {'Node2', 'Node3', 'Node1'}
2017-10-23 15:56:45,660 | DEBUG    | node.py              (2593) | send | Node4 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node1']
2017-10-23 15:56:45,660 | DEBUG    | node.py              ( 832) | _ask_for_ledger_status | Node4 asking Node1 for ledger status of ledger 2
2017-10-23 15:56:45,660 | DEBUG    | node.py              (2593) | send | Node4 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node2']
2017-10-23 15:56:45,660 | DEBUG    | node.py              ( 832) | _ask_for_ledger_status | Node4 asking Node2 for ledger status of ledger 2
2017-10-23 15:56:45,661 | DEBUG    | node.py              (2593) | send | Node4 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3']
2017-10-23 15:56:45,661 | DEBUG    | node.py              ( 832) | _ask_for_ledger_status | Node4 asking Node3 for ledger status of ledger 2
2017-10-23 15:56:45,661 | DEBUG    | node.py              (2593) | send | Node4 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to all recipients: ['Node2', 'Node3', 'Node1']
2017-10-23 15:56:45,661 | DEBUG    | node.py              ( 832) | _ask_for_ledger_status | Node4 asking Node4 for ledger status of ledger 2
2017-10-23 15:56:45,661 | DEBUG    | ledger_manager.py    (1006) | processStashedLedgerStatuses | Node4 going to process 1 stashed ledger statuses for ledger 2
2017-10-23 15:56:45,661 | DEBUG    | ledger_manager.py    ( 244) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None, 'txnSeqNo': 0} from Node1
2017-10-23 15:56:45,661 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 2 of size 0 with 0
2017-10-23 15:56:45,661 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 2 of size 0 with 0
2017-10-23 15:56:45,661 | INFO     | ledger_manager.py    ( 831) | catchupCompleted | Node4 completed catching up ledger 0, caught up 0 in total
2017-10-23 15:56:45,662 | DEBUG    | ledger_manager.py    ( 244) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'ledgerId': 1, 'viewNo': None, 'txnSeqNo': 6} from Node3
2017-10-23 15:56:45,662 | DEBUG    | ledger_manager.py    ( 997) | stashLedgerStatus | Node4 stashing ledger status LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'ledgerId': 1, 'viewNo': None, 'txnSeqNo': 6} from Node3
2017-10-23 15:56:45,662 | DEBUG    | ledger_manager.py    ( 244) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None, 'txnSeqNo': 0} from Node3
2017-10-23 15:56:45,662 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 2 of size 0 with 0
2017-10-23 15:56:45,662 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 2 of size 0 with 0
2017-10-23 15:56:45,663 | DEBUG    | ledger_manager.py    ( 309) | processLedgerStatus | Node4 found out from {'Node3', 'Node1'} that its ledger of type 2 is latest
2017-10-23 15:56:45,663 | DEBUG    | ledger_manager.py    ( 312) | processLedgerStatus | Node4 found from ledger status LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None, 'txnSeqNo': 0} that it does not need catchup
2017-10-23 15:56:45,663 | DEBUG    | node.py              (1537) | preLedgerCatchUp | Node4 going to process any ordered requests before starting catchup.
2017-10-23 15:56:45,663 | DEBUG    | replica.py           (2130) | _remove_ordered_from_queue | Node4:0 going to remove 0 Ordered messages from outbox
2017-10-23 15:56:45,663 | DEBUG    | node.py              (1918) | force_process_ordered | Node4 processed 0 Ordered batches for instance 0 before starting catch up
2017-10-23 15:56:45,663 | DEBUG    | replica.py           (2130) | _remove_ordered_from_queue | Node4:1 going to remove 0 Ordered messages from outbox
2017-10-23 15:56:45,663 | DEBUG    | node.py              (1918) | force_process_ordered | Node4 processed 0 Ordered batches for instance 1 before starting catch up
2017-10-23 15:56:45,663 | DEBUG    | node.py              (2451) | processStashedOrderedReqs | Node4 processed 0 stashed ordered requests
2017-10-23 15:56:45,663 | DEBUG    | monitor.py           ( 183) | reset | Monitor being reset
2017-10-23 15:56:45,663 | DEBUG    | node.py              (1547) | preLedgerCatchUp | Node4 reverted 0 batches before starting catch up for ledger 2
2017-10-23 15:56:45,666 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 2 of size 0 with 0
2017-10-23 15:56:45,666 | INFO     | pool_config.py       (  33) | processLedger | <sovrin_node.server.pool_config.PoolConfig object at 0x7f677995d4e0> processing config ledger for any POOL_CONFIGs
2017-10-23 15:56:45,666 | INFO     | upgrader.py          ( 145) | processLedger | 4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA processing config ledger for any upgrades
2017-10-23 15:56:45,667 | DEBUG    | node.py              (2593) | send | Node4 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node1']
2017-10-23 15:56:45,667 | DEBUG    | node.py              ( 832) | _ask_for_ledger_status | Node4 asking Node1 for ledger status of ledger 1
2017-10-23 15:56:45,667 | DEBUG    | node.py              (2593) | send | Node4 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node2']
2017-10-23 15:56:45,667 | DEBUG    | node.py              ( 832) | _ask_for_ledger_status | Node4 asking Node2 for ledger status of ledger 1
2017-10-23 15:56:45,667 | DEBUG    | node.py              (2593) | send | Node4 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3']
2017-10-23 15:56:45,667 | DEBUG    | node.py              ( 832) | _ask_for_ledger_status | Node4 asking Node3 for ledger status of ledger 1
2017-10-23 15:56:45,667 | DEBUG    | node.py              (2593) | send | Node4 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to all recipients: ['Node2', 'Node3', 'Node1']
2017-10-23 15:56:45,667 | DEBUG    | node.py              ( 832) | _ask_for_ledger_status | Node4 asking Node4 for ledger status of ledger 1
2017-10-23 15:56:45,667 | DEBUG    | ledger_manager.py    (1006) | processStashedLedgerStatuses | Node4 going to process 2 stashed ledger statuses for ledger 1
2017-10-23 15:56:45,668 | DEBUG    | ledger_manager.py    ( 244) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'ledgerId': 1, 'viewNo': None, 'txnSeqNo': 6} from Node1
2017-10-23 15:56:45,668 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 1 of size 6 with 6
2017-10-23 15:56:45,668 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 1 of size 6 with 6
2017-10-23 15:56:45,668 | DEBUG    | ledger_manager.py    ( 244) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'ledgerId': 1, 'viewNo': None, 'txnSeqNo': 6} from Node3
2017-10-23 15:56:45,668 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 1 of size 6 with 6
2017-10-23 15:56:45,668 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 1 of size 6 with 6
2017-10-23 15:56:45,668 | DEBUG    | ledger_manager.py    ( 309) | processLedgerStatus | Node4 found out from {'Node3', 'Node1'} that its ledger of type 1 is latest
2017-10-23 15:56:45,668 | DEBUG    | ledger_manager.py    ( 312) | processLedgerStatus | Node4 found from ledger status LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'ledgerId': 1, 'viewNo': None, 'txnSeqNo': 6} that it does not need catchup
2017-10-23 15:56:45,668 | DEBUG    | node.py              (1537) | preLedgerCatchUp | Node4 going to process any ordered requests before starting catchup.
2017-10-23 15:56:45,669 | DEBUG    | replica.py           (2130) | _remove_ordered_from_queue | Node4:0 going to remove 0 Ordered messages from outbox
2017-10-23 15:56:45,669 | DEBUG    | node.py              (1918) | force_process_ordered | Node4 processed 0 Ordered batches for instance 0 before starting catch up
2017-10-23 15:56:45,669 | DEBUG    | replica.py           (2130) | _remove_ordered_from_queue | Node4:1 going to remove 0 Ordered messages from outbox
2017-10-23 15:56:45,669 | DEBUG    | node.py              (1918) | force_process_ordered | Node4 processed 0 Ordered batches for instance 1 before starting catch up
2017-10-23 15:56:45,669 | DEBUG    | node.py              (2451) | processStashedOrderedReqs | Node4 processed 0 stashed ordered requests
2017-10-23 15:56:45,669 | DEBUG    | monitor.py           ( 183) | reset | Monitor being reset
2017-10-23 15:56:45,669 | DEBUG    | node.py              (1547) | preLedgerCatchUp | Node4 reverted 0 batches before starting catch up for ledger 1
2017-10-23 15:56:45,669 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 1 of size 6 with 6
2017-10-23 15:56:45,669 | INFO     | ledger_manager.py    ( 831) | catchupCompleted | Node4 completed catching up ledger 1, caught up 0 in total
2017-10-23 15:56:45,669 | DEBUG    | node.py              (1653) | num_txns_caught_up_in_last_catchup | Node4 caught up to 0 txns in the last catchup
2017-10-23 15:56:45,670 | DEBUG    | node.py              (2451) | processStashedOrderedReqs | Node4 processed 0 stashed ordered requests
2017-10-23 15:56:45,670 | DEBUG    | monitor.py           ( 183) | reset | Monitor being reset
2017-10-23 15:56:45,670 | INFO     | primary_selector.py  ( 192) | _hasViewChangeQuorum | Node4 got view change quorum (3 >= 2)
2017-10-23 15:56:45,670 | INFO     | node.py              (1605) | is_catchup_needed | Node4 is caught up for the current view 0
2017-10-23 15:56:45,670 | INFO     | node.py              (1593) | allLedgersCaughtUp | Node4 does not need any more catchups
2017-10-23 15:56:45,671 | DEBUG    | primary_decider.py   ( 131) | send | Node4's elector sending VIEW_CHANGE_DONE{'ledgerInfo': [(0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'), (1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'viewNo': 0, 'name': 'Node1'}
2017-10-23 15:56:45,672 | DEBUG    | primary_selector.py  ( 272) | _startSelection | Node4 starting selection
2017-10-23 15:56:45,673 | DISPLAY  | primary_selector.py  ( 284) | _startSelection | Node4:0 selected primary Node1:0 for instance 0 (view 0)
2017-10-23 15:56:45,673 | INFO     | node.py              ( 482) | start_participating | Node4 started participating
2017-10-23 15:56:45,673 | INFO     | replica.py           ( 393) | primaryName | Node4:0 setting primaryName for view no 0 to: Node1:0
2017-10-23 15:56:45,673 | DEBUG    | replica.py           (1625) | _gc | Node4:0 cleaning up till (0, 0)
2017-10-23 15:56:45,673 | DEBUG    | replica.py           (1640) | _gc | Node4:0 found 0 3-phase keys to clean
2017-10-23 15:56:45,673 | DEBUG    | replica.py           (1642) | _gc | Node4:0 found 0 request keys to clean
2017-10-23 15:56:45,673 | INFO     | replica.py           ( 300) | h | Node4:0 set watermarks as 0 300
2017-10-23 15:56:45,673 | DISPLAY  | primary_selector.py  ( 307) | _startSelection | Node4:0 declares view change 0 as completed for instance 0, new primary is Node1:0, ledger info is [(0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'), (1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')]
2017-10-23 15:56:45,681 | DISPLAY  | primary_selector.py  ( 284) | _startSelection | Node4:1 selected primary Node2:1 for instance 1 (view 0)
2017-10-23 15:56:45,681 | INFO     | replica.py           ( 393) | primaryName | Node4:1 setting primaryName for view no 0 to: Node2:1
2017-10-23 15:56:45,681 | DEBUG    | replica.py           (1625) | _gc | Node4:1 cleaning up till (0, 0)
2017-10-23 15:56:45,681 | DEBUG    | replica.py           (1640) | _gc | Node4:1 found 0 3-phase keys to clean
2017-10-23 15:56:45,681 | DEBUG    | replica.py           (1642) | _gc | Node4:1 found 0 request keys to clean
2017-10-23 15:56:45,681 | INFO     | replica.py           ( 300) | h | Node4:1 set watermarks as 0 300
2017-10-23 15:56:45,681 | DEBUG    | replica.py           ( 494) | _setup_for_non_master | Node4:1 Setting last ordered for non-master as (0, 0)
2017-10-23 15:56:45,681 | DEBUG    | replica.py           ( 310) | last_ordered_3pc | Node4:1 set last ordered as (0, 0)
2017-10-23 15:56:45,682 | DISPLAY  | primary_selector.py  ( 307) | _startSelection | Node4:1 declares view change 0 as completed for instance 1, new primary is Node2:1, ledger info is [(0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'), (1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')]
2017-10-23 15:56:45,682 | INFO     | ledger_manager.py    ( 831) | catchupCompleted | Node4 completed catching up ledger 2, caught up 0 in total
2017-10-23 15:56:45,682 | DEBUG    | node.py              (1653) | num_txns_caught_up_in_last_catchup | Node4 caught up to 0 txns in the last catchup
2017-10-23 15:56:45,682 | DEBUG    | node.py              (2451) | processStashedOrderedReqs | Node4 processed 0 stashed ordered requests
2017-10-23 15:56:45,682 | DEBUG    | monitor.py           ( 183) | reset | Monitor being reset
2017-10-23 15:56:45,682 | INFO     | primary_selector.py  ( 192) | _hasViewChangeQuorum | Node4 got view change quorum (4 >= 2)
2017-10-23 15:56:45,682 | INFO     | node.py              (1605) | is_catchup_needed | Node4 is caught up for the current view 0
2017-10-23 15:56:45,682 | INFO     | node.py              (1593) | allLedgersCaughtUp | Node4 does not need any more catchups
2017-10-23 15:56:45,684 | DEBUG    | primary_selector.py  ( 272) | _startSelection | Node4 starting selection
2017-10-23 15:56:45,684 | DEBUG    | primary_selector.py  ( 275) | _startSelection | Node4:0 already has a primary
2017-10-23 15:56:45,684 | DEBUG    | primary_selector.py  ( 275) | _startSelection | Node4:1 already has a primary
2017-10-23 15:56:45,684 | DEBUG    | node.py              (1667) | no_more_catchups_needed | Node4 starting to participate since catchup is done, primaries are selected but mode was not set to participating
2017-10-23 15:56:45,684 | INFO     | node.py              ( 482) | start_participating | Node4 started participating
2017-10-23 15:56:45,686 | DEBUG    | node.py              (2593) | send | Node4 sending message VIEW_CHANGE_DONE{'ledgerInfo': [(0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'), (1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'viewNo': 0, 'name': 'Node1'} to all recipients: ['Node2', 'Node3', 'Node1']
2017-10-23 15:56:45,686 | DEBUG    | primary_selector.py  ( 103) | _processViewChangeDoneMessage | Node4's primary selector started processing of ViewChangeDone msg from Node3 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}
2017-10-23 15:56:45,686 | DEBUG    | message_processor.py (  28) | discard | Node4 discarding message VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'} because it already decided primary which is Node1:0
2017-10-23 15:56:45,686 | DEBUG    | batched.py           (  89) | flushOutBoxes | Node4 batching 5 msgs to Node2 into one transmission
2017-10-23 15:56:45,687 | TRACE    | batched.py           (  90) | flushOutBoxes |     messages: deque([b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', b'{"op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,6,"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":0,"name":"Node1"}'])
2017-10-23 15:56:45,687 | TRACE    | batched.py           (  98) | flushOutBoxes | Node4 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"VIEW_CHANGE_DONE\\",\\"ledgerInfo\\":[[0,4,\\"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA\\"],[1,6,\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"viewNo\\":0,\\"name\\":\\"Node1\\"}"],"signature":null}'
2017-10-23 15:56:45,687 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"op":"BATCH","messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"VIEW_CHANGE_DONE\\",\\"ledgerInfo\\":[[0,4,\\"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA\\"],[1,6,\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"viewNo\\":0,\\"name\\":\\"Node1\\"}"],"signature":null}' to Node2
2017-10-23 15:56:45,690 | DEBUG    | batched.py           (  89) | flushOutBoxes | Node4 batching 5 msgs to Node3 into one transmission
2017-10-23 15:56:45,691 | TRACE    | batched.py           (  90) | flushOutBoxes |     messages: deque([b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', b'{"op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,6,"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":0,"name":"Node1"}'])
2017-10-23 15:56:45,691 | TRACE    | batched.py           (  98) | flushOutBoxes | Node4 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"VIEW_CHANGE_DONE\\",\\"ledgerInfo\\":[[0,4,\\"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA\\"],[1,6,\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"viewNo\\":0,\\"name\\":\\"Node1\\"}"],"signature":null}'
2017-10-23 15:56:45,698 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"op":"BATCH","messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"VIEW_CHANGE_DONE\\",\\"ledgerInfo\\":[[0,4,\\"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA\\"],[1,6,\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"viewNo\\":0,\\"name\\":\\"Node1\\"}"],"signature":null}' to Node3
2017-10-23 15:56:45,698 | DEBUG    | batched.py           (  89) | flushOutBoxes | Node4 batching 5 msgs to Node1 into one transmission
2017-10-23 15:56:45,699 | TRACE    | batched.py           (  90) | flushOutBoxes |     messages: deque([b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', b'{"op":"VIEW_CHANGE_DONE","ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,6,"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"viewNo":0,"name":"Node1"}'])
2017-10-23 15:56:45,699 | TRACE    | batched.py           (  98) | flushOutBoxes | Node4 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"VIEW_CHANGE_DONE\\",\\"ledgerInfo\\":[[0,4,\\"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA\\"],[1,6,\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"viewNo\\":0,\\"name\\":\\"Node1\\"}"],"signature":null}'
2017-10-23 15:56:45,699 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"op":"BATCH","messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"op\\":\\"VIEW_CHANGE_DONE\\",\\"ledgerInfo\\":[[0,4,\\"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA\\"],[1,6,\\"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"viewNo\\":0,\\"name\\":\\"Node1\\"}"],"signature":null}' to Node1
2017-10-23 15:56:45,704 | TRACE    | zstack.py            ( 472) | _receiveFromListener | Node4 got 2 messages through listener
2017-10-23 15:56:45,704 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node2: BATCH{'messages': ['{"viewNo":0,"primary":[{"ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,6,"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0}],"op":"CURRENT_STATE"}', '{"txnSeqNo":4,"viewNo":null,"ppSeqNo":null,"ledgerId":0,"op":"LEDGER_STATUS","merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"}', '{"txnSeqNo":6,"viewNo":null,"ppSeqNo":null,"ledgerId":1,"op":"LEDGER_STATUS","merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4"}', '{"txnSeqNo":0,"viewNo":null,"ppSeqNo":null,"ledgerId":2,"op":"LEDGER_STATUS","merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"}'], 'signature': None}
2017-10-23 15:56:45,704 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'messages': ['{"viewNo":0,"primary":[{"ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,6,"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0}],"op":"CURRENT_STATE"}', '{"txnSeqNo":4,"viewNo":null,"ppSeqNo":null,"ledgerId":0,"op":"LEDGER_STATUS","merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"}', '{"txnSeqNo":6,"viewNo":null,"ppSeqNo":null,"ledgerId":1,"op":"LEDGER_STATUS","merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4"}', '{"txnSeqNo":0,"viewNo":null,"ppSeqNo":null,"ledgerId":2,"op":"LEDGER_STATUS","merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"}'], 'signature': None}, 'Node2')
2017-10-23 15:56:45,705 | DEBUG    | node.py              (1328) | unpackNodeMsg | Node4 processing a batch BATCH{'messages': ['{"viewNo":0,"primary":[{"ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,6,"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0}],"op":"CURRENT_STATE"}', '{"txnSeqNo":4,"viewNo":null,"ppSeqNo":null,"ledgerId":0,"op":"LEDGER_STATUS","merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"}', '{"txnSeqNo":6,"viewNo":null,"ppSeqNo":null,"ledgerId":1,"op":"LEDGER_STATUS","merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4"}', '{"txnSeqNo":0,"viewNo":null,"ppSeqNo":null,"ledgerId":2,"op":"LEDGER_STATUS","merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"}'], 'signature': None}
2017-10-23 15:56:45,705 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node2: CURRENT_STATE{'viewNo': 0, 'primary': [{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}]}
2017-10-23 15:56:45,705 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'op': 'CURRENT_STATE', 'viewNo': 0, 'primary': [{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}]}, 'Node2')
2017-10-23 15:56:45,705 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox CURRENT_STATE{'viewNo': 0, 'primary': [{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}]}
2017-10-23 15:56:45,705 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node2: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'ledgerId': 0, 'viewNo': None, 'txnSeqNo': 4}
2017-10-23 15:56:45,705 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': None, 'ledgerId': 0, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'txnSeqNo': 4, 'op': 'LEDGER_STATUS', 'viewNo': None}, 'Node2')
2017-10-23 15:56:45,705 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'ledgerId': 0, 'viewNo': None, 'txnSeqNo': 4}
2017-10-23 15:56:45,706 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node2: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'ledgerId': 1, 'viewNo': None, 'txnSeqNo': 6}
2017-10-23 15:56:45,706 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': None, 'ledgerId': 1, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'txnSeqNo': 6, 'op': 'LEDGER_STATUS', 'viewNo': None}, 'Node2')
2017-10-23 15:56:45,706 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'ledgerId': 1, 'viewNo': None, 'txnSeqNo': 6}
2017-10-23 15:56:45,711 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node2: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None, 'txnSeqNo': 0}
2017-10-23 15:56:45,711 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': None, 'ledgerId': 2, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'op': 'LEDGER_STATUS', 'viewNo': None}, 'Node2')
2017-10-23 15:56:45,711 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None, 'txnSeqNo': 0}
2017-10-23 15:56:45,711 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node3: BATCH{'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}'], 'signature': None}
2017-10-23 15:56:45,711 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}'], 'signature': None}, 'Node3')
2017-10-23 15:56:45,711 | DEBUG    | node.py              (1328) | unpackNodeMsg | Node4 processing a batch BATCH{'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}'], 'signature': None}
2017-10-23 15:56:45,711 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node3: MESSAGE_RESPONSE{'msg': {'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}
2017-10-23 15:56:45,712 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'msg': {'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}}, 'Node3')
2017-10-23 15:56:45,712 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}
2017-10-23 15:56:45,712 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node3: MESSAGE_RESPONSE{'msg': {'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}
2017-10-23 15:56:45,712 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'msg': {'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}}, 'Node3')
2017-10-23 15:56:45,712 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}
2017-10-23 15:56:45,712 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node3: MESSAGE_RESPONSE{'msg': {'ppSeqNo': None, 'txnSeqNo': 6, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}
2017-10-23 15:56:45,712 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'msg': {'ppSeqNo': None, 'txnSeqNo': 6, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}}, 'Node3')
2017-10-23 15:56:45,712 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'ppSeqNo': None, 'txnSeqNo': 6, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}
2017-10-23 15:56:45,713 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node3: MESSAGE_RESPONSE{'msg': {'ppSeqNo': None, 'txnSeqNo': 6, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}
2017-10-23 15:56:45,713 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'msg': {'ppSeqNo': None, 'txnSeqNo': 6, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}}, 'Node3')
2017-10-23 15:56:45,713 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'ppSeqNo': None, 'txnSeqNo': 6, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}
2017-10-23 15:56:45,713 | DEBUG    | node.py              ( 884) | process_current_state_message | Node4 processing current state CURRENT_STATE{'viewNo': 0, 'primary': [{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}]} from Node2
2017-10-23 15:56:45,713 | DEBUG    | node.py              (1261) | sendToElector | Node4 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}, 'Node2')
2017-10-23 15:56:45,714 | DEBUG    | ledger_manager.py    ( 244) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'ledgerId': 0, 'viewNo': None, 'txnSeqNo': 4} from Node2
2017-10-23 15:56:45,714 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 0 of size 4 with 4
2017-10-23 15:56:45,714 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 0 of size 4 with 4
2017-10-23 15:56:45,714 | DEBUG    | ledger_manager.py    ( 244) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'ledgerId': 1, 'viewNo': None, 'txnSeqNo': 6} from Node2
2017-10-23 15:56:45,714 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 1 of size 6 with 6
2017-10-23 15:56:45,715 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 1 of size 6 with 6
2017-10-23 15:56:45,715 | DEBUG    | ledger_manager.py    ( 244) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None, 'txnSeqNo': 0} from Node2
2017-10-23 15:56:45,715 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 2 of size 0 with 0
2017-10-23 15:56:45,715 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 2 of size 0 with 0
2017-10-23 15:56:45,716 | DEBUG    | ledger_manager.py    ( 244) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None, 'txnSeqNo': 0} from Node3
2017-10-23 15:56:45,716 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 2 of size 0 with 0
2017-10-23 15:56:45,716 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 2 of size 0 with 0
2017-10-23 15:56:45,716 | DEBUG    | ledger_manager.py    ( 309) | processLedgerStatus | Node4 found out from {'Node2', 'Node3'} that its ledger of type 2 is latest
2017-10-23 15:56:45,716 | DEBUG    | ledger_manager.py    ( 244) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None, 'txnSeqNo': 0} from Node3
2017-10-23 15:56:45,716 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 2 of size 0 with 0
2017-10-23 15:56:45,716 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 2 of size 0 with 0
2017-10-23 15:56:45,717 | DEBUG    | ledger_manager.py    ( 309) | processLedgerStatus | Node4 found out from {'Node2', 'Node3'} that its ledger of type 2 is latest
2017-10-23 15:56:45,717 | DEBUG    | ledger_manager.py    ( 244) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'ledgerId': 1, 'viewNo': None, 'txnSeqNo': 6} from Node3
2017-10-23 15:56:45,717 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 1 of size 6 with 6
2017-10-23 15:56:45,717 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 1 of size 6 with 6
2017-10-23 15:56:45,717 | DEBUG    | ledger_manager.py    ( 309) | processLedgerStatus | Node4 found out from {'Node2', 'Node3'} that its ledger of type 1 is latest
2017-10-23 15:56:45,718 | DEBUG    | ledger_manager.py    ( 244) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'ledgerId': 1, 'viewNo': None, 'txnSeqNo': 6} from Node3
2017-10-23 15:56:45,718 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 1 of size 6 with 6
2017-10-23 15:56:45,718 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 1 of size 6 with 6
2017-10-23 15:56:45,718 | DEBUG    | ledger_manager.py    ( 309) | processLedgerStatus | Node4 found out from {'Node2', 'Node3'} that its ledger of type 1 is latest
2017-10-23 15:56:45,719 | DEBUG    | primary_selector.py  ( 103) | _processViewChangeDoneMessage | Node4's primary selector started processing of ViewChangeDone msg from Node2 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'}
2017-10-23 15:56:45,719 | DEBUG    | message_processor.py (  28) | discard | Node4 discarding message VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 6, 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'viewNo': 0, 'name': 'Node1'} because it already decided primary which is Node1:0
2017-10-23 15:56:45,722 | TRACE    | zstack.py            ( 472) | _receiveFromListener | Node4 got 2 messages through listener
2017-10-23 15:56:45,723 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node2: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}'], 'signature': None}
2017-10-23 15:56:45,723 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}'], 'signature': None}, 'Node2')
2017-10-23 15:56:45,723 | DEBUG    | node.py              (1328) | unpackNodeMsg | Node4 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}'], 'signature': None}
2017-10-23 15:56:45,723 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node2: MESSAGE_RESPONSE{'msg': {'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}
2017-10-23 15:56:45,723 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'msg': {'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}}, 'Node2')
2017-10-23 15:56:45,723 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}
2017-10-23 15:56:45,723 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node2: MESSAGE_RESPONSE{'msg': {'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}
2017-10-23 15:56:45,724 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'msg': {'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}}, 'Node2')
2017-10-23 15:56:45,724 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}
2017-10-23 15:56:45,724 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node2: MESSAGE_RESPONSE{'msg': {'ppSeqNo': None, 'txnSeqNo': 6, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}
2017-10-23 15:56:45,724 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'msg': {'ppSeqNo': None, 'txnSeqNo': 6, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}}, 'Node2')
2017-10-23 15:56:45,724 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'ppSeqNo': None, 'txnSeqNo': 6, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}
2017-10-23 15:56:45,724 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node2: MESSAGE_RESPONSE{'msg': {'ppSeqNo': None, 'txnSeqNo': 6, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}
2017-10-23 15:56:45,724 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'msg': {'ppSeqNo': None, 'txnSeqNo': 6, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}}, 'Node2')
2017-10-23 15:56:45,724 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'ppSeqNo': None, 'txnSeqNo': 6, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}
2017-10-23 15:56:45,725 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node1: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}'], 'signature': None}
2017-10-23 15:56:45,725 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}'], 'signature': None}, 'Node1')
2017-10-23 15:56:45,725 | DEBUG    | node.py              (1328) | unpackNodeMsg | Node4 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4","ppSeqNo":null,"txnSeqNo":6,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}'], 'signature': None}
2017-10-23 15:56:45,725 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node1: MESSAGE_RESPONSE{'msg': {'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}
2017-10-23 15:56:45,725 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'msg': {'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}}, 'Node1')
2017-10-23 15:56:45,725 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}
2017-10-23 15:56:45,725 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node1: MESSAGE_RESPONSE{'msg': {'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}
2017-10-23 15:56:45,725 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'msg': {'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}}, 'Node1')
2017-10-23 15:56:45,725 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'ppSeqNo': None, 'txnSeqNo': 0, 'ledgerId': 2, 'viewNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}}
2017-10-23 15:56:45,726 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node1: MESSAGE_RESPONSE{'msg': {'ppSeqNo': None, 'txnSeqNo': 6, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}
2017-10-23 15:56:45,726 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'msg': {'ppSeqNo': None, 'txnSeqNo': 6, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}}, 'Node1')
2017-10-23 15:56:45,726 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'ppSeqNo': None, 'txnSeqNo': 6, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}
2017-10-23 15:56:45,726 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node1: MESSAGE_RESPONSE{'msg': {'ppSeqNo': None, 'txnSeqNo': 6, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}
2017-10-23 15:56:45,726 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'msg': {'ppSeqNo': None, 'txnSeqNo': 6, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}}, 'Node1')
2017-10-23 15:56:45,726 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'ppSeqNo': None, 'txnSeqNo': 6, 'ledgerId': 1, 'viewNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4'}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}}
2017-10-23 15:56:45,726 | DEBUG    | ledger_manager.py    ( 244) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None, 'txnSeqNo': 0} from Node2
2017-10-23 15:56:45,727 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 2 of size 0 with 0
2017-10-23 15:56:45,727 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 2 of size 0 with 0
2017-10-23 15:56:45,727 | DEBUG    | ledger_manager.py    ( 309) | processLedgerStatus | Node4 found out from {'Node2', 'Node3'} that its ledger of type 2 is latest
2017-10-23 15:56:45,727 | DEBUG    | ledger_manager.py    ( 244) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None, 'txnSeqNo': 0} from Node2
2017-10-23 15:56:45,728 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 2 of size 0 with 0
2017-10-23 15:56:45,730 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 2 of size 0 with 0
2017-10-23 15:56:45,731 | DEBUG    | ledger_manager.py    ( 309) | processLedgerStatus | Node4 found out from {'Node2', 'Node3'} that its ledger of type 2 is latest
2017-10-23 15:56:45,731 | DEBUG    | ledger_manager.py    ( 244) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'ledgerId': 1, 'viewNo': None, 'txnSeqNo': 6} from Node2
2017-10-23 15:56:45,731 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 1 of size 6 with 6
2017-10-23 15:56:45,731 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 1 of size 6 with 6
2017-10-23 15:56:45,731 | DEBUG    | ledger_manager.py    ( 309) | processLedgerStatus | Node4 found out from {'Node2', 'Node3'} that its ledger of type 1 is latest
2017-10-23 15:56:45,732 | DEBUG    | ledger_manager.py    ( 244) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'ledgerId': 1, 'viewNo': None, 'txnSeqNo': 6} from Node2
2017-10-23 15:56:45,732 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 1 of size 6 with 6
2017-10-23 15:56:45,732 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 1 of size 6 with 6
2017-10-23 15:56:45,732 | DEBUG    | ledger_manager.py    ( 309) | processLedgerStatus | Node4 found out from {'Node2', 'Node3'} that its ledger of type 1 is latest
2017-10-23 15:56:45,732 | DEBUG    | ledger_manager.py    ( 244) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None, 'txnSeqNo': 0} from Node1
2017-10-23 15:56:45,732 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 2 of size 0 with 0
2017-10-23 15:56:45,732 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 2 of size 0 with 0
2017-10-23 15:56:45,733 | DEBUG    | ledger_manager.py    ( 309) | processLedgerStatus | Node4 found out from {'Node2', 'Node3', 'Node1'} that its ledger of type 2 is latest
2017-10-23 15:56:45,733 | DEBUG    | ledger_manager.py    ( 244) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'ledgerId': 2, 'viewNo': None, 'txnSeqNo': 0} from Node1
2017-10-23 15:56:45,733 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 2 of size 0 with 0
2017-10-23 15:56:45,733 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 2 of size 0 with 0
2017-10-23 15:56:45,733 | DEBUG    | ledger_manager.py    ( 309) | processLedgerStatus | Node4 found out from {'Node2', 'Node3', 'Node1'} that its ledger of type 2 is latest
2017-10-23 15:56:45,733 | DEBUG    | ledger_manager.py    ( 244) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'ledgerId': 1, 'viewNo': None, 'txnSeqNo': 6} from Node1
2017-10-23 15:56:45,734 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 1 of size 6 with 6
2017-10-23 15:56:45,734 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 1 of size 6 with 6
2017-10-23 15:56:45,734 | DEBUG    | ledger_manager.py    ( 309) | processLedgerStatus | Node4 found out from {'Node2', 'Node3', 'Node1'} that its ledger of type 1 is latest
2017-10-23 15:56:45,734 | DEBUG    | ledger_manager.py    ( 244) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': 'HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4', 'ledgerId': 1, 'viewNo': None, 'txnSeqNo': 6} from Node1
2017-10-23 15:56:45,734 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 1 of size 6 with 6
2017-10-23 15:56:45,735 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 1 of size 6 with 6
2017-10-23 15:56:45,735 | DEBUG    | ledger_manager.py    ( 309) | processLedgerStatus | Node4 found out from {'Node2', 'Node3', 'Node1'} that its ledger of type 1 is latest
2017-10-23 15:56:55,444 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 1
2017-10-23 15:56:55,444 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 15:56:55,444 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 15:56:55,444 | DEBUG    | monitor.py           ( 290) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 15:56:55,445 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 15:56:55,445 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 15:56:55,445 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 15:56:55,445 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 3 to run in 10 seconds
2017-10-23 15:57:00,518 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 15:57:00,519 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 15:57:00,519 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 15:57:00,519 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 15:57:05,457 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 3
2017-10-23 15:57:05,458 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 15:57:05,458 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 15:57:05,458 | DEBUG    | monitor.py           ( 290) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 15:57:05,458 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 15:57:05,458 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 15:57:05,458 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 15:57:05,458 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 4 to run in 10 seconds
2017-10-23 15:57:15,469 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 4
2017-10-23 15:57:15,469 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 15:57:15,469 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 15:57:15,469 | DEBUG    | monitor.py           ( 290) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 15:57:15,469 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 15:57:15,469 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 15:57:15,469 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 15:57:15,470 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 5 to run in 10 seconds
2017-10-23 15:57:15,528 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 15:57:15,528 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 15:57:15,528 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 15:57:15,528 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 15:57:25,473 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 5
2017-10-23 15:57:25,474 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 15:57:25,474 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 15:57:25,474 | DEBUG    | monitor.py           ( 290) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 15:57:25,474 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 15:57:25,474 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 15:57:25,474 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 15:57:25,474 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 6 to run in 10 seconds
2017-10-23 15:57:30,539 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 15:57:30,539 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 15:57:30,539 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 15:57:30,539 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 15:57:35,493 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 6
2017-10-23 15:57:35,494 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 15:57:35,494 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 15:57:35,494 | DEBUG    | monitor.py           ( 290) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 15:57:35,494 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 15:57:35,494 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 15:57:35,494 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 15:57:35,494 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 7 to run in 10 seconds
2017-10-23 15:57:45,449 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkNodeRequestSpike with id 2
2017-10-23 15:57:45,449 | DEBUG    | node.py              (2026) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 15:57:45,449 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 15:57:45,449 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 8 to run in 60 seconds
2017-10-23 15:57:45,449 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 1
2017-10-23 15:57:45,449 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 15:57:45,449 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 2 to run in 60 seconds
2017-10-23 15:57:45,496 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 7
2017-10-23 15:57:45,496 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 15:57:45,496 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 15:57:45,496 | DEBUG    | monitor.py           ( 290) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 15:57:45,496 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 15:57:45,496 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 15:57:45,497 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 15:57:45,497 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 9 to run in 10 seconds
2017-10-23 15:57:45,541 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 15:57:45,541 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 15:57:45,541 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 15:57:45,542 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 15:57:55,499 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 9
2017-10-23 15:57:55,499 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 15:57:55,499 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 15:57:55,500 | DEBUG    | monitor.py           ( 290) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 15:57:55,500 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 15:57:55,500 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 15:57:55,500 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 15:57:55,500 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 10 to run in 10 seconds
2017-10-23 15:58:00,551 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 15:58:00,551 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 15:58:00,551 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 15:58:00,552 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 15:58:05,509 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 10
2017-10-23 15:58:05,509 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 15:58:05,509 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 15:58:05,509 | DEBUG    | monitor.py           ( 290) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 15:58:05,509 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 15:58:05,509 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 15:58:05,510 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 15:58:05,510 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 11 to run in 10 seconds
2017-10-23 15:58:15,522 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 11
2017-10-23 15:58:15,522 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 15:58:15,522 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 15:58:15,523 | DEBUG    | monitor.py           ( 290) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 15:58:15,523 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 15:58:15,523 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 15:58:15,523 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 15:58:15,523 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 12 to run in 10 seconds
2017-10-23 15:58:15,560 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 15:58:15,560 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 15:58:15,560 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 15:58:15,561 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 15:58:25,525 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 12
2017-10-23 15:58:25,525 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 15:58:25,525 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 15:58:25,525 | DEBUG    | monitor.py           ( 290) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 15:58:25,525 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 15:58:25,525 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 15:58:25,525 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 15:58:25,525 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 13 to run in 10 seconds
2017-10-23 15:58:30,574 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 15:58:30,575 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 15:58:30,575 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 15:58:30,575 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 15:58:35,539 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 13
2017-10-23 15:58:35,539 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 15:58:35,539 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 15:58:35,539 | DEBUG    | monitor.py           ( 290) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 15:58:35,539 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 15:58:35,539 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 15:58:35,539 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 15:58:35,539 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 14 to run in 10 seconds
2017-10-23 15:58:45,451 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkNodeRequestSpike with id 8
2017-10-23 15:58:45,452 | DEBUG    | node.py              (2026) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 15:58:45,452 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 15:58:45,452 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 15 to run in 60 seconds
2017-10-23 15:58:45,452 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 2
2017-10-23 15:58:45,452 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 15:58:45,452 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 3 to run in 60 seconds
2017-10-23 15:58:45,552 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 14
2017-10-23 15:58:45,553 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 15:58:45,553 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 15:58:45,553 | DEBUG    | monitor.py           ( 290) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 15:58:45,553 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 15:58:45,553 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 15:58:45,553 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 15:58:45,553 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 16 to run in 10 seconds
2017-10-23 15:58:45,585 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 15:58:45,585 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 15:58:45,585 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 15:58:45,585 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 15:58:55,561 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 16
2017-10-23 15:58:55,561 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 15:58:55,561 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 15:58:55,561 | DEBUG    | monitor.py           ( 290) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 15:58:55,561 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 15:58:55,562 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 15:58:55,562 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 15:58:55,562 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 17 to run in 10 seconds
2017-10-23 15:59:00,596 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 15:59:00,597 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 15:59:00,597 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 15:59:00,597 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 15:59:05,574 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 17
2017-10-23 15:59:05,575 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 15:59:05,575 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 15:59:05,575 | DEBUG    | monitor.py           ( 290) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 15:59:05,575 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 15:59:05,575 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 15:59:05,575 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 15:59:05,575 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 18 to run in 10 seconds
2017-10-23 15:59:15,586 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 18
2017-10-23 15:59:15,586 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 15:59:15,586 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 15:59:15,586 | DEBUG    | monitor.py           ( 290) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 15:59:15,586 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 15:59:15,586 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 15:59:15,586 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 15:59:15,586 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 19 to run in 10 seconds
2017-10-23 15:59:15,603 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 15:59:15,603 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 15:59:15,603 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 15:59:15,603 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 15:59:25,593 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 19
2017-10-23 15:59:25,593 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 15:59:25,593 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 15:59:25,593 | DEBUG    | monitor.py           ( 290) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 15:59:25,593 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 15:59:25,593 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 15:59:25,593 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 15:59:25,593 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 20 to run in 10 seconds
2017-10-23 15:59:30,605 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 15:59:30,606 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 15:59:30,606 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 15:59:30,606 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 15:59:35,604 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 20
2017-10-23 15:59:35,604 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 15:59:35,604 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 15:59:35,604 | DEBUG    | monitor.py           ( 290) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 15:59:35,604 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 15:59:35,604 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 15:59:35,604 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 15:59:35,604 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 21 to run in 10 seconds
2017-10-23 15:59:41,656 | TRACE    | zstack.py            ( 472) | _receiveFromListener | Node4C got 1 messages through listener
2017-10-23 15:59:41,656 | DEBUG    | zstack.py            ( 652) | handlePingPong | Node4C got ping from b'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*'
2017-10-23 15:59:41,656 | TRACE    | zstack.py            ( 726) | transmitThroughListener | Node4C transmitting b'po' to b'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*' through listener socket
2017-10-23 15:59:41,656 | DEBUG    | zstack.py            ( 635) | sendPingPong | Node4C ponged b'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*'
2017-10-23 15:59:41,677 | TRACE    | zstack.py            ( 472) | _receiveFromListener | Node4C got 1 messages through listener
2017-10-23 15:59:41,677 | TRACE    | node.py              (1444) | validateClientMsg | Node4C received CLIENT message: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'ledgerId': 0, 'viewNo': None, 'txnSeqNo': 4}
2017-10-23 15:59:41,678 | DISPLAY  | node.py              (1490) | processClientInBox | Node4C processing b'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*' request LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'ledgerId': 0, 'viewNo': None, 'txnSeqNo': 4}
2017-10-23 15:59:41,678 | DEBUG    | ledger_manager.py    ( 244) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'ppSeqNo': None, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'ledgerId': 0, 'viewNo': None, 'txnSeqNo': 4} from b'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*'
2017-10-23 15:59:41,678 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 0 of size 4 with 4
2017-10-23 15:59:41,678 | TRACE    | zstack.py            ( 726) | transmitThroughListener | Node4C transmitting b'{"ppSeqNo":null,"ledgerId":0,"merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA","txnSeqNo":4,"op":"LEDGER_STATUS","viewNo":null}' to b'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*' through listener socket
2017-10-23 15:59:41,679 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 0 of size 4 with 4
2017-10-23 15:59:41,679 | DEBUG    | ledger_manager.py    ( 965) | _compareLedger | Node4 comparing its ledger 0 of size 4 with 4
2017-10-23 15:59:45,457 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkNodeRequestSpike with id 15
2017-10-23 15:59:45,458 | DEBUG    | node.py              (2026) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 15:59:45,458 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 15:59:45,458 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 22 to run in 60 seconds
2017-10-23 15:59:45,458 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 3
2017-10-23 15:59:45,458 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 15:59:45,458 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 4 to run in 60 seconds
2017-10-23 15:59:45,612 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 21
2017-10-23 15:59:45,612 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 15:59:45,612 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 15:59:45,612 | DEBUG    | monitor.py           ( 290) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 15:59:45,612 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 15:59:45,612 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 15:59:45,612 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 15:59:45,613 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 23 to run in 10 seconds
2017-10-23 15:59:45,613 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 15:59:45,614 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 15:59:45,614 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 15:59:45,614 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 15:59:55,618 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 23
2017-10-23 15:59:55,618 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 15:59:55,618 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 15:59:55,618 | DEBUG    | monitor.py           ( 290) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 15:59:55,618 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 15:59:55,619 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 15:59:55,619 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 15:59:55,619 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 24 to run in 10 seconds
2017-10-23 16:00:00,620 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:00:00,621 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:00:00,621 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:00:00,621 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:00:05,620 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 24
2017-10-23 16:00:05,621 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:00:05,621 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:00:05,621 | DEBUG    | monitor.py           ( 290) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:00:05,621 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:00:05,621 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:00:05,621 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:00:05,621 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 25 to run in 10 seconds
2017-10-23 16:00:15,628 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 25
2017-10-23 16:00:15,628 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:00:15,628 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:00:15,628 | DEBUG    | monitor.py           ( 290) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:00:15,628 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:00:15,628 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:00:15,628 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:00:15,628 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 26 to run in 10 seconds
2017-10-23 16:00:15,629 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:00:15,629 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:00:15,629 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:00:15,630 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:00:25,637 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 26
2017-10-23 16:00:25,637 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:00:25,637 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:00:25,638 | DEBUG    | monitor.py           ( 290) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:00:25,638 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:00:25,638 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:00:25,638 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:00:25,638 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 27 to run in 10 seconds
2017-10-23 16:00:30,637 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:00:30,638 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:00:30,638 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:00:30,638 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:00:35,649 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 27
2017-10-23 16:00:35,649 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:00:35,650 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:00:35,650 | DEBUG    | monitor.py           ( 290) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:00:35,650 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:00:35,650 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:00:35,650 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:00:35,650 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 28 to run in 10 seconds
2017-10-23 16:00:45,465 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkNodeRequestSpike with id 22
2017-10-23 16:00:45,466 | DEBUG    | node.py              (2026) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:00:45,466 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:00:45,466 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 29 to run in 60 seconds
2017-10-23 16:00:45,466 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 4
2017-10-23 16:00:45,466 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:00:45,466 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 5 to run in 60 seconds
2017-10-23 16:00:45,648 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:00:45,648 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:00:45,648 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:00:45,648 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:00:45,661 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 28
2017-10-23 16:00:45,662 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:00:45,662 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:00:45,662 | DEBUG    | monitor.py           ( 290) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:00:45,662 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:00:45,662 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:00:45,662 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:00:45,662 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 30 to run in 10 seconds
2017-10-23 16:00:55,671 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 30
2017-10-23 16:00:55,671 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:00:55,671 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:00:55,671 | DEBUG    | monitor.py           ( 290) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:00:55,671 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:00:55,671 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:00:55,672 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:00:55,672 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 31 to run in 10 seconds
2017-10-23 16:00:55,916 | TRACE    | zstack.py            ( 472) | _receiveFromListener | Node4 got 1 messages through listener
2017-10-23 16:00:55,917 | TRACE    | signing.py           (  98) | serializeMsg | serialized msg {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'reqId': 1508774455875574, 'signature': '2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:dest:V4SGRU86Z58d6TV7PBUe61|type:1|reqId:1508774455875574
2017-10-23 16:00:55,918 | INFO     | node.py              (2250) | verifySignature | Node4 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate  request 1508774455875574
2017-10-23 16:00:55,918 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node3: PROPAGATE{'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*', 'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'reqId': 1508774455875574, 'signature': '2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}
2017-10-23 16:00:55,918 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'op': 'PROPAGATE', 'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*', 'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'reqId': 1508774455875574, 'signature': '2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node3')
2017-10-23 16:00:55,918 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox PROPAGATE{'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*', 'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'reqId': 1508774455875574, 'signature': '2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}
2017-10-23 16:00:55,918 | DEBUG    | node.py              (1799) | processPropagate | Node Node4 received propagated request: PROPAGATE{'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*', 'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'reqId': 1508774455875574, 'signature': '2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}
2017-10-23 16:00:55,919 | TRACE    | signing.py           (  98) | serializeMsg | serialized msg {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'reqId': 1508774455875574, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:dest:V4SGRU86Z58d6TV7PBUe61|type:1|reqId:1508774455875574
2017-10-23 16:00:55,919 | TRACE    | propagator.py        ( 159) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'reqId': 1508774455875574, 'signature': '2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}
2017-10-23 16:00:55,919 | INFO     | propagator.py        ( 145) | propagate | Node4 propagating V4SGRU86Z58d6TV7PBUe6f request 1508774455875574 from client xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*
2017-10-23 16:00:55,919 | DEBUG    | node.py              (2593) | send | Node4 sending message PROPAGATE{'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*', 'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'reqId': 1508774455875574, 'signature': '2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} to all recipients: ['Node2', 'Node3', 'Node1']
2017-10-23 16:00:55,919 | DEBUG    | propagator.py        ( 208) | forward | Node4 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1508774455875574) to 2 replicas
2017-10-23 16:00:55,920 | TRACE    | zstack.py            ( 472) | _receiveFromListener | Node4C got 1 messages through listener
2017-10-23 16:00:55,920 | TRACE    | signing.py           (  98) | serializeMsg | serialized msg {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'reqId': 1508774455875574, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:dest:V4SGRU86Z58d6TV7PBUe61|type:1|reqId:1508774455875574
2017-10-23 16:00:55,921 | TRACE    | signing.py           (  98) | serializeMsg | serialized msg {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'reqId': 1508774455875574, 'signature': '2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:dest:V4SGRU86Z58d6TV7PBUe61|type:1|reqId:1508774455875574
2017-10-23 16:00:55,921 | INFO     | node.py              (2250) | verifySignature | Node4 authenticated V4SGRU86Z58d6TV7PBUe6f signature on  request 1508774455875574
2017-10-23 16:00:55,921 | TRACE    | node.py              (1444) | validateClientMsg | Node4C received CLIENT message: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'reqId': 1508774455875574, 'signature': '2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}
2017-10-23 16:00:55,922 | DISPLAY  | node.py              (1490) | processClientInBox | Node4C processing b'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*' request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'reqId': 1508774455875574, 'signature': '2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}
2017-10-23 16:00:55,922 | DEBUG    | node.py              (1751) | processRequest | Node4 received client request: SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'reqId': 1508774455875574, 'signature': '2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} from b'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*'
2017-10-23 16:00:55,922 | TRACE    | propagator.py        ( 138) | propagate | Node4 already propagated SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'reqId': 1508774455875574, 'signature': '2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}
2017-10-23 16:00:55,922 | DEBUG    | propagator.py        ( 241) | tryForwarding | Node4 not forwarding request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'reqId': 1508774455875574, 'signature': '2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} to its replicas since already forwarded
2017-10-23 16:00:55,924 | TRACE    | zstack.py            ( 726) | transmitThroughListener | Node4C transmitting b'{"reqId":1508774455875574,"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f"}' to b'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*' through listener socket
2017-10-23 16:00:55,925 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"op":"PROPAGATE","senderClient":"xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE\\/:%0nV5Wx*","request":{"operation":{"dest":"V4SGRU86Z58d6TV7PBUe61","type":"1"},"reqId":1508774455875574,"signature":"2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN","identifier":"V4SGRU86Z58d6TV7PBUe6f"}}' to Node2
2017-10-23 16:00:55,925 | TRACE    | batched.py           (  85) | flushOutBoxes | Node4 sending msg b'{"op":"PROPAGATE","senderClient":"xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE\\/:%0nV5Wx*","request":{"operation":{"dest":"V4SGRU86Z58d6TV7PBUe61","type":"1"},"reqId":1508774455875574,"signature":"2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN","identifier":"V4SGRU86Z58d6TV7PBUe6f"}}' to Node2
2017-10-23 16:00:55,926 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"op":"PROPAGATE","senderClient":"xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE\\/:%0nV5Wx*","request":{"operation":{"dest":"V4SGRU86Z58d6TV7PBUe61","type":"1"},"reqId":1508774455875574,"signature":"2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN","identifier":"V4SGRU86Z58d6TV7PBUe6f"}}' to Node3
2017-10-23 16:00:55,926 | TRACE    | batched.py           (  85) | flushOutBoxes | Node4 sending msg b'{"op":"PROPAGATE","senderClient":"xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE\\/:%0nV5Wx*","request":{"operation":{"dest":"V4SGRU86Z58d6TV7PBUe61","type":"1"},"reqId":1508774455875574,"signature":"2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN","identifier":"V4SGRU86Z58d6TV7PBUe6f"}}' to Node3
2017-10-23 16:00:55,926 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"op":"PROPAGATE","senderClient":"xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE\\/:%0nV5Wx*","request":{"operation":{"dest":"V4SGRU86Z58d6TV7PBUe61","type":"1"},"reqId":1508774455875574,"signature":"2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN","identifier":"V4SGRU86Z58d6TV7PBUe6f"}}' to Node1
2017-10-23 16:00:55,926 | TRACE    | batched.py           (  85) | flushOutBoxes | Node4 sending msg b'{"op":"PROPAGATE","senderClient":"xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE\\/:%0nV5Wx*","request":{"operation":{"dest":"V4SGRU86Z58d6TV7PBUe61","type":"1"},"reqId":1508774455875574,"signature":"2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN","identifier":"V4SGRU86Z58d6TV7PBUe6f"}}' to Node1
2017-10-23 16:00:55,928 | TRACE    | zstack.py            ( 472) | _receiveFromListener | Node4 got 1 messages through listener
2017-10-23 16:00:55,929 | TRACE    | signing.py           (  98) | serializeMsg | serialized msg {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'reqId': 1508774455875574, 'signature': '2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:dest:V4SGRU86Z58d6TV7PBUe61|type:1|reqId:1508774455875574
2017-10-23 16:00:55,929 | INFO     | node.py              (2250) | verifySignature | Node4 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate  request 1508774455875574
2017-10-23 16:00:55,929 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node2: PROPAGATE{'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*', 'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'reqId': 1508774455875574, 'signature': '2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}
2017-10-23 16:00:55,929 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'op': 'PROPAGATE', 'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*', 'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'reqId': 1508774455875574, 'signature': '2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node2')
2017-10-23 16:00:55,929 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox PROPAGATE{'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*', 'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'reqId': 1508774455875574, 'signature': '2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}
2017-10-23 16:00:55,930 | DEBUG    | node.py              (1799) | processPropagate | Node Node4 received propagated request: PROPAGATE{'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*', 'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'reqId': 1508774455875574, 'signature': '2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}
2017-10-23 16:00:55,930 | TRACE    | signing.py           (  98) | serializeMsg | serialized msg {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'reqId': 1508774455875574, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:dest:V4SGRU86Z58d6TV7PBUe61|type:1|reqId:1508774455875574
2017-10-23 16:00:55,930 | TRACE    | propagator.py        ( 138) | propagate | Node4 already propagated SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'reqId': 1508774455875574, 'signature': '2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}
2017-10-23 16:00:55,937 | DEBUG    | propagator.py        ( 241) | tryForwarding | Node4 not forwarding request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'reqId': 1508774455875574, 'signature': '2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} to its replicas since already forwarded
2017-10-23 16:00:55,939 | TRACE    | zstack.py            ( 472) | _receiveFromListener | Node4 got 1 messages through listener
2017-10-23 16:00:55,940 | TRACE    | signing.py           (  98) | serializeMsg | serialized msg {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'reqId': 1508774455875574, 'signature': '2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:dest:V4SGRU86Z58d6TV7PBUe61|type:1|reqId:1508774455875574
2017-10-23 16:00:55,940 | INFO     | node.py              (2250) | verifySignature | Node4 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate  request 1508774455875574
2017-10-23 16:00:55,941 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node1: PROPAGATE{'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*', 'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'reqId': 1508774455875574, 'signature': '2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}
2017-10-23 16:00:55,946 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'op': 'PROPAGATE', 'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*', 'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'reqId': 1508774455875574, 'signature': '2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node1')
2017-10-23 16:00:55,948 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox PROPAGATE{'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*', 'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'reqId': 1508774455875574, 'signature': '2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}
2017-10-23 16:00:55,948 | DEBUG    | node.py              (1799) | processPropagate | Node Node4 received propagated request: PROPAGATE{'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*', 'request': {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'reqId': 1508774455875574, 'signature': '2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}
2017-10-23 16:00:55,949 | TRACE    | signing.py           (  98) | serializeMsg | serialized msg {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'reqId': 1508774455875574, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:dest:V4SGRU86Z58d6TV7PBUe61|type:1|reqId:1508774455875574
2017-10-23 16:00:55,949 | TRACE    | propagator.py        ( 138) | propagate | Node4 already propagated SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'reqId': 1508774455875574, 'signature': '2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}
2017-10-23 16:00:55,949 | DEBUG    | propagator.py        ( 241) | tryForwarding | Node4 not forwarding request SafeRequest: {'operation': {'dest': 'V4SGRU86Z58d6TV7PBUe61', 'type': '1'}, 'reqId': 1508774455875574, 'signature': '2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} to its replicas since already forwarded
2017-10-23 16:00:55,960 | TRACE    | zstack.py            ( 472) | _receiveFromListener | Node4 got 1 messages through listener
2017-10-23 16:00:55,961 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node2: PREPREPARE{'ppSeqNo': 1, 'digest': '4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455', 'stateRootHash': None, 'instId': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1508774455875574]], 'txnRootHash': None, 'ledgerId': 1, 'discarded': 1, 'viewNo': 0, 'ppTime': 1508774455}
2017-10-23 16:00:55,961 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': 1, 'digest': '4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455', 'stateRootHash': None, 'instId': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1508774455875574]], 'txnRootHash': None, 'ledgerId': 1, 'discarded': 1, 'op': 'PREPREPARE', 'viewNo': 0, 'ppTime': 1508774455}, 'Node2')
2017-10-23 16:00:55,961 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox PREPREPARE{'ppSeqNo': 1, 'digest': '4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455', 'stateRootHash': None, 'instId': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1508774455875574]], 'txnRootHash': None, 'ledgerId': 1, 'discarded': 1, 'viewNo': 0, 'ppTime': 1508774455}
2017-10-23 16:00:55,963 | DEBUG    | replica.py           ( 790) | processPrePrepare | Node4:1 received PRE-PREPARE(0, 1) from Node2:1 at 955956.325381281
2017-10-23 16:00:55,963 | DEBUG    | replica.py           ( 903) | doPrepare | Node4:1 Sending PREPARE(0, 1) at 955956.325708166
2017-10-23 16:00:55,964 | INFO     | replica.py           (2064) | send | Node4:1 sending Prepare
2017-10-23 16:00:55,964 | TRACE    | replica.py           (2065) | send | Node4:1 sending PREPARE{'ppSeqNo': 1, 'digest': '4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455', 'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'ppTime': 1508774455}
2017-10-23 16:00:55,964 | DEBUG    | replica.py           ( 886) | tryCommit | Node4:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'ppSeqNo': 1, 'digest': '4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455', 'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'ppTime': 1508774455}
2017-10-23 16:00:55,964 | DEBUG    | replica.py           ( 594) | trackBatches | Node4:1 tracking batch for PREPREPARE{'ppSeqNo': 1, 'digest': '4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455', 'stateRootHash': None, 'instId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1508774455875574)], 'txnRootHash': None, 'ledgerId': 1, 'discarded': 1, 'viewNo': 0, 'ppTime': 1508774455} with state root None
2017-10-23 16:00:55,964 | DEBUG    | replica.py           ( 809) | processPrePrepare | Node4:1 processed incoming PRE-PREPARE(0, 1)
2017-10-23 16:00:55,965 | TRACE    | zstack.py            ( 472) | _receiveFromListener | Node4 got 1 messages through listener
2017-10-23 16:00:55,965 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node1: PREPREPARE{'ppSeqNo': 1, 'digest': '4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455', 'stateRootHash': 'BHooVaXUFXgirpXhPkNh2xVvamcQ2y9f3nP5ADA3cmEA', 'instId': 0, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1508774455875574]], 'txnRootHash': 'HpAj5eTvBvLAFxXcS9UP84u8X45aPci4442boEVpp6Kf', 'ledgerId': 1, 'discarded': 1, 'viewNo': 0, 'ppTime': 1508774455}
2017-10-23 16:00:55,965 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': 1, 'digest': '4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455', 'stateRootHash': 'BHooVaXUFXgirpXhPkNh2xVvamcQ2y9f3nP5ADA3cmEA', 'instId': 0, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1508774455875574]], 'txnRootHash': 'HpAj5eTvBvLAFxXcS9UP84u8X45aPci4442boEVpp6Kf', 'ledgerId': 1, 'discarded': 1, 'op': 'PREPREPARE', 'viewNo': 0, 'ppTime': 1508774455}, 'Node1')
2017-10-23 16:00:55,966 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox PREPREPARE{'ppSeqNo': 1, 'digest': '4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455', 'stateRootHash': 'BHooVaXUFXgirpXhPkNh2xVvamcQ2y9f3nP5ADA3cmEA', 'instId': 0, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1508774455875574]], 'txnRootHash': 'HpAj5eTvBvLAFxXcS9UP84u8X45aPci4442boEVpp6Kf', 'ledgerId': 1, 'discarded': 1, 'viewNo': 0, 'ppTime': 1508774455}
2017-10-23 16:00:55,972 | DEBUG    | node.py              (2593) | send | Node4 sending message PREPARE{'ppSeqNo': 1, 'digest': '4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455', 'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'ppTime': 1508774455} to all recipients: ['Node2', 'Node3', 'Node1']
2017-10-23 16:00:55,973 | DEBUG    | replica.py           ( 790) | processPrePrepare | Node4:0 received PRE-PREPARE(0, 1) from Node1:0 at 955956.334800231
2017-10-23 16:00:55,973 | DEBUG    | replica.py           (1002) | validate_pre_prepare | Node4:0 state root before processing PREPREPARE{'ppSeqNo': 1, 'digest': '4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455', 'stateRootHash': 'BHooVaXUFXgirpXhPkNh2xVvamcQ2y9f3nP5ADA3cmEA', 'instId': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1508774455875574)], 'txnRootHash': 'HpAj5eTvBvLAFxXcS9UP84u8X45aPci4442boEVpp6Kf', 'ledgerId': 1, 'discarded': 1, 'viewNo': 0, 'ppTime': 1508774455} is b'v\xe4\xb0\x14k@YY \x00\xa2\x9d\x97PT(\x10?\xb97\xc7\x15\xa8pC\x81\x98\x86-z\xaa\x0f', HPjbpQSVy894XiscZWtYK2vYyZishn1idyFLSCAEwr4
2017-10-23 16:00:55,975 | DEBUG    | replica.py           ( 903) | doPrepare | Node4:0 Sending PREPARE(0, 1) at 955956.336791413
2017-10-23 16:00:55,975 | INFO     | replica.py           (2064) | send | Node4:0 sending Prepare
2017-10-23 16:00:55,975 | TRACE    | replica.py           (2065) | send | Node4:0 sending PREPARE{'ppSeqNo': 1, 'digest': '4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455', 'stateRootHash': 'BHooVaXUFXgirpXhPkNh2xVvamcQ2y9f3nP5ADA3cmEA', 'instId': 0, 'txnRootHash': 'HpAj5eTvBvLAFxXcS9UP84u8X45aPci4442boEVpp6Kf', 'viewNo': 0, 'ppTime': 1508774455}
2017-10-23 16:00:55,975 | DEBUG    | replica.py           ( 886) | tryCommit | Node4:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'ppSeqNo': 1, 'digest': '4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455', 'stateRootHash': 'BHooVaXUFXgirpXhPkNh2xVvamcQ2y9f3nP5ADA3cmEA', 'instId': 0, 'txnRootHash': 'HpAj5eTvBvLAFxXcS9UP84u8X45aPci4442boEVpp6Kf', 'viewNo': 0, 'ppTime': 1508774455}
2017-10-23 16:00:55,975 | DEBUG    | replica.py           ( 594) | trackBatches | Node4:0 tracking batch for PREPREPARE{'ppSeqNo': 1, 'digest': '4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455', 'stateRootHash': 'BHooVaXUFXgirpXhPkNh2xVvamcQ2y9f3nP5ADA3cmEA', 'instId': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1508774455875574)], 'txnRootHash': 'HpAj5eTvBvLAFxXcS9UP84u8X45aPci4442boEVpp6Kf', 'ledgerId': 1, 'discarded': 1, 'viewNo': 0, 'ppTime': 1508774455} with state root b'v\xe4\xb0\x14k@YY \x00\xa2\x9d\x97PT(\x10?\xb97\xc7\x15\xa8pC\x81\x98\x86-z\xaa\x0f'
2017-10-23 16:00:55,975 | DEBUG    | replica.py           ( 809) | processPrePrepare | Node4:0 processed incoming PRE-PREPARE(0, 1)
2017-10-23 16:00:55,976 | TRACE    | zstack.py            ( 472) | _receiveFromListener | Node4 got 1 messages through listener
2017-10-23 16:00:55,977 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node1: PREPARE{'ppSeqNo': 1, 'digest': '4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455', 'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'ppTime': 1508774455}
2017-10-23 16:00:55,977 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': 1, 'digest': '4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455', 'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'op': 'PREPARE', 'viewNo': 0, 'ppTime': 1508774455}, 'Node1')
2017-10-23 16:00:55,977 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox PREPARE{'ppSeqNo': 1, 'digest': '4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455', 'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'ppTime': 1508774455}
2017-10-23 16:00:55,978 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"ppSeqNo":1,"txnRootHash":null,"op":"PREPARE","viewNo":0,"digest":"4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455","stateRootHash":null,"instId":1,"ppTime":1508774455}' to Node2
2017-10-23 16:00:55,979 | TRACE    | batched.py           (  85) | flushOutBoxes | Node4 sending msg b'{"ppSeqNo":1,"txnRootHash":null,"op":"PREPARE","viewNo":0,"digest":"4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455","stateRootHash":null,"instId":1,"ppTime":1508774455}' to Node2
2017-10-23 16:00:55,979 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"ppSeqNo":1,"txnRootHash":null,"op":"PREPARE","viewNo":0,"digest":"4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455","stateRootHash":null,"instId":1,"ppTime":1508774455}' to Node3
2017-10-23 16:00:55,979 | TRACE    | batched.py           (  85) | flushOutBoxes | Node4 sending msg b'{"ppSeqNo":1,"txnRootHash":null,"op":"PREPARE","viewNo":0,"digest":"4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455","stateRootHash":null,"instId":1,"ppTime":1508774455}' to Node3
2017-10-23 16:00:55,979 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"ppSeqNo":1,"txnRootHash":null,"op":"PREPARE","viewNo":0,"digest":"4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455","stateRootHash":null,"instId":1,"ppTime":1508774455}' to Node1
2017-10-23 16:00:55,979 | TRACE    | batched.py           (  85) | flushOutBoxes | Node4 sending msg b'{"ppSeqNo":1,"txnRootHash":null,"op":"PREPARE","viewNo":0,"digest":"4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455","stateRootHash":null,"instId":1,"ppTime":1508774455}' to Node1
2017-10-23 16:00:55,980 | DEBUG    | node.py              (2593) | send | Node4 sending message PREPARE{'ppSeqNo': 1, 'digest': '4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455', 'stateRootHash': 'BHooVaXUFXgirpXhPkNh2xVvamcQ2y9f3nP5ADA3cmEA', 'instId': 0, 'txnRootHash': 'HpAj5eTvBvLAFxXcS9UP84u8X45aPci4442boEVpp6Kf', 'viewNo': 0, 'ppTime': 1508774455} to all recipients: ['Node2', 'Node3', 'Node1']
2017-10-23 16:00:55,980 | DEBUG    | replica.py           ( 834) | processPrepare | Node4:1 received PREPARE(0, 1) from Node1:1
2017-10-23 16:00:55,981 | DEBUG    | replica.py           ( 922) | doCommit | Node4:1 Sending COMMIT(0, 1) at 955956.342833392
2017-10-23 16:00:55,981 | INFO     | replica.py           (2064) | send | Node4:1 sending Commit
2017-10-23 16:00:55,981 | TRACE    | replica.py           (2065) | send | Node4:1 sending COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 0}
2017-10-23 16:00:55,981 | DEBUG    | replica.py           ( 898) | tryOrder | Node4:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 0} commits where f is 1
2017-10-23 16:00:55,981 | DEBUG    | replica.py           ( 845) | processPrepare | Node4:1 processed incoming PREPARE (0, 1)
2017-10-23 16:00:55,983 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"ppSeqNo":1,"txnRootHash":"HpAj5eTvBvLAFxXcS9UP84u8X45aPci4442boEVpp6Kf","op":"PREPARE","viewNo":0,"digest":"4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455","stateRootHash":"BHooVaXUFXgirpXhPkNh2xVvamcQ2y9f3nP5ADA3cmEA","instId":0,"ppTime":1508774455}' to Node2
2017-10-23 16:00:55,983 | TRACE    | batched.py           (  85) | flushOutBoxes | Node4 sending msg b'{"ppSeqNo":1,"txnRootHash":"HpAj5eTvBvLAFxXcS9UP84u8X45aPci4442boEVpp6Kf","op":"PREPARE","viewNo":0,"digest":"4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455","stateRootHash":"BHooVaXUFXgirpXhPkNh2xVvamcQ2y9f3nP5ADA3cmEA","instId":0,"ppTime":1508774455}' to Node2
2017-10-23 16:00:55,983 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"ppSeqNo":1,"txnRootHash":"HpAj5eTvBvLAFxXcS9UP84u8X45aPci4442boEVpp6Kf","op":"PREPARE","viewNo":0,"digest":"4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455","stateRootHash":"BHooVaXUFXgirpXhPkNh2xVvamcQ2y9f3nP5ADA3cmEA","instId":0,"ppTime":1508774455}' to Node3
2017-10-23 16:00:55,984 | TRACE    | batched.py           (  85) | flushOutBoxes | Node4 sending msg b'{"ppSeqNo":1,"txnRootHash":"HpAj5eTvBvLAFxXcS9UP84u8X45aPci4442boEVpp6Kf","op":"PREPARE","viewNo":0,"digest":"4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455","stateRootHash":"BHooVaXUFXgirpXhPkNh2xVvamcQ2y9f3nP5ADA3cmEA","instId":0,"ppTime":1508774455}' to Node3
2017-10-23 16:00:55,984 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"ppSeqNo":1,"txnRootHash":"HpAj5eTvBvLAFxXcS9UP84u8X45aPci4442boEVpp6Kf","op":"PREPARE","viewNo":0,"digest":"4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455","stateRootHash":"BHooVaXUFXgirpXhPkNh2xVvamcQ2y9f3nP5ADA3cmEA","instId":0,"ppTime":1508774455}' to Node1
2017-10-23 16:00:55,984 | TRACE    | batched.py           (  85) | flushOutBoxes | Node4 sending msg b'{"ppSeqNo":1,"txnRootHash":"HpAj5eTvBvLAFxXcS9UP84u8X45aPci4442boEVpp6Kf","op":"PREPARE","viewNo":0,"digest":"4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455","stateRootHash":"BHooVaXUFXgirpXhPkNh2xVvamcQ2y9f3nP5ADA3cmEA","instId":0,"ppTime":1508774455}' to Node1
2017-10-23 16:00:55,985 | DEBUG    | node.py              (2593) | send | Node4 sending message COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 0} to all recipients: ['Node2', 'Node3', 'Node1']
2017-10-23 16:00:55,987 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"ppSeqNo":1,"op":"COMMIT","instId":1,"viewNo":0}' to Node2
2017-10-23 16:00:55,987 | TRACE    | batched.py           (  85) | flushOutBoxes | Node4 sending msg b'{"ppSeqNo":1,"op":"COMMIT","instId":1,"viewNo":0}' to Node2
2017-10-23 16:00:55,987 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"ppSeqNo":1,"op":"COMMIT","instId":1,"viewNo":0}' to Node3
2017-10-23 16:00:55,988 | TRACE    | batched.py           (  85) | flushOutBoxes | Node4 sending msg b'{"ppSeqNo":1,"op":"COMMIT","instId":1,"viewNo":0}' to Node3
2017-10-23 16:00:55,988 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"ppSeqNo":1,"op":"COMMIT","instId":1,"viewNo":0}' to Node1
2017-10-23 16:00:55,988 | TRACE    | batched.py           (  85) | flushOutBoxes | Node4 sending msg b'{"ppSeqNo":1,"op":"COMMIT","instId":1,"viewNo":0}' to Node1
2017-10-23 16:00:56,011 | TRACE    | zstack.py            ( 472) | _receiveFromListener | Node4 got 3 messages through listener
2017-10-23 16:00:56,012 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node3: PREPARE{'ppSeqNo': 1, 'digest': '4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455', 'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'ppTime': 1508774455}
2017-10-23 16:00:56,012 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': 1, 'digest': '4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455', 'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'op': 'PREPARE', 'viewNo': 0, 'ppTime': 1508774455}, 'Node3')
2017-10-23 16:00:56,012 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox PREPARE{'ppSeqNo': 1, 'digest': '4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455', 'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'ppTime': 1508774455}
2017-10-23 16:00:56,012 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node3: BATCH{'messages': ['{"stateRootHash":"BHooVaXUFXgirpXhPkNh2xVvamcQ2y9f3nP5ADA3cmEA","ppSeqNo":1,"viewNo":0,"op":"PREPARE","ppTime":1508774455,"digest":"4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455","instId":0,"txnRootHash":"HpAj5eTvBvLAFxXcS9UP84u8X45aPci4442boEVpp6Kf"}', '{"instId":1,"ppSeqNo":1,"viewNo":0,"op":"COMMIT"}'], 'signature': None}
2017-10-23 16:00:56,012 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'messages': ['{"stateRootHash":"BHooVaXUFXgirpXhPkNh2xVvamcQ2y9f3nP5ADA3cmEA","ppSeqNo":1,"viewNo":0,"op":"PREPARE","ppTime":1508774455,"digest":"4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455","instId":0,"txnRootHash":"HpAj5eTvBvLAFxXcS9UP84u8X45aPci4442boEVpp6Kf"}', '{"instId":1,"ppSeqNo":1,"viewNo":0,"op":"COMMIT"}'], 'signature': None}, 'Node3')
2017-10-23 16:00:56,012 | DEBUG    | node.py              (1328) | unpackNodeMsg | Node4 processing a batch BATCH{'messages': ['{"stateRootHash":"BHooVaXUFXgirpXhPkNh2xVvamcQ2y9f3nP5ADA3cmEA","ppSeqNo":1,"viewNo":0,"op":"PREPARE","ppTime":1508774455,"digest":"4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455","instId":0,"txnRootHash":"HpAj5eTvBvLAFxXcS9UP84u8X45aPci4442boEVpp6Kf"}', '{"instId":1,"ppSeqNo":1,"viewNo":0,"op":"COMMIT"}'], 'signature': None}
2017-10-23 16:00:56,012 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node3: PREPARE{'ppSeqNo': 1, 'digest': '4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455', 'stateRootHash': 'BHooVaXUFXgirpXhPkNh2xVvamcQ2y9f3nP5ADA3cmEA', 'instId': 0, 'txnRootHash': 'HpAj5eTvBvLAFxXcS9UP84u8X45aPci4442boEVpp6Kf', 'viewNo': 0, 'ppTime': 1508774455}
2017-10-23 16:00:56,013 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': 1, 'digest': '4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455', 'stateRootHash': 'BHooVaXUFXgirpXhPkNh2xVvamcQ2y9f3nP5ADA3cmEA', 'instId': 0, 'txnRootHash': 'HpAj5eTvBvLAFxXcS9UP84u8X45aPci4442boEVpp6Kf', 'op': 'PREPARE', 'viewNo': 0, 'ppTime': 1508774455}, 'Node3')
2017-10-23 16:00:56,013 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox PREPARE{'ppSeqNo': 1, 'digest': '4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455', 'stateRootHash': 'BHooVaXUFXgirpXhPkNh2xVvamcQ2y9f3nP5ADA3cmEA', 'instId': 0, 'txnRootHash': 'HpAj5eTvBvLAFxXcS9UP84u8X45aPci4442boEVpp6Kf', 'viewNo': 0, 'ppTime': 1508774455}
2017-10-23 16:00:56,013 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node3: COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 0}
2017-10-23 16:00:56,013 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': 1, 'op': 'COMMIT', 'instId': 1, 'viewNo': 0}, 'Node3')
2017-10-23 16:00:56,013 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 0}
2017-10-23 16:00:56,013 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node3: COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 0}
2017-10-23 16:00:56,013 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': 1, 'op': 'COMMIT', 'instId': 0, 'viewNo': 0}, 'Node3')
2017-10-23 16:00:56,013 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 0}
2017-10-23 16:00:56,016 | DEBUG    | replica.py           ( 834) | processPrepare | Node4:0 received PREPARE(0, 1) from Node3:0
2017-10-23 16:00:56,016 | DEBUG    | replica.py           ( 922) | doCommit | Node4:0 Sending COMMIT(0, 1) at 955956.378550313
2017-10-23 16:00:56,016 | INFO     | replica.py           (2064) | send | Node4:0 sending Commit
2017-10-23 16:00:56,016 | TRACE    | replica.py           (2065) | send | Node4:0 sending COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 0}
2017-10-23 16:00:56,017 | DEBUG    | replica.py           ( 898) | tryOrder | Node4:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 0} commits where f is 1
2017-10-23 16:00:56,017 | DEBUG    | replica.py           ( 845) | processPrepare | Node4:0 processed incoming PREPARE (0, 1)
2017-10-23 16:00:56,017 | DEBUG    | replica.py           ( 863) | processCommit | Node4:0 received COMMIT(0, 1) from Node3:0
2017-10-23 16:00:56,017 | DEBUG    | replica.py           ( 898) | tryOrder | Node4:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 0} commits where f is 1
2017-10-23 16:00:56,017 | DEBUG    | replica.py           ( 874) | processCommit | Node4:0 processed incoming COMMIT(0, 1)
2017-10-23 16:00:56,017 | DEBUG    | replica.py           ( 834) | processPrepare | Node4:1 received PREPARE(0, 1) from Node3:1
2017-10-23 16:00:56,017 | DEBUG    | replica.py           ( 886) | tryCommit | Node4:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'ppSeqNo': 1, 'digest': '4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455', 'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'ppTime': 1508774455}
2017-10-23 16:00:56,017 | DEBUG    | replica.py           ( 845) | processPrepare | Node4:1 processed incoming PREPARE (0, 1)
2017-10-23 16:00:56,017 | DEBUG    | replica.py           ( 863) | processCommit | Node4:1 received COMMIT(0, 1) from Node3:1
2017-10-23 16:00:56,018 | DEBUG    | replica.py           ( 898) | tryOrder | Node4:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 0} commits where f is 1
2017-10-23 16:00:56,018 | DEBUG    | replica.py           ( 874) | processCommit | Node4:1 processed incoming COMMIT(0, 1)
2017-10-23 16:00:56,023 | TRACE    | zstack.py            ( 472) | _receiveFromListener | Node4 got 2 messages through listener
2017-10-23 16:00:56,023 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node2: PREPARE{'ppSeqNo': 1, 'digest': '4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455', 'stateRootHash': 'BHooVaXUFXgirpXhPkNh2xVvamcQ2y9f3nP5ADA3cmEA', 'instId': 0, 'txnRootHash': 'HpAj5eTvBvLAFxXcS9UP84u8X45aPci4442boEVpp6Kf', 'viewNo': 0, 'ppTime': 1508774455}
2017-10-23 16:00:56,024 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': 1, 'digest': '4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455', 'stateRootHash': 'BHooVaXUFXgirpXhPkNh2xVvamcQ2y9f3nP5ADA3cmEA', 'instId': 0, 'txnRootHash': 'HpAj5eTvBvLAFxXcS9UP84u8X45aPci4442boEVpp6Kf', 'op': 'PREPARE', 'ppTime': 1508774455, 'viewNo': 0}, 'Node2')
2017-10-23 16:00:56,024 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox PREPARE{'ppSeqNo': 1, 'digest': '4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455', 'stateRootHash': 'BHooVaXUFXgirpXhPkNh2xVvamcQ2y9f3nP5ADA3cmEA', 'instId': 0, 'txnRootHash': 'HpAj5eTvBvLAFxXcS9UP84u8X45aPci4442boEVpp6Kf', 'viewNo': 0, 'ppTime': 1508774455}
2017-10-23 16:00:56,024 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node2: BATCH{'messages': ['{"viewNo":0,"instId":0,"ppSeqNo":1,"op":"COMMIT"}', '{"viewNo":0,"instId":1,"ppSeqNo":1,"op":"COMMIT"}'], 'signature': None}
2017-10-23 16:00:56,024 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'messages': ['{"viewNo":0,"instId":0,"ppSeqNo":1,"op":"COMMIT"}', '{"viewNo":0,"instId":1,"ppSeqNo":1,"op":"COMMIT"}'], 'signature': None}, 'Node2')
2017-10-23 16:00:56,024 | DEBUG    | node.py              (1328) | unpackNodeMsg | Node4 processing a batch BATCH{'messages': ['{"viewNo":0,"instId":0,"ppSeqNo":1,"op":"COMMIT"}', '{"viewNo":0,"instId":1,"ppSeqNo":1,"op":"COMMIT"}'], 'signature': None}
2017-10-23 16:00:56,024 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node2: COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 0}
2017-10-23 16:00:56,024 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': 1, 'op': 'COMMIT', 'instId': 0, 'viewNo': 0}, 'Node2')
2017-10-23 16:00:56,024 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 0}
2017-10-23 16:00:56,024 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node2: COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 0}
2017-10-23 16:00:56,024 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': 1, 'op': 'COMMIT', 'instId': 1, 'viewNo': 0}, 'Node2')
2017-10-23 16:00:56,025 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 0}
2017-10-23 16:00:56,027 | DEBUG    | node.py              (2593) | send | Node4 sending message COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 0} to all recipients: ['Node2', 'Node3', 'Node1']
2017-10-23 16:00:56,027 | DEBUG    | replica.py           ( 834) | processPrepare | Node4:0 received PREPARE(0, 1) from Node2:0
2017-10-23 16:00:56,028 | DEBUG    | replica.py           ( 886) | tryCommit | Node4:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'ppSeqNo': 1, 'digest': '4a73eb0669d7007f5f6c937e9dde7ab75408bdaa4a017394d3a59b42719e0455', 'stateRootHash': 'BHooVaXUFXgirpXhPkNh2xVvamcQ2y9f3nP5ADA3cmEA', 'instId': 0, 'txnRootHash': 'HpAj5eTvBvLAFxXcS9UP84u8X45aPci4442boEVpp6Kf', 'viewNo': 0, 'ppTime': 1508774455}
2017-10-23 16:00:56,028 | DEBUG    | replica.py           ( 845) | processPrepare | Node4:0 processed incoming PREPARE (0, 1)
2017-10-23 16:00:56,028 | DEBUG    | replica.py           ( 863) | processCommit | Node4:0 received COMMIT(0, 1) from Node2:0
2017-10-23 16:00:56,028 | TRACE    | replica.py           ( 894) | tryOrder | Node4:0 returning request to node
2017-10-23 16:00:56,028 | INFO     | replica.py           (1395) | doOrder | Node4:0 ordering COMMIT(0, 1)
2017-10-23 16:00:56,028 | DEBUG    | replica.py           ( 310) | last_ordered_3pc | Node4:0 set last ordered as (0, 1)
2017-10-23 16:00:56,028 | INFO     | replica.py           (2064) | send | Node4:0 sending Ordered
2017-10-23 16:00:56,028 | TRACE    | replica.py           (2065) | send | Node4:0 sending ORDERED{'ppSeqNo': 1, 'stateRootHash': 'BHooVaXUFXgirpXhPkNh2xVvamcQ2y9f3nP5ADA3cmEA', 'instId': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1508774455875574)], 'txnRootHash': 'HpAj5eTvBvLAFxXcS9UP84u8X45aPci4442boEVpp6Kf', 'ledgerId': 1, 'viewNo': 0, 'ppTime': 1508774455}
2017-10-23 16:00:56,029 | DEBUG    | replica.py           (1426) | order_3pc_key | Node4:0 ordered request (0, 1)
2017-10-23 16:00:56,029 | DEBUG    | replica.py           (1494) | _newCheckpointState | Node4:0 adding new checkpoint state for (1, 100)
2017-10-23 16:00:56,029 | DEBUG    | replica.py           ( 874) | processCommit | Node4:0 processed incoming COMMIT(0, 1)
2017-10-23 16:00:56,029 | DEBUG    | replica.py           ( 863) | processCommit | Node4:1 received COMMIT(0, 1) from Node2:1
2017-10-23 16:00:56,029 | TRACE    | replica.py           ( 894) | tryOrder | Node4:1 returning request to node
2017-10-23 16:00:56,029 | INFO     | replica.py           (1395) | doOrder | Node4:1 ordering COMMIT(0, 1)
2017-10-23 16:00:56,029 | DEBUG    | replica.py           ( 310) | last_ordered_3pc | Node4:1 set last ordered as (0, 1)
2017-10-23 16:00:56,030 | INFO     | replica.py           (2064) | send | Node4:1 sending Ordered
2017-10-23 16:00:56,030 | TRACE    | replica.py           (2065) | send | Node4:1 sending ORDERED{'ppSeqNo': 1, 'stateRootHash': None, 'instId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1508774455875574)], 'txnRootHash': None, 'ledgerId': 1, 'viewNo': 0, 'ppTime': 1508774455}
2017-10-23 16:00:56,031 | DEBUG    | replica.py           (1426) | order_3pc_key | Node4:1 ordered request (0, 1)
2017-10-23 16:00:56,031 | DEBUG    | replica.py           (1494) | _newCheckpointState | Node4:1 adding new checkpoint state for (1, 100)
2017-10-23 16:00:56,031 | DEBUG    | replica.py           ( 874) | processCommit | Node4:1 processed incoming COMMIT(0, 1)
2017-10-23 16:00:56,033 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"ppSeqNo":1,"op":"COMMIT","instId":0,"viewNo":0}' to Node2
2017-10-23 16:00:56,033 | TRACE    | batched.py           (  85) | flushOutBoxes | Node4 sending msg b'{"ppSeqNo":1,"op":"COMMIT","instId":0,"viewNo":0}' to Node2
2017-10-23 16:00:56,033 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"ppSeqNo":1,"op":"COMMIT","instId":0,"viewNo":0}' to Node3
2017-10-23 16:00:56,033 | TRACE    | batched.py           (  85) | flushOutBoxes | Node4 sending msg b'{"ppSeqNo":1,"op":"COMMIT","instId":0,"viewNo":0}' to Node3
2017-10-23 16:00:56,034 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"ppSeqNo":1,"op":"COMMIT","instId":0,"viewNo":0}' to Node1
2017-10-23 16:00:56,034 | TRACE    | batched.py           (  85) | flushOutBoxes | Node4 sending msg b'{"ppSeqNo":1,"op":"COMMIT","instId":0,"viewNo":0}' to Node1
2017-10-23 16:00:56,035 | DEBUG    | node.py              (1881) | processOrdered | Node4 executing Ordered batch 0 1 of 1 requests
2017-10-23 16:00:56,049 | DEBUG    | ledger.py            (  68) | commitTxns | Committed 1 txns, 0 are uncommitted
2017-10-23 16:00:56,050 | DEBUG    | node.py              (2374) | sendReplyToClient | Node4 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1508774455875574) to client
2017-10-23 16:00:56,050 | TRACE    | zstack.py            ( 726) | transmitThroughListener | Node4C transmitting b'{"result":{"txnTime":1508774455,"seqNo":7,"type":"1","signature":"2Tge9bgHPSjjkwMnV9AFXYQFcdeHB9mYdjDHPL8AZa9aj3jXfsHXQcwQM8qD1ZEw76uqPH1yDWTe4sk4LdtEvKmN","auditPath":["75bHzK8dAu3kqVUNuTYp1S3eRWjLJn8anMZLi4PananM","FckQgFWTjpBeboPFpwZ2YKB8CnhFQ1rVGuJNnwygmBHS"],"reqId":1508774455875574,"dest":"V4SGRU86Z58d6TV7PBUe61","rootHash":"HpAj5eTvBvLAFxXcS9UP84u8X45aPci4442boEVpp6Kf","identifier":"V4SGRU86Z58d6TV7PBUe6f"},"op":"REPLY"}' to b'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*' through listener socket
2017-10-23 16:00:56,050 | DEBUG    | node.py              (2295) | executeBatch | Node4 storing 3PC key (0, 1) for ledger 1 range (7, 7)
2017-10-23 16:00:56,050 | TRACE    | node.py              (1892) | processOrdered | Node4 got ordered requests from backup replica 1
2017-10-23 16:00:56,050 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:00:56,051 | TRACE    | zstack.py            ( 472) | _receiveFromListener | Node4 got 1 messages through listener
2017-10-23 16:00:56,052 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node1: COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 0}
2017-10-23 16:00:56,052 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': 1, 'op': 'COMMIT', 'instId': 1, 'viewNo': 0}, 'Node1')
2017-10-23 16:00:56,052 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 0}
2017-10-23 16:00:56,055 | TRACE    | message_processor.py (  28) | discard | Node4:1 discarding message COMMIT{'ppSeqNo': 1, 'instId': 1, 'viewNo': 0} because already ordered 3 phase message
2017-10-23 16:00:56,056 | TRACE    | zstack.py            ( 472) | _receiveFromListener | Node4 got 1 messages through listener
2017-10-23 16:00:56,057 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node1: COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 0}
2017-10-23 16:00:56,057 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': 1, 'op': 'COMMIT', 'instId': 0, 'viewNo': 0}, 'Node1')
2017-10-23 16:00:56,062 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 0}
2017-10-23 16:00:56,065 | TRACE    | message_processor.py (  28) | discard | Node4:0 discarding message COMMIT{'ppSeqNo': 1, 'instId': 0, 'viewNo': 0} because already ordered 3 phase message
2017-10-23 16:01:00,651 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:01:00,651 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:01:00,651 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:01:00,652 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:01:05,683 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 31
2017-10-23 16:01:05,684 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:01:05,684 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:01:05,684 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.0015339007205772 is acceptable.
2017-10-23 16:01:05,684 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:01:05,684 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:01:05,684 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:01:05,684 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 32 to run in 10 seconds
2017-10-23 16:01:15,658 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:01:15,658 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:01:15,658 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:01:15,659 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:01:15,686 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 32
2017-10-23 16:01:15,687 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:01:15,687 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:01:15,687 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.0015339007205772 is acceptable.
2017-10-23 16:01:15,687 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:01:15,687 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:01:15,687 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:01:15,687 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 33 to run in 10 seconds
2017-10-23 16:01:25,693 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 33
2017-10-23 16:01:25,693 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:01:25,693 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:01:25,693 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.0015339007205772 is acceptable.
2017-10-23 16:01:25,693 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:01:25,693 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:01:25,694 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:01:25,694 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 34 to run in 10 seconds
2017-10-23 16:01:30,670 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:01:30,670 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:01:30,670 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:01:30,670 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:01:35,713 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 34
2017-10-23 16:01:35,713 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:01:35,713 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:01:35,713 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.0015339007205772 is acceptable.
2017-10-23 16:01:35,713 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:01:35,714 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:01:35,714 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:01:35,714 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 35 to run in 10 seconds
2017-10-23 16:01:45,469 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkNodeRequestSpike with id 29
2017-10-23 16:01:45,470 | DEBUG    | node.py              (2026) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:01:45,470 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:01:45,470 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 36 to run in 60 seconds
2017-10-23 16:01:45,470 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 5
2017-10-23 16:01:45,470 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:01:45,470 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 6 to run in 60 seconds
2017-10-23 16:01:45,676 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:01:45,676 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:01:45,676 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:01:45,676 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:01:45,718 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 35
2017-10-23 16:01:45,718 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:01:45,718 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:01:45,718 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.0015339007205772 is acceptable.
2017-10-23 16:01:45,718 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:01:45,719 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:01:45,719 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:01:45,719 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 37 to run in 10 seconds
2017-10-23 16:01:55,720 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 37
2017-10-23 16:01:55,720 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:01:55,720 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:01:55,720 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.0015339007205772 is acceptable.
2017-10-23 16:01:55,720 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:01:55,721 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:01:55,721 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:01:55,721 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 38 to run in 10 seconds
2017-10-23 16:02:00,679 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:02:00,679 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:02:00,679 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:02:00,680 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:02:05,735 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 38
2017-10-23 16:02:05,735 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:02:05,735 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:02:05,735 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.0015339007205772 is acceptable.
2017-10-23 16:02:05,735 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:02:05,736 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:02:05,736 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:02:05,736 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 39 to run in 10 seconds
2017-10-23 16:02:15,687 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:02:15,687 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:02:15,687 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:02:15,688 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:02:15,745 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 39
2017-10-23 16:02:15,745 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:02:15,745 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:02:15,745 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.0015339007205772 is acceptable.
2017-10-23 16:02:15,745 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:02:15,746 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:02:15,746 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:02:15,746 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 40 to run in 10 seconds
2017-10-23 16:02:25,748 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 40
2017-10-23 16:02:25,748 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:02:25,748 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:02:25,748 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.0015339007205772 is acceptable.
2017-10-23 16:02:25,748 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:02:25,748 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:02:25,748 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:02:25,748 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 41 to run in 10 seconds
2017-10-23 16:02:30,698 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:02:30,698 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:02:30,698 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:02:30,699 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:02:35,757 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 41
2017-10-23 16:02:35,758 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:02:35,758 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:02:35,758 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.0015339007205772 is acceptable.
2017-10-23 16:02:35,758 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:02:35,758 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:02:35,758 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:02:35,758 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 42 to run in 10 seconds
2017-10-23 16:02:45,477 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkNodeRequestSpike with id 36
2017-10-23 16:02:45,477 | DEBUG    | node.py              (2026) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:02:45,477 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:02:45,477 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 43 to run in 60 seconds
2017-10-23 16:02:45,477 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 6
2017-10-23 16:02:45,477 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:02:45,477 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 7 to run in 60 seconds
2017-10-23 16:02:45,711 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:02:45,711 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:02:45,711 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:02:45,712 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:02:45,766 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 42
2017-10-23 16:02:45,766 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:02:45,766 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:02:45,766 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.0015339007205772 is acceptable.
2017-10-23 16:02:45,766 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:02:45,766 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:02:45,766 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:02:45,767 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 44 to run in 10 seconds
2017-10-23 16:02:55,769 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 44
2017-10-23 16:02:55,770 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:02:55,770 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:02:55,770 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.0015339007205772 is acceptable.
2017-10-23 16:02:55,770 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:02:55,770 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:02:55,770 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:02:55,770 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 45 to run in 10 seconds
2017-10-23 16:03:00,721 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:03:00,721 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:03:00,721 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:03:00,721 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:03:05,784 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 45
2017-10-23 16:03:05,784 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:03:05,784 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:03:05,784 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.0015339007205772 is acceptable.
2017-10-23 16:03:05,784 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:03:05,785 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:03:05,785 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:03:05,785 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 46 to run in 10 seconds
2017-10-23 16:03:15,732 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:03:15,732 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:03:15,732 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:03:15,733 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:03:15,792 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 46
2017-10-23 16:03:15,793 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:03:15,793 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:03:15,793 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.0015339007205772 is acceptable.
2017-10-23 16:03:15,793 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:03:15,793 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:03:15,793 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:03:15,793 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 47 to run in 10 seconds
2017-10-23 16:03:25,794 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 47
2017-10-23 16:03:25,794 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:03:25,794 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:03:25,794 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.0015339007205772 is acceptable.
2017-10-23 16:03:25,794 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:03:25,794 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:03:25,794 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:03:25,794 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 48 to run in 10 seconds
2017-10-23 16:03:30,732 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:03:30,733 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:03:30,733 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:03:30,733 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:03:35,795 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 48
2017-10-23 16:03:35,795 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:03:35,795 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:03:35,795 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.0015339007205772 is acceptable.
2017-10-23 16:03:35,795 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:03:35,795 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:03:35,795 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:03:35,796 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 49 to run in 10 seconds
2017-10-23 16:03:45,486 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkNodeRequestSpike with id 43
2017-10-23 16:03:45,486 | DEBUG    | node.py              (2026) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:03:45,486 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:03:45,487 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 50 to run in 60 seconds
2017-10-23 16:03:45,487 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 7
2017-10-23 16:03:45,487 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:03:45,487 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 8 to run in 60 seconds
2017-10-23 16:03:45,736 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:03:45,736 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:03:45,736 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:03:45,737 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:03:45,809 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 49
2017-10-23 16:03:45,809 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:03:45,809 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:03:45,809 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.0015339007205772 is acceptable.
2017-10-23 16:03:45,809 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:03:45,809 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:03:45,809 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:03:45,809 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 51 to run in 10 seconds
2017-10-23 16:03:55,812 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 51
2017-10-23 16:03:55,813 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:03:55,813 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:03:55,813 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.0015339007205772 is acceptable.
2017-10-23 16:03:55,813 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:03:55,813 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:03:55,813 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:03:55,813 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 52 to run in 10 seconds
2017-10-23 16:04:00,749 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:04:00,750 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:04:00,750 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:04:00,750 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:04:05,823 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 52
2017-10-23 16:04:05,823 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:04:05,823 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:04:05,823 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.0015339007205772 is acceptable.
2017-10-23 16:04:05,823 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:04:05,824 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:04:05,824 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:04:05,824 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 53 to run in 10 seconds
2017-10-23 16:04:15,757 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:04:15,758 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:04:15,758 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:04:15,758 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:04:15,826 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 53
2017-10-23 16:04:15,826 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:04:15,826 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:04:15,826 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.0015339007205772 is acceptable.
2017-10-23 16:04:15,826 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:04:15,827 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:04:15,827 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:04:15,827 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 54 to run in 10 seconds
2017-10-23 16:04:25,844 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 54
2017-10-23 16:04:25,845 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:04:25,845 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:04:25,845 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.0015339007205772 is acceptable.
2017-10-23 16:04:25,845 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:04:25,845 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:04:25,846 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:04:25,846 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 55 to run in 10 seconds
2017-10-23 16:04:30,767 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:04:30,767 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:04:30,767 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:04:30,767 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:04:35,852 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 55
2017-10-23 16:04:35,852 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:04:35,852 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:04:35,852 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.0015339007205772 is acceptable.
2017-10-23 16:04:35,852 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:04:35,852 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:04:35,852 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:04:35,853 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 56 to run in 10 seconds
2017-10-23 16:04:45,500 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkNodeRequestSpike with id 50
2017-10-23 16:04:45,500 | DEBUG    | node.py              (2026) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:04:45,500 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:04:45,500 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 57 to run in 60 seconds
2017-10-23 16:04:45,500 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 8
2017-10-23 16:04:45,500 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:04:45,500 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 9 to run in 60 seconds
2017-10-23 16:04:45,780 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:04:45,780 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:04:45,780 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:04:45,781 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:04:45,865 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 56
2017-10-23 16:04:45,865 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:04:45,865 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:04:45,865 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.0015339007205772 is acceptable.
2017-10-23 16:04:45,865 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:04:45,865 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:04:45,865 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:04:45,866 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 58 to run in 10 seconds
2017-10-23 16:04:55,872 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 58
2017-10-23 16:04:55,872 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:04:55,872 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:04:55,872 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.0015339007205772 is acceptable.
2017-10-23 16:04:55,872 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:04:55,873 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:04:55,873 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:04:55,873 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 59 to run in 10 seconds
2017-10-23 16:05:00,781 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:05:00,781 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:05:00,782 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:05:00,782 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:05:05,883 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 59
2017-10-23 16:05:05,884 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:05:05,884 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:05:05,884 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.0015339007205772 is acceptable.
2017-10-23 16:05:05,884 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:05:05,884 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:05:05,884 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:05:05,884 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 60 to run in 10 seconds
2017-10-23 16:05:15,782 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:05:15,782 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:05:15,785 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:05:15,786 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:05:15,889 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 60
2017-10-23 16:05:15,889 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:05:15,889 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:05:15,889 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.0015339007205772 is acceptable.
2017-10-23 16:05:15,889 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:05:15,889 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:05:15,889 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:05:15,890 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 61 to run in 10 seconds
2017-10-23 16:05:25,896 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 61
2017-10-23 16:05:25,897 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:05:25,897 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:05:25,897 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.0015339007205772 is acceptable.
2017-10-23 16:05:25,897 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:05:25,897 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:05:25,897 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:05:25,897 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 62 to run in 10 seconds
2017-10-23 16:05:30,791 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:05:30,791 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:05:30,791 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:05:30,792 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:05:34,752 | TRACE    | zstack.py            ( 472) | _receiveFromListener | Node4C got 1 messages through listener
2017-10-23 16:05:34,753 | TRACE    | signing.py           (  98) | serializeMsg | serialized msg {'operation': {'timeout': 10, 'type': '109', 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'action': 'start', 'justification': None, 'force': False, 'version': '1.1.37'}, 'reqId': 1508774734648330, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:action:start|force:False|justification:|name:upgrade-1137|schedule:4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA:2017-10-17T11:35:00.000000+00:00|4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc:2017-10-17T11:40:00.000000+00:00|8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb:2017-10-17T11:25:00.000000+00:00|DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya:2017-10-17T11:30:00.000000+00:00|Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv:2017-10-17T11:20:00.000000+00:00|sha256:f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3|timeout:10|type:109|version:1.1.37|reqId:1508774734648330
2017-10-23 16:05:34,754 | TRACE    | zstack.py            ( 726) | transmitThroughListener | Node4C transmitting b'{"reason":"client request invalid: InvalidClientRequest(\\"{\'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb\': \'2017-10-17T11:25:00.000000+00:00\', \'4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA\': \'2017-10-17T11:35:00.000000+00:00\', \'4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc\': \'2017-10-17T11:40:00.000000+00:00\', \'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv\': \'2017-10-17T11:20:00.000000+00:00\', \'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya\': \'2017-10-17T11:30:00.000000+00:00\'} not a valid schedule since Schedule should contain id of all nodes\\",)","reqId":1508774734648330,"op":"REQNACK","identifier":"V4SGRU86Z58d6TV7PBUe6f"}' to b'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*' through listener socket
2017-10-23 16:05:34,754 | WARNING  | message_processor.py (  28) | discard | Node4 discarding message ({'operation': {'timeout': 10, 'type': '109', 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'action': 'start', 'justification': None, 'force': False, 'version': '1.1.37'}, 'reqId': 1508774734648330, 'signature': '3XPXgtXqTcbNq7NvLPcDSusWe97daPb7cfWr5jCv9ZNHW9a85LGVHh6fLdaoUu8GzNSAk6qQZXyKoKJ4NiYNzDzN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}, b'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*') because InvalidClientRequest("{'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'} not a valid schedule since Schedule should contain id of all nodes",)
2017-10-23 16:05:35,903 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 62
2017-10-23 16:05:35,903 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:05:35,903 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:05:35,903 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.0015339007205772 is acceptable.
2017-10-23 16:05:35,903 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:05:35,903 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:05:35,903 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:05:35,903 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 63 to run in 10 seconds
2017-10-23 16:05:45,515 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkNodeRequestSpike with id 57
2017-10-23 16:05:45,515 | DEBUG    | node.py              (2026) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:05:45,515 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:05:45,515 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 64 to run in 60 seconds
2017-10-23 16:05:45,515 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 9
2017-10-23 16:05:45,516 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:05:45,516 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 10 to run in 60 seconds
2017-10-23 16:05:45,808 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:05:45,808 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:05:45,808 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:05:45,808 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:05:45,908 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 63
2017-10-23 16:05:45,909 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:05:45,909 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:05:45,909 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.0015339007205772 is acceptable.
2017-10-23 16:05:45,909 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:05:45,909 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:05:45,909 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:05:45,909 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 65 to run in 10 seconds
2017-10-23 16:05:54,247 | TRACE    | zstack.py            ( 472) | _receiveFromListener | Node4C got 1 messages through listener
2017-10-23 16:05:54,250 | TRACE    | signing.py           (  98) | serializeMsg | serialized msg {'operation': {'timeout': 10, 'type': '109', 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'action': 'start', 'justification': None, 'force': True, 'version': '1.1.37'}, 'reqId': 1508774754234674, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:action:start|force:True|justification:|name:upgrade-1137|schedule:4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA:2017-10-17T11:35:00.000000+00:00|4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc:2017-10-17T11:40:00.000000+00:00|8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb:2017-10-17T11:25:00.000000+00:00|DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya:2017-10-17T11:30:00.000000+00:00|Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv:2017-10-17T11:20:00.000000+00:00|sha256:f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3|timeout:10|type:109|version:1.1.37|reqId:1508774754234674
2017-10-23 16:05:54,252 | TRACE    | signing.py           (  98) | serializeMsg | serialized msg {'operation': {'timeout': 10, 'type': '109', 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'action': 'start', 'justification': None, 'force': True, 'version': '1.1.37'}, 'reqId': 1508774754234674, 'signature': '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:action:start|force:True|justification:|name:upgrade-1137|schedule:4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA:2017-10-17T11:35:00.000000+00:00|4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc:2017-10-17T11:40:00.000000+00:00|8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb:2017-10-17T11:25:00.000000+00:00|DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya:2017-10-17T11:30:00.000000+00:00|Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv:2017-10-17T11:20:00.000000+00:00|sha256:f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3|timeout:10|type:109|version:1.1.37|reqId:1508774754234674
2017-10-23 16:05:54,252 | INFO     | node.py              (2250) | verifySignature | Node4 authenticated V4SGRU86Z58d6TV7PBUe6f signature on  request 1508774754234674
2017-10-23 16:05:54,253 | TRACE    | node.py              (1444) | validateClientMsg | Node4C received CLIENT message: SafeRequest: {'operation': {'timeout': 10, 'type': '109', 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'action': 'start', 'justification': None, 'force': True, 'version': '1.1.37'}, 'reqId': 1508774754234674, 'signature': '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}
2017-10-23 16:05:54,253 | DISPLAY  | node.py              (1490) | processClientInBox | Node4C processing b'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*' request SafeRequest: {'operation': {'timeout': 10, 'type': '109', 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'action': 'start', 'justification': None, 'force': True, 'version': '1.1.37'}, 'reqId': 1508774754234674, 'signature': '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}
2017-10-23 16:05:54,253 | INFO     | upgrader.py          ( 308) | _scheduleUpgrade | 4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA's upgrader processing upgrade for version 1.1.37
2017-10-23 16:05:54,254 | INFO     | upgrader.py          ( 369) | _callUpgradeAgent | 4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA's upgrader calling agent for upgrade
2017-10-23 16:05:54,263 | DEBUG    | node.py              (1751) | processRequest | Node4 received client request: SafeRequest: {'operation': {'timeout': 10, 'type': '109', 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'action': 'start', 'justification': None, 'force': True, 'version': '1.1.37'}, 'reqId': 1508774754234674, 'signature': '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} from b'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*'
2017-10-23 16:05:54,263 | TRACE    | propagator.py        ( 159) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'timeout': 10, 'type': '109', 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'action': 'start', 'justification': None, 'force': True, 'version': '1.1.37'}, 'reqId': 1508774754234674, 'signature': '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}
2017-10-23 16:05:54,263 | INFO     | propagator.py        ( 145) | propagate | Node4 propagating V4SGRU86Z58d6TV7PBUe6f request 1508774754234674 from client b'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*'
2017-10-23 16:05:54,263 | DEBUG    | node.py              (2593) | send | Node4 sending message PROPAGATE{'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*', 'request': {'operation': {'timeout': 10, 'type': '109', 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'action': 'start', 'justification': None, 'force': True, 'version': '1.1.37'}, 'reqId': 1508774754234674, 'signature': '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}} to all recipients: ['Node2', 'Node3', 'Node1']
2017-10-23 16:05:54,263 | DEBUG    | propagator.py        ( 241) | tryForwarding | Node4 not forwarding request SafeRequest: {'operation': {'timeout': 10, 'type': '109', 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'action': 'start', 'justification': None, 'force': True, 'version': '1.1.37'}, 'reqId': 1508774754234674, 'signature': '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} to its replicas since not finalised
2017-10-23 16:05:54,264 | TRACE    | zstack.py            ( 726) | transmitThroughListener | Node4C transmitting b'{"reqId":1508774754234674,"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f"}' to b'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*' through listener socket
2017-10-23 16:05:54,269 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"op":"PROPAGATE","senderClient":"xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE\\/:%0nV5Wx*","request":{"operation":{"timeout":10,"type":"109","name":"upgrade-1137","schedule":{"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb":"2017-10-17T11:25:00.000000+00:00","4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA":"2017-10-17T11:35:00.000000+00:00","4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc":"2017-10-17T11:40:00.000000+00:00","Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv":"2017-10-17T11:20:00.000000+00:00","DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya":"2017-10-17T11:30:00.000000+00:00"},"sha256":"f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3","action":"start","justification":null,"force":true,"version":"1.1.37"},"reqId":1508774754234674,"signature":"3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ","identifier":"V4SGRU86Z58d6TV7PBUe6f"}}' to Node2
2017-10-23 16:05:54,270 | TRACE    | batched.py           (  85) | flushOutBoxes | Node4 sending msg b'{"op":"PROPAGATE","senderClient":"xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE\\/:%0nV5Wx*","request":{"operation":{"timeout":10,"type":"109","name":"upgrade-1137","schedule":{"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb":"2017-10-17T11:25:00.000000+00:00","4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA":"2017-10-17T11:35:00.000000+00:00","4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc":"2017-10-17T11:40:00.000000+00:00","Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv":"2017-10-17T11:20:00.000000+00:00","DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya":"2017-10-17T11:30:00.000000+00:00"},"sha256":"f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3","action":"start","justification":null,"force":true,"version":"1.1.37"},"reqId":1508774754234674,"signature":"3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ","identifier":"V4SGRU86Z58d6TV7PBUe6f"}}' to Node2
2017-10-23 16:05:54,270 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"op":"PROPAGATE","senderClient":"xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE\\/:%0nV5Wx*","request":{"operation":{"timeout":10,"type":"109","name":"upgrade-1137","schedule":{"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb":"2017-10-17T11:25:00.000000+00:00","4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA":"2017-10-17T11:35:00.000000+00:00","4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc":"2017-10-17T11:40:00.000000+00:00","Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv":"2017-10-17T11:20:00.000000+00:00","DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya":"2017-10-17T11:30:00.000000+00:00"},"sha256":"f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3","action":"start","justification":null,"force":true,"version":"1.1.37"},"reqId":1508774754234674,"signature":"3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ","identifier":"V4SGRU86Z58d6TV7PBUe6f"}}' to Node3
2017-10-23 16:05:54,270 | TRACE    | batched.py           (  85) | flushOutBoxes | Node4 sending msg b'{"op":"PROPAGATE","senderClient":"xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE\\/:%0nV5Wx*","request":{"operation":{"timeout":10,"type":"109","name":"upgrade-1137","schedule":{"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb":"2017-10-17T11:25:00.000000+00:00","4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA":"2017-10-17T11:35:00.000000+00:00","4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc":"2017-10-17T11:40:00.000000+00:00","Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv":"2017-10-17T11:20:00.000000+00:00","DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya":"2017-10-17T11:30:00.000000+00:00"},"sha256":"f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3","action":"start","justification":null,"force":true,"version":"1.1.37"},"reqId":1508774754234674,"signature":"3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ","identifier":"V4SGRU86Z58d6TV7PBUe6f"}}' to Node3
2017-10-23 16:05:54,270 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"op":"PROPAGATE","senderClient":"xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE\\/:%0nV5Wx*","request":{"operation":{"timeout":10,"type":"109","name":"upgrade-1137","schedule":{"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb":"2017-10-17T11:25:00.000000+00:00","4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA":"2017-10-17T11:35:00.000000+00:00","4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc":"2017-10-17T11:40:00.000000+00:00","Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv":"2017-10-17T11:20:00.000000+00:00","DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya":"2017-10-17T11:30:00.000000+00:00"},"sha256":"f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3","action":"start","justification":null,"force":true,"version":"1.1.37"},"reqId":1508774754234674,"signature":"3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ","identifier":"V4SGRU86Z58d6TV7PBUe6f"}}' to Node1
2017-10-23 16:05:54,270 | TRACE    | batched.py           (  85) | flushOutBoxes | Node4 sending msg b'{"op":"PROPAGATE","senderClient":"xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE\\/:%0nV5Wx*","request":{"operation":{"timeout":10,"type":"109","name":"upgrade-1137","schedule":{"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb":"2017-10-17T11:25:00.000000+00:00","4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA":"2017-10-17T11:35:00.000000+00:00","4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc":"2017-10-17T11:40:00.000000+00:00","Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv":"2017-10-17T11:20:00.000000+00:00","DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya":"2017-10-17T11:30:00.000000+00:00"},"sha256":"f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3","action":"start","justification":null,"force":true,"version":"1.1.37"},"reqId":1508774754234674,"signature":"3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ","identifier":"V4SGRU86Z58d6TV7PBUe6f"}}' to Node1
2017-10-23 16:05:54,272 | TRACE    | zstack.py            ( 472) | _receiveFromListener | Node4 got 1 messages through listener
2017-10-23 16:05:54,273 | TRACE    | signing.py           (  98) | serializeMsg | serialized msg {'operation': {'type': '109', 'timeout': 10, 'force': True, 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'action': 'start', 'justification': None, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'version': '1.1.37'}, 'reqId': 1508774754234674, 'signature': '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:action:start|force:True|justification:|name:upgrade-1137|schedule:4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA:2017-10-17T11:35:00.000000+00:00|4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc:2017-10-17T11:40:00.000000+00:00|8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb:2017-10-17T11:25:00.000000+00:00|DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya:2017-10-17T11:30:00.000000+00:00|Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv:2017-10-17T11:20:00.000000+00:00|sha256:f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3|timeout:10|type:109|version:1.1.37|reqId:1508774754234674
2017-10-23 16:05:54,273 | INFO     | node.py              (2250) | verifySignature | Node4 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate  request 1508774754234674
2017-10-23 16:05:54,273 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node3: PROPAGATE{'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*', 'request': {'operation': {'type': '109', 'timeout': 10, 'force': True, 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'action': 'start', 'justification': None, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'version': '1.1.37'}, 'reqId': 1508774754234674, 'signature': '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}
2017-10-23 16:05:54,273 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'op': 'PROPAGATE', 'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*', 'request': {'operation': {'type': '109', 'timeout': 10, 'force': True, 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'action': 'start', 'justification': None, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'version': '1.1.37'}, 'reqId': 1508774754234674, 'signature': '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node3')
2017-10-23 16:05:54,273 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox PROPAGATE{'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*', 'request': {'operation': {'type': '109', 'timeout': 10, 'force': True, 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'action': 'start', 'justification': None, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'version': '1.1.37'}, 'reqId': 1508774754234674, 'signature': '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}
2017-10-23 16:05:54,282 | DEBUG    | node.py              (1799) | processPropagate | Node Node4 received propagated request: PROPAGATE{'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*', 'request': {'operation': {'type': '109', 'timeout': 10, 'force': True, 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'action': 'start', 'justification': None, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'version': '1.1.37'}, 'reqId': 1508774754234674, 'signature': '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}
2017-10-23 16:05:54,282 | TRACE    | signing.py           (  98) | serializeMsg | serialized msg {'operation': {'type': '109', 'timeout': 10, 'force': True, 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'action': 'start', 'justification': None, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'version': '1.1.37'}, 'reqId': 1508774754234674, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:action:start|force:True|justification:|name:upgrade-1137|schedule:4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA:2017-10-17T11:35:00.000000+00:00|4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc:2017-10-17T11:40:00.000000+00:00|8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb:2017-10-17T11:25:00.000000+00:00|DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya:2017-10-17T11:30:00.000000+00:00|Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv:2017-10-17T11:20:00.000000+00:00|sha256:f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3|timeout:10|type:109|version:1.1.37|reqId:1508774754234674
2017-10-23 16:05:54,283 | TRACE    | propagator.py        ( 138) | propagate | Node4 already propagated SafeRequest: {'operation': {'type': '109', 'timeout': 10, 'force': True, 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'action': 'start', 'justification': None, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'version': '1.1.37'}, 'reqId': 1508774754234674, 'signature': '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}
2017-10-23 16:05:54,283 | DEBUG    | propagator.py        ( 208) | forward | Node4 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1508774754234674) to 2 replicas
2017-10-23 16:05:54,286 | TRACE    | zstack.py            ( 472) | _receiveFromListener | Node4 got 2 messages through listener
2017-10-23 16:05:54,286 | TRACE    | signing.py           (  98) | serializeMsg | serialized msg {'operation': {'force': True, 'timeout': 10, 'type': '109', 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'action': 'start', 'justification': None, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'version': '1.1.37'}, 'reqId': 1508774754234674, 'signature': '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:action:start|force:True|justification:|name:upgrade-1137|schedule:4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA:2017-10-17T11:35:00.000000+00:00|4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc:2017-10-17T11:40:00.000000+00:00|8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb:2017-10-17T11:25:00.000000+00:00|DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya:2017-10-17T11:30:00.000000+00:00|Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv:2017-10-17T11:20:00.000000+00:00|sha256:f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3|timeout:10|type:109|version:1.1.37|reqId:1508774754234674
2017-10-23 16:05:54,287 | INFO     | node.py              (2250) | verifySignature | Node4 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate  request 1508774754234674
2017-10-23 16:05:54,287 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node2: PROPAGATE{'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*', 'request': {'operation': {'force': True, 'timeout': 10, 'type': '109', 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'action': 'start', 'justification': None, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'version': '1.1.37'}, 'reqId': 1508774754234674, 'signature': '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}
2017-10-23 16:05:54,287 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'op': 'PROPAGATE', 'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*', 'request': {'operation': {'force': True, 'timeout': 10, 'type': '109', 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'action': 'start', 'justification': None, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'version': '1.1.37'}, 'reqId': 1508774754234674, 'signature': '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node2')
2017-10-23 16:05:54,287 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox PROPAGATE{'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*', 'request': {'operation': {'force': True, 'timeout': 10, 'type': '109', 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'action': 'start', 'justification': None, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'version': '1.1.37'}, 'reqId': 1508774754234674, 'signature': '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}
2017-10-23 16:05:54,287 | TRACE    | signing.py           (  98) | serializeMsg | serialized msg {'operation': {'version': '1.1.37', 'timeout': 10, 'force': True, 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'action': 'start', 'justification': None, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'type': '109'}, 'reqId': 1508774754234674, 'signature': '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:action:start|force:True|justification:|name:upgrade-1137|schedule:4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA:2017-10-17T11:35:00.000000+00:00|4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc:2017-10-17T11:40:00.000000+00:00|8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb:2017-10-17T11:25:00.000000+00:00|DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya:2017-10-17T11:30:00.000000+00:00|Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv:2017-10-17T11:20:00.000000+00:00|sha256:f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3|timeout:10|type:109|version:1.1.37|reqId:1508774754234674
2017-10-23 16:05:54,288 | INFO     | node.py              (2250) | verifySignature | Node4 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate  request 1508774754234674
2017-10-23 16:05:54,288 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node1: PROPAGATE{'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*', 'request': {'operation': {'version': '1.1.37', 'timeout': 10, 'force': True, 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'action': 'start', 'justification': None, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'type': '109'}, 'reqId': 1508774754234674, 'signature': '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}
2017-10-23 16:05:54,288 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'op': 'PROPAGATE', 'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*', 'request': {'operation': {'version': '1.1.37', 'timeout': 10, 'force': True, 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'action': 'start', 'justification': None, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'type': '109'}, 'reqId': 1508774754234674, 'signature': '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}, 'Node1')
2017-10-23 16:05:54,288 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox PROPAGATE{'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*', 'request': {'operation': {'version': '1.1.37', 'timeout': 10, 'force': True, 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'action': 'start', 'justification': None, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'type': '109'}, 'reqId': 1508774754234674, 'signature': '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}
2017-10-23 16:05:54,288 | DEBUG    | node.py              (1799) | processPropagate | Node Node4 received propagated request: PROPAGATE{'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*', 'request': {'operation': {'force': True, 'timeout': 10, 'type': '109', 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'action': 'start', 'justification': None, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'version': '1.1.37'}, 'reqId': 1508774754234674, 'signature': '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}
2017-10-23 16:05:54,289 | TRACE    | signing.py           (  98) | serializeMsg | serialized msg {'operation': {'force': True, 'timeout': 10, 'type': '109', 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'action': 'start', 'justification': None, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'version': '1.1.37'}, 'reqId': 1508774754234674, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:action:start|force:True|justification:|name:upgrade-1137|schedule:4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA:2017-10-17T11:35:00.000000+00:00|4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc:2017-10-17T11:40:00.000000+00:00|8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb:2017-10-17T11:25:00.000000+00:00|DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya:2017-10-17T11:30:00.000000+00:00|Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv:2017-10-17T11:20:00.000000+00:00|sha256:f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3|timeout:10|type:109|version:1.1.37|reqId:1508774754234674
2017-10-23 16:05:54,289 | TRACE    | propagator.py        ( 138) | propagate | Node4 already propagated SafeRequest: {'operation': {'force': True, 'timeout': 10, 'type': '109', 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'action': 'start', 'justification': None, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'version': '1.1.37'}, 'reqId': 1508774754234674, 'signature': '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}
2017-10-23 16:05:54,289 | DEBUG    | propagator.py        ( 241) | tryForwarding | Node4 not forwarding request SafeRequest: {'operation': {'force': True, 'timeout': 10, 'type': '109', 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'action': 'start', 'justification': None, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'version': '1.1.37'}, 'reqId': 1508774754234674, 'signature': '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} to its replicas since already forwarded
2017-10-23 16:05:54,289 | DEBUG    | node.py              (1799) | processPropagate | Node Node4 received propagated request: PROPAGATE{'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*', 'request': {'operation': {'version': '1.1.37', 'timeout': 10, 'force': True, 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'action': 'start', 'justification': None, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'type': '109'}, 'reqId': 1508774754234674, 'signature': '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}}
2017-10-23 16:05:54,289 | TRACE    | signing.py           (  98) | serializeMsg | serialized msg {'operation': {'version': '1.1.37', 'timeout': 10, 'force': True, 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'action': 'start', 'justification': None, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'type': '109'}, 'reqId': 1508774754234674, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:action:start|force:True|justification:|name:upgrade-1137|schedule:4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA:2017-10-17T11:35:00.000000+00:00|4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc:2017-10-17T11:40:00.000000+00:00|8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb:2017-10-17T11:25:00.000000+00:00|DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya:2017-10-17T11:30:00.000000+00:00|Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv:2017-10-17T11:20:00.000000+00:00|sha256:f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3|timeout:10|type:109|version:1.1.37|reqId:1508774754234674
2017-10-23 16:05:54,289 | TRACE    | propagator.py        ( 138) | propagate | Node4 already propagated SafeRequest: {'operation': {'version': '1.1.37', 'timeout': 10, 'force': True, 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'action': 'start', 'justification': None, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'type': '109'}, 'reqId': 1508774754234674, 'signature': '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'}
2017-10-23 16:05:54,290 | DEBUG    | propagator.py        ( 241) | tryForwarding | Node4 not forwarding request SafeRequest: {'operation': {'version': '1.1.37', 'timeout': 10, 'force': True, 'name': 'upgrade-1137', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00'}, 'action': 'start', 'justification': None, 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'type': '109'}, 'reqId': 1508774754234674, 'signature': '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f'} to its replicas since already forwarded
2017-10-23 16:05:54,304 | DEBUG    | base_events.py       ( 681) | create_connection | connect <socket.socket fd=75, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('0.0.0.0', 0)> to ('127.0.0.1', 30003)
2017-10-23 16:05:54,306 | DEBUG    | base_events.py       (1270) | _run_once | poll 7.874 ms took 0.011 ms: 1 events
2017-10-23 16:05:54,313 | DEBUG    | base_events.py       ( 719) | create_connection | <socket.socket fd=75, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('127.0.0.1', 36130), raddr=('127.0.0.1', 30003)> connected to 127.0.0.1:'30003': (<_SelectorSocketTransport fd=75 read=polling write=<idle, bufsize=0>>, <asyncio.streams.StreamReaderProtocol object at 0x7f678023be10>)
2017-10-23 16:05:54,314 | TRACE    | has_action_queue.py  (  34) | _schedule | 4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA scheduling action partial(_declareTimeoutExceeded) with id 1 to run in 600 seconds
2017-10-23 16:05:54,331 | TRACE    | zstack.py            ( 472) | _receiveFromListener | Node4 got 2 messages through listener
2017-10-23 16:05:54,332 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node1: PREPREPARE{'ppSeqNo': 2, 'digest': '068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226', 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'instId': 0, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1508774754234674]], 'txnRootHash': 'DXRK9Y7vAfHsb2GbvkXUFggkmHfnFv2uf54gqUELAcmS', 'ledgerId': 2, 'discarded': 1, 'viewNo': 0, 'ppTime': 1508774754}
2017-10-23 16:05:54,332 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': 2, 'digest': '068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226', 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'instId': 0, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1508774754234674]], 'txnRootHash': 'DXRK9Y7vAfHsb2GbvkXUFggkmHfnFv2uf54gqUELAcmS', 'ledgerId': 2, 'discarded': 1, 'op': 'PREPREPARE', 'viewNo': 0, 'ppTime': 1508774754}, 'Node1')
2017-10-23 16:05:54,332 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox PREPREPARE{'ppSeqNo': 2, 'digest': '068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226', 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'instId': 0, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1508774754234674]], 'txnRootHash': 'DXRK9Y7vAfHsb2GbvkXUFggkmHfnFv2uf54gqUELAcmS', 'ledgerId': 2, 'discarded': 1, 'viewNo': 0, 'ppTime': 1508774754}
2017-10-23 16:05:54,332 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node2: PREPREPARE{'ppSeqNo': 2, 'digest': '068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226', 'stateRootHash': None, 'instId': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1508774754234674]], 'txnRootHash': None, 'ledgerId': 2, 'discarded': 1, 'viewNo': 0, 'ppTime': 1508774754}
2017-10-23 16:05:54,332 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': 2, 'digest': '068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226', 'stateRootHash': None, 'instId': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1508774754234674]], 'txnRootHash': None, 'ledgerId': 2, 'discarded': 1, 'op': 'PREPREPARE', 'viewNo': 0, 'ppTime': 1508774754}, 'Node2')
2017-10-23 16:05:54,333 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox PREPREPARE{'ppSeqNo': 2, 'digest': '068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226', 'stateRootHash': None, 'instId': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1508774754234674]], 'txnRootHash': None, 'ledgerId': 2, 'discarded': 1, 'viewNo': 0, 'ppTime': 1508774754}
2017-10-23 16:05:54,347 | DEBUG    | replica.py           ( 790) | processPrePrepare | Node4:0 received PRE-PREPARE(0, 2) from Node1:0 at 956254.709346045
2017-10-23 16:05:54,348 | DEBUG    | replica.py           (1002) | validate_pre_prepare | Node4:0 state root before processing PREPREPARE{'ppSeqNo': 2, 'digest': '068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226', 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'instId': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1508774754234674)], 'txnRootHash': 'DXRK9Y7vAfHsb2GbvkXUFggkmHfnFv2uf54gqUELAcmS', 'ledgerId': 2, 'discarded': 1, 'viewNo': 0, 'ppTime': 1508774754} is b'\xbc q\xa4\xde\x84o(W\x02D\x7f%\x89\xdd\x166x\xe0\x97*\x8a\x1b\r(\xb0N\xd5\xc0\x94T\x7f', GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn
2017-10-23 16:05:54,348 | DEBUG    | replica.py           ( 903) | doPrepare | Node4:0 Sending PREPARE(0, 2) at 956254.7103487
2017-10-23 16:05:54,348 | INFO     | replica.py           (2064) | send | Node4:0 sending Prepare
2017-10-23 16:05:54,348 | TRACE    | replica.py           (2065) | send | Node4:0 sending PREPARE{'ppSeqNo': 2, 'digest': '068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226', 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'instId': 0, 'txnRootHash': 'DXRK9Y7vAfHsb2GbvkXUFggkmHfnFv2uf54gqUELAcmS', 'viewNo': 0, 'ppTime': 1508774754}
2017-10-23 16:05:54,349 | DEBUG    | replica.py           ( 886) | tryCommit | Node4:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'ppSeqNo': 2, 'digest': '068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226', 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'instId': 0, 'txnRootHash': 'DXRK9Y7vAfHsb2GbvkXUFggkmHfnFv2uf54gqUELAcmS', 'viewNo': 0, 'ppTime': 1508774754}
2017-10-23 16:05:54,349 | DEBUG    | replica.py           ( 594) | trackBatches | Node4:0 tracking batch for PREPREPARE{'ppSeqNo': 2, 'digest': '068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226', 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'instId': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1508774754234674)], 'txnRootHash': 'DXRK9Y7vAfHsb2GbvkXUFggkmHfnFv2uf54gqUELAcmS', 'ledgerId': 2, 'discarded': 1, 'viewNo': 0, 'ppTime': 1508774754} with state root b'\xbc q\xa4\xde\x84o(W\x02D\x7f%\x89\xdd\x166x\xe0\x97*\x8a\x1b\r(\xb0N\xd5\xc0\x94T\x7f'
2017-10-23 16:05:54,349 | DEBUG    | replica.py           ( 809) | processPrePrepare | Node4:0 processed incoming PRE-PREPARE(0, 2)
2017-10-23 16:05:54,349 | DEBUG    | replica.py           ( 790) | processPrePrepare | Node4:1 received PRE-PREPARE(0, 2) from Node2:1 at 956254.711093104
2017-10-23 16:05:54,349 | DEBUG    | replica.py           ( 903) | doPrepare | Node4:1 Sending PREPARE(0, 2) at 956254.71136197
2017-10-23 16:05:54,349 | INFO     | replica.py           (2064) | send | Node4:1 sending Prepare
2017-10-23 16:05:54,349 | TRACE    | replica.py           (2065) | send | Node4:1 sending PREPARE{'ppSeqNo': 2, 'digest': '068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226', 'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'ppTime': 1508774754}
2017-10-23 16:05:54,349 | DEBUG    | replica.py           ( 886) | tryCommit | Node4:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'ppSeqNo': 2, 'digest': '068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226', 'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'ppTime': 1508774754}
2017-10-23 16:05:54,349 | DEBUG    | replica.py           ( 594) | trackBatches | Node4:1 tracking batch for PREPREPARE{'ppSeqNo': 2, 'digest': '068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226', 'stateRootHash': None, 'instId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1508774754234674)], 'txnRootHash': None, 'ledgerId': 2, 'discarded': 1, 'viewNo': 0, 'ppTime': 1508774754} with state root None
2017-10-23 16:05:54,350 | DEBUG    | replica.py           ( 809) | processPrePrepare | Node4:1 processed incoming PRE-PREPARE(0, 2)
2017-10-23 16:05:54,360 | TRACE    | zstack.py            ( 472) | _receiveFromListener | Node4 got 3 messages through listener
2017-10-23 16:05:54,361 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node2: PREPARE{'ppSeqNo': 2, 'digest': '068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226', 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'instId': 0, 'txnRootHash': 'DXRK9Y7vAfHsb2GbvkXUFggkmHfnFv2uf54gqUELAcmS', 'viewNo': 0, 'ppTime': 1508774754}
2017-10-23 16:05:54,361 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': 2, 'digest': '068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226', 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'instId': 0, 'txnRootHash': 'DXRK9Y7vAfHsb2GbvkXUFggkmHfnFv2uf54gqUELAcmS', 'op': 'PREPARE', 'ppTime': 1508774754, 'viewNo': 0}, 'Node2')
2017-10-23 16:05:54,361 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox PREPARE{'ppSeqNo': 2, 'digest': '068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226', 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'instId': 0, 'txnRootHash': 'DXRK9Y7vAfHsb2GbvkXUFggkmHfnFv2uf54gqUELAcmS', 'viewNo': 0, 'ppTime': 1508774754}
2017-10-23 16:05:54,361 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node1: PREPARE{'ppSeqNo': 2, 'digest': '068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226', 'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'ppTime': 1508774754}
2017-10-23 16:05:54,361 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': 2, 'digest': '068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226', 'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'op': 'PREPARE', 'viewNo': 0, 'ppTime': 1508774754}, 'Node1')
2017-10-23 16:05:54,361 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox PREPARE{'ppSeqNo': 2, 'digest': '068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226', 'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'ppTime': 1508774754}
2017-10-23 16:05:54,362 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node3: BATCH{'messages': ['{"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","ppSeqNo":2,"viewNo":0,"op":"PREPARE","ppTime":1508774754,"digest":"068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226","instId":0,"txnRootHash":"DXRK9Y7vAfHsb2GbvkXUFggkmHfnFv2uf54gqUELAcmS"}', '{"stateRootHash":null,"ppSeqNo":2,"viewNo":0,"op":"PREPARE","ppTime":1508774754,"digest":"068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226","instId":1,"txnRootHash":null}'], 'signature': None}
2017-10-23 16:05:54,362 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'messages': ['{"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","ppSeqNo":2,"viewNo":0,"op":"PREPARE","ppTime":1508774754,"digest":"068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226","instId":0,"txnRootHash":"DXRK9Y7vAfHsb2GbvkXUFggkmHfnFv2uf54gqUELAcmS"}', '{"stateRootHash":null,"ppSeqNo":2,"viewNo":0,"op":"PREPARE","ppTime":1508774754,"digest":"068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226","instId":1,"txnRootHash":null}'], 'signature': None}, 'Node3')
2017-10-23 16:05:54,362 | DEBUG    | node.py              (1328) | unpackNodeMsg | Node4 processing a batch BATCH{'messages': ['{"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","ppSeqNo":2,"viewNo":0,"op":"PREPARE","ppTime":1508774754,"digest":"068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226","instId":0,"txnRootHash":"DXRK9Y7vAfHsb2GbvkXUFggkmHfnFv2uf54gqUELAcmS"}', '{"stateRootHash":null,"ppSeqNo":2,"viewNo":0,"op":"PREPARE","ppTime":1508774754,"digest":"068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226","instId":1,"txnRootHash":null}'], 'signature': None}
2017-10-23 16:05:54,362 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node3: PREPARE{'ppSeqNo': 2, 'digest': '068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226', 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'instId': 0, 'txnRootHash': 'DXRK9Y7vAfHsb2GbvkXUFggkmHfnFv2uf54gqUELAcmS', 'viewNo': 0, 'ppTime': 1508774754}
2017-10-23 16:05:54,362 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': 2, 'digest': '068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226', 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'instId': 0, 'txnRootHash': 'DXRK9Y7vAfHsb2GbvkXUFggkmHfnFv2uf54gqUELAcmS', 'op': 'PREPARE', 'viewNo': 0, 'ppTime': 1508774754}, 'Node3')
2017-10-23 16:05:54,362 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox PREPARE{'ppSeqNo': 2, 'digest': '068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226', 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'instId': 0, 'txnRootHash': 'DXRK9Y7vAfHsb2GbvkXUFggkmHfnFv2uf54gqUELAcmS', 'viewNo': 0, 'ppTime': 1508774754}
2017-10-23 16:05:54,362 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node3: PREPARE{'ppSeqNo': 2, 'digest': '068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226', 'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'ppTime': 1508774754}
2017-10-23 16:05:54,362 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': 2, 'digest': '068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226', 'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'op': 'PREPARE', 'viewNo': 0, 'ppTime': 1508774754}, 'Node3')
2017-10-23 16:05:54,363 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox PREPARE{'ppSeqNo': 2, 'digest': '068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226', 'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'ppTime': 1508774754}
2017-10-23 16:05:54,372 | DEBUG    | node.py              (2593) | send | Node4 sending message PREPARE{'ppSeqNo': 2, 'digest': '068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226', 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'instId': 0, 'txnRootHash': 'DXRK9Y7vAfHsb2GbvkXUFggkmHfnFv2uf54gqUELAcmS', 'viewNo': 0, 'ppTime': 1508774754} to all recipients: ['Node2', 'Node3', 'Node1']
2017-10-23 16:05:54,372 | DEBUG    | node.py              (2593) | send | Node4 sending message PREPARE{'ppSeqNo': 2, 'digest': '068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226', 'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'ppTime': 1508774754} to all recipients: ['Node2', 'Node3', 'Node1']
2017-10-23 16:05:54,373 | DEBUG    | replica.py           ( 834) | processPrepare | Node4:0 received PREPARE(0, 2) from Node2:0
2017-10-23 16:05:54,373 | DEBUG    | replica.py           ( 922) | doCommit | Node4:0 Sending COMMIT(0, 2) at 956254.735098054
2017-10-23 16:05:54,373 | INFO     | replica.py           (2064) | send | Node4:0 sending Commit
2017-10-23 16:05:54,373 | TRACE    | replica.py           (2065) | send | Node4:0 sending COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 0}
2017-10-23 16:05:54,373 | DEBUG    | replica.py           ( 898) | tryOrder | Node4:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 0} commits where f is 1
2017-10-23 16:05:54,373 | DEBUG    | replica.py           ( 845) | processPrepare | Node4:0 processed incoming PREPARE (0, 2)
2017-10-23 16:05:54,373 | DEBUG    | replica.py           ( 834) | processPrepare | Node4:0 received PREPARE(0, 2) from Node3:0
2017-10-23 16:05:54,373 | DEBUG    | replica.py           ( 886) | tryCommit | Node4:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'ppSeqNo': 2, 'digest': '068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226', 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'instId': 0, 'txnRootHash': 'DXRK9Y7vAfHsb2GbvkXUFggkmHfnFv2uf54gqUELAcmS', 'viewNo': 0, 'ppTime': 1508774754}
2017-10-23 16:05:54,374 | DEBUG    | replica.py           ( 845) | processPrepare | Node4:0 processed incoming PREPARE (0, 2)
2017-10-23 16:05:54,374 | DEBUG    | replica.py           ( 834) | processPrepare | Node4:1 received PREPARE(0, 2) from Node1:1
2017-10-23 16:05:54,388 | DEBUG    | replica.py           ( 922) | doCommit | Node4:1 Sending COMMIT(0, 2) at 956254.736068004
2017-10-23 16:05:54,388 | INFO     | replica.py           (2064) | send | Node4:1 sending Commit
2017-10-23 16:05:54,388 | TRACE    | replica.py           (2065) | send | Node4:1 sending COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 0}
2017-10-23 16:05:54,388 | DEBUG    | replica.py           ( 898) | tryOrder | Node4:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 0} commits where f is 1
2017-10-23 16:05:54,388 | DEBUG    | replica.py           ( 845) | processPrepare | Node4:1 processed incoming PREPARE (0, 2)
2017-10-23 16:05:54,388 | DEBUG    | replica.py           ( 834) | processPrepare | Node4:1 received PREPARE(0, 2) from Node3:1
2017-10-23 16:05:54,388 | DEBUG    | replica.py           ( 886) | tryCommit | Node4:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'ppSeqNo': 2, 'digest': '068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226', 'stateRootHash': None, 'instId': 1, 'txnRootHash': None, 'viewNo': 0, 'ppTime': 1508774754}
2017-10-23 16:05:54,388 | DEBUG    | replica.py           ( 845) | processPrepare | Node4:1 processed incoming PREPARE (0, 2)
2017-10-23 16:05:54,389 | TRACE    | zstack.py            ( 472) | _receiveFromListener | Node4 got 3 messages through listener
2017-10-23 16:05:54,390 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node3: BATCH{'messages': ['{"instId":0,"ppSeqNo":2,"viewNo":0,"op":"COMMIT"}', '{"instId":1,"ppSeqNo":2,"viewNo":0,"op":"COMMIT"}'], 'signature': None}
2017-10-23 16:05:54,390 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'messages': ['{"instId":0,"ppSeqNo":2,"viewNo":0,"op":"COMMIT"}', '{"instId":1,"ppSeqNo":2,"viewNo":0,"op":"COMMIT"}'], 'signature': None}, 'Node3')
2017-10-23 16:05:54,390 | DEBUG    | node.py              (1328) | unpackNodeMsg | Node4 processing a batch BATCH{'messages': ['{"instId":0,"ppSeqNo":2,"viewNo":0,"op":"COMMIT"}', '{"instId":1,"ppSeqNo":2,"viewNo":0,"op":"COMMIT"}'], 'signature': None}
2017-10-23 16:05:54,390 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node3: COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 0}
2017-10-23 16:05:54,390 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': 2, 'op': 'COMMIT', 'instId': 0, 'viewNo': 0}, 'Node3')
2017-10-23 16:05:54,390 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 0}
2017-10-23 16:05:54,393 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node3: COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 0}
2017-10-23 16:05:54,394 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': 2, 'op': 'COMMIT', 'instId': 1, 'viewNo': 0}, 'Node3')
2017-10-23 16:05:54,394 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 0}
2017-10-23 16:05:54,394 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node1: BATCH{'messages': ['{"op":"COMMIT","instId":0,"ppSeqNo":2,"viewNo":0}', '{"op":"COMMIT","instId":1,"ppSeqNo":2,"viewNo":0}'], 'signature': None}
2017-10-23 16:05:54,394 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'messages': ['{"op":"COMMIT","instId":0,"ppSeqNo":2,"viewNo":0}', '{"op":"COMMIT","instId":1,"ppSeqNo":2,"viewNo":0}'], 'signature': None}, 'Node1')
2017-10-23 16:05:54,394 | DEBUG    | node.py              (1328) | unpackNodeMsg | Node4 processing a batch BATCH{'messages': ['{"op":"COMMIT","instId":0,"ppSeqNo":2,"viewNo":0}', '{"op":"COMMIT","instId":1,"ppSeqNo":2,"viewNo":0}'], 'signature': None}
2017-10-23 16:05:54,394 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node1: COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 0}
2017-10-23 16:05:54,394 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': 2, 'op': 'COMMIT', 'instId': 0, 'viewNo': 0}, 'Node1')
2017-10-23 16:05:54,394 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 0}
2017-10-23 16:05:54,395 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node1: COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 0}
2017-10-23 16:05:54,395 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': 2, 'op': 'COMMIT', 'instId': 1, 'viewNo': 0}, 'Node1')
2017-10-23 16:05:54,395 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 0}
2017-10-23 16:05:54,395 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node2: BATCH{'messages': ['{"viewNo":0,"instId":0,"ppSeqNo":2,"op":"COMMIT"}', '{"viewNo":0,"instId":1,"ppSeqNo":2,"op":"COMMIT"}'], 'signature': None}
2017-10-23 16:05:54,395 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'messages': ['{"viewNo":0,"instId":0,"ppSeqNo":2,"op":"COMMIT"}', '{"viewNo":0,"instId":1,"ppSeqNo":2,"op":"COMMIT"}'], 'signature': None}, 'Node2')
2017-10-23 16:05:54,395 | DEBUG    | node.py              (1328) | unpackNodeMsg | Node4 processing a batch BATCH{'messages': ['{"viewNo":0,"instId":0,"ppSeqNo":2,"op":"COMMIT"}', '{"viewNo":0,"instId":1,"ppSeqNo":2,"op":"COMMIT"}'], 'signature': None}
2017-10-23 16:05:54,395 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node2: COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 0}
2017-10-23 16:05:54,395 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': 2, 'op': 'COMMIT', 'instId': 0, 'viewNo': 0}, 'Node2')
2017-10-23 16:05:54,395 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 0}
2017-10-23 16:05:54,395 | DEBUG    | node.py              (1313) | validateNodeMsg | Node4 received node message from Node2: COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 0}
2017-10-23 16:05:54,396 | INFO     | node.py              (1275) | handleOneNodeMsg | Node4 msg validated ({'ppSeqNo': 2, 'op': 'COMMIT', 'instId': 1, 'viewNo': 0}, 'Node2')
2017-10-23 16:05:54,396 | DEBUG    | node.py              (1342) | postToNodeInBox | Node4 appending to nodeInbox COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 0}
2017-10-23 16:05:54,400 | DEBUG    | batched.py           (  89) | flushOutBoxes | Node4 batching 2 msgs to Node2 into one transmission
2017-10-23 16:05:54,400 | TRACE    | batched.py           (  90) | flushOutBoxes |     messages: deque([b'{"ppSeqNo":2,"txnRootHash":"DXRK9Y7vAfHsb2GbvkXUFggkmHfnFv2uf54gqUELAcmS","op":"PREPARE","viewNo":0,"digest":"068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226","stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","instId":0,"ppTime":1508774754}', b'{"ppSeqNo":2,"txnRootHash":null,"op":"PREPARE","viewNo":0,"digest":"068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226","stateRootHash":null,"instId":1,"ppTime":1508774754}'])
2017-10-23 16:05:54,400 | TRACE    | batched.py           (  98) | flushOutBoxes | Node4 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"ppSeqNo\\":2,\\"txnRootHash\\":\\"DXRK9Y7vAfHsb2GbvkXUFggkmHfnFv2uf54gqUELAcmS\\",\\"op\\":\\"PREPARE\\",\\"viewNo\\":0,\\"digest\\":\\"068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226\\",\\"stateRootHash\\":\\"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA\\",\\"instId\\":0,\\"ppTime\\":1508774754}","{\\"ppSeqNo\\":2,\\"txnRootHash\\":null,\\"op\\":\\"PREPARE\\",\\"viewNo\\":0,\\"digest\\":\\"068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226\\",\\"stateRootHash\\":null,\\"instId\\":1,\\"ppTime\\":1508774754}"],"signature":null}'
2017-10-23 16:05:54,400 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"op":"BATCH","messages":["{\\"ppSeqNo\\":2,\\"txnRootHash\\":\\"DXRK9Y7vAfHsb2GbvkXUFggkmHfnFv2uf54gqUELAcmS\\",\\"op\\":\\"PREPARE\\",\\"viewNo\\":0,\\"digest\\":\\"068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226\\",\\"stateRootHash\\":\\"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA\\",\\"instId\\":0,\\"ppTime\\":1508774754}","{\\"ppSeqNo\\":2,\\"txnRootHash\\":null,\\"op\\":\\"PREPARE\\",\\"viewNo\\":0,\\"digest\\":\\"068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226\\",\\"stateRootHash\\":null,\\"instId\\":1,\\"ppTime\\":1508774754}"],"signature":null}' to Node2
2017-10-23 16:05:54,400 | DEBUG    | batched.py           (  89) | flushOutBoxes | Node4 batching 2 msgs to Node3 into one transmission
2017-10-23 16:05:54,400 | TRACE    | batched.py           (  90) | flushOutBoxes |     messages: deque([b'{"ppSeqNo":2,"txnRootHash":"DXRK9Y7vAfHsb2GbvkXUFggkmHfnFv2uf54gqUELAcmS","op":"PREPARE","viewNo":0,"digest":"068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226","stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","instId":0,"ppTime":1508774754}', b'{"ppSeqNo":2,"txnRootHash":null,"op":"PREPARE","viewNo":0,"digest":"068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226","stateRootHash":null,"instId":1,"ppTime":1508774754}'])
2017-10-23 16:05:54,401 | TRACE    | batched.py           (  98) | flushOutBoxes | Node4 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"ppSeqNo\\":2,\\"txnRootHash\\":\\"DXRK9Y7vAfHsb2GbvkXUFggkmHfnFv2uf54gqUELAcmS\\",\\"op\\":\\"PREPARE\\",\\"viewNo\\":0,\\"digest\\":\\"068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226\\",\\"stateRootHash\\":\\"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA\\",\\"instId\\":0,\\"ppTime\\":1508774754}","{\\"ppSeqNo\\":2,\\"txnRootHash\\":null,\\"op\\":\\"PREPARE\\",\\"viewNo\\":0,\\"digest\\":\\"068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226\\",\\"stateRootHash\\":null,\\"instId\\":1,\\"ppTime\\":1508774754}"],"signature":null}'
2017-10-23 16:05:54,401 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"op":"BATCH","messages":["{\\"ppSeqNo\\":2,\\"txnRootHash\\":\\"DXRK9Y7vAfHsb2GbvkXUFggkmHfnFv2uf54gqUELAcmS\\",\\"op\\":\\"PREPARE\\",\\"viewNo\\":0,\\"digest\\":\\"068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226\\",\\"stateRootHash\\":\\"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA\\",\\"instId\\":0,\\"ppTime\\":1508774754}","{\\"ppSeqNo\\":2,\\"txnRootHash\\":null,\\"op\\":\\"PREPARE\\",\\"viewNo\\":0,\\"digest\\":\\"068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226\\",\\"stateRootHash\\":null,\\"instId\\":1,\\"ppTime\\":1508774754}"],"signature":null}' to Node3
2017-10-23 16:05:54,401 | DEBUG    | batched.py           (  89) | flushOutBoxes | Node4 batching 2 msgs to Node1 into one transmission
2017-10-23 16:05:54,401 | TRACE    | batched.py           (  90) | flushOutBoxes |     messages: deque([b'{"ppSeqNo":2,"txnRootHash":"DXRK9Y7vAfHsb2GbvkXUFggkmHfnFv2uf54gqUELAcmS","op":"PREPARE","viewNo":0,"digest":"068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226","stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","instId":0,"ppTime":1508774754}', b'{"ppSeqNo":2,"txnRootHash":null,"op":"PREPARE","viewNo":0,"digest":"068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226","stateRootHash":null,"instId":1,"ppTime":1508774754}'])
2017-10-23 16:05:54,401 | TRACE    | batched.py           (  98) | flushOutBoxes | Node4 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"ppSeqNo\\":2,\\"txnRootHash\\":\\"DXRK9Y7vAfHsb2GbvkXUFggkmHfnFv2uf54gqUELAcmS\\",\\"op\\":\\"PREPARE\\",\\"viewNo\\":0,\\"digest\\":\\"068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226\\",\\"stateRootHash\\":\\"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA\\",\\"instId\\":0,\\"ppTime\\":1508774754}","{\\"ppSeqNo\\":2,\\"txnRootHash\\":null,\\"op\\":\\"PREPARE\\",\\"viewNo\\":0,\\"digest\\":\\"068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226\\",\\"stateRootHash\\":null,\\"instId\\":1,\\"ppTime\\":1508774754}"],"signature":null}'
2017-10-23 16:05:54,401 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"op":"BATCH","messages":["{\\"ppSeqNo\\":2,\\"txnRootHash\\":\\"DXRK9Y7vAfHsb2GbvkXUFggkmHfnFv2uf54gqUELAcmS\\",\\"op\\":\\"PREPARE\\",\\"viewNo\\":0,\\"digest\\":\\"068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226\\",\\"stateRootHash\\":\\"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA\\",\\"instId\\":0,\\"ppTime\\":1508774754}","{\\"ppSeqNo\\":2,\\"txnRootHash\\":null,\\"op\\":\\"PREPARE\\",\\"viewNo\\":0,\\"digest\\":\\"068d49f5e5fce9941fdfca611c7d3d96af19f36d40291f8e5065aa067903d226\\",\\"stateRootHash\\":null,\\"instId\\":1,\\"ppTime\\":1508774754}"],"signature":null}' to Node1
2017-10-23 16:05:54,402 | DEBUG    | node.py              (2593) | send | Node4 sending message COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 0} to all recipients: ['Node2', 'Node3', 'Node1']
2017-10-23 16:05:54,403 | DEBUG    | node.py              (2593) | send | Node4 sending message COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 0} to all recipients: ['Node2', 'Node3', 'Node1']
2017-10-23 16:05:54,403 | DEBUG    | replica.py           ( 863) | processCommit | Node4:0 received COMMIT(0, 2) from Node3:0
2017-10-23 16:05:54,403 | DEBUG    | replica.py           ( 898) | tryOrder | Node4:0 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 0} commits where f is 1
2017-10-23 16:05:54,403 | DEBUG    | replica.py           ( 874) | processCommit | Node4:0 processed incoming COMMIT(0, 2)
2017-10-23 16:05:54,403 | DEBUG    | replica.py           ( 863) | processCommit | Node4:0 received COMMIT(0, 2) from Node1:0
2017-10-23 16:05:54,403 | TRACE    | replica.py           ( 894) | tryOrder | Node4:0 returning request to node
2017-10-23 16:05:54,403 | INFO     | replica.py           (1395) | doOrder | Node4:0 ordering COMMIT(0, 2)
2017-10-23 16:05:54,403 | DEBUG    | replica.py           ( 310) | last_ordered_3pc | Node4:0 set last ordered as (0, 2)
2017-10-23 16:05:54,404 | INFO     | replica.py           (2064) | send | Node4:0 sending Ordered
2017-10-23 16:05:54,404 | TRACE    | replica.py           (2065) | send | Node4:0 sending ORDERED{'ppSeqNo': 2, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'instId': 0, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1508774754234674)], 'txnRootHash': 'DXRK9Y7vAfHsb2GbvkXUFggkmHfnFv2uf54gqUELAcmS', 'ledgerId': 2, 'viewNo': 0, 'ppTime': 1508774754}
2017-10-23 16:05:54,404 | DEBUG    | replica.py           (1426) | order_3pc_key | Node4:0 ordered request (0, 2)
2017-10-23 16:05:54,404 | DEBUG    | replica.py           ( 874) | processCommit | Node4:0 processed incoming COMMIT(0, 2)
2017-10-23 16:05:54,404 | TRACE    | message_processor.py (  28) | discard | Node4:0 discarding message COMMIT{'ppSeqNo': 2, 'instId': 0, 'viewNo': 0} because already ordered 3 phase message
2017-10-23 16:05:54,404 | DEBUG    | replica.py           ( 863) | processCommit | Node4:1 received COMMIT(0, 2) from Node3:1
2017-10-23 16:05:54,404 | DEBUG    | replica.py           ( 898) | tryOrder | Node4:1 cannot return request to node: no quorum (3): COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 0} commits where f is 1
2017-10-23 16:05:54,404 | DEBUG    | replica.py           ( 874) | processCommit | Node4:1 processed incoming COMMIT(0, 2)
2017-10-23 16:05:54,404 | DEBUG    | replica.py           ( 863) | processCommit | Node4:1 received COMMIT(0, 2) from Node1:1
2017-10-23 16:05:54,405 | TRACE    | replica.py           ( 894) | tryOrder | Node4:1 returning request to node
2017-10-23 16:05:54,405 | INFO     | replica.py           (1395) | doOrder | Node4:1 ordering COMMIT(0, 2)
2017-10-23 16:05:54,405 | DEBUG    | replica.py           ( 310) | last_ordered_3pc | Node4:1 set last ordered as (0, 2)
2017-10-23 16:05:54,405 | INFO     | replica.py           (2064) | send | Node4:1 sending Ordered
2017-10-23 16:05:54,405 | TRACE    | replica.py           (2065) | send | Node4:1 sending ORDERED{'ppSeqNo': 2, 'stateRootHash': None, 'instId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1508774754234674)], 'txnRootHash': None, 'ledgerId': 2, 'viewNo': 0, 'ppTime': 1508774754}
2017-10-23 16:05:54,405 | DEBUG    | replica.py           (1426) | order_3pc_key | Node4:1 ordered request (0, 2)
2017-10-23 16:05:54,405 | DEBUG    | replica.py           ( 874) | processCommit | Node4:1 processed incoming COMMIT(0, 2)
2017-10-23 16:05:54,405 | TRACE    | message_processor.py (  28) | discard | Node4:1 discarding message COMMIT{'ppSeqNo': 2, 'instId': 1, 'viewNo': 0} because already ordered 3 phase message
2017-10-23 16:05:54,408 | DEBUG    | batched.py           (  89) | flushOutBoxes | Node4 batching 2 msgs to Node2 into one transmission
2017-10-23 16:05:54,425 | TRACE    | batched.py           (  90) | flushOutBoxes |     messages: deque([b'{"ppSeqNo":2,"op":"COMMIT","instId":0,"viewNo":0}', b'{"ppSeqNo":2,"op":"COMMIT","instId":1,"viewNo":0}'])
2017-10-23 16:05:54,425 | TRACE    | batched.py           (  98) | flushOutBoxes | Node4 sending payload to Node2: b'{"op":"BATCH","messages":["{\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\",\\"instId\\":0,\\"viewNo\\":0}","{\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\",\\"instId\\":1,\\"viewNo\\":0}"],"signature":null}'
2017-10-23 16:05:54,425 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"op":"BATCH","messages":["{\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\",\\"instId\\":0,\\"viewNo\\":0}","{\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\",\\"instId\\":1,\\"viewNo\\":0}"],"signature":null}' to Node2
2017-10-23 16:05:54,426 | DEBUG    | batched.py           (  89) | flushOutBoxes | Node4 batching 2 msgs to Node3 into one transmission
2017-10-23 16:05:54,426 | TRACE    | batched.py           (  90) | flushOutBoxes |     messages: deque([b'{"ppSeqNo":2,"op":"COMMIT","instId":0,"viewNo":0}', b'{"ppSeqNo":2,"op":"COMMIT","instId":1,"viewNo":0}'])
2017-10-23 16:05:54,426 | TRACE    | batched.py           (  98) | flushOutBoxes | Node4 sending payload to Node3: b'{"op":"BATCH","messages":["{\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\",\\"instId\\":0,\\"viewNo\\":0}","{\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\",\\"instId\\":1,\\"viewNo\\":0}"],"signature":null}'
2017-10-23 16:05:54,426 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"op":"BATCH","messages":["{\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\",\\"instId\\":0,\\"viewNo\\":0}","{\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\",\\"instId\\":1,\\"viewNo\\":0}"],"signature":null}' to Node3
2017-10-23 16:05:54,426 | DEBUG    | batched.py           (  89) | flushOutBoxes | Node4 batching 2 msgs to Node1 into one transmission
2017-10-23 16:05:54,426 | TRACE    | batched.py           (  90) | flushOutBoxes |     messages: deque([b'{"ppSeqNo":2,"op":"COMMIT","instId":0,"viewNo":0}', b'{"ppSeqNo":2,"op":"COMMIT","instId":1,"viewNo":0}'])
2017-10-23 16:05:54,426 | TRACE    | batched.py           (  98) | flushOutBoxes | Node4 sending payload to Node1: b'{"op":"BATCH","messages":["{\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\",\\"instId\\":0,\\"viewNo\\":0}","{\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\",\\"instId\\":1,\\"viewNo\\":0}"],"signature":null}'
2017-10-23 16:05:54,426 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"op":"BATCH","messages":["{\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\",\\"instId\\":0,\\"viewNo\\":0}","{\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\",\\"instId\\":1,\\"viewNo\\":0}"],"signature":null}' to Node1
2017-10-23 16:05:54,427 | DEBUG    | node.py              (1881) | processOrdered | Node4 executing Ordered batch 0 2 of 1 requests
2017-10-23 16:05:54,566 | DEBUG    | ledger.py            (  68) | commitTxns | Committed 1 txns, 0 are uncommitted
2017-10-23 16:05:54,566 | INFO     | upgrader.py          ( 308) | _scheduleUpgrade | 4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA's upgrader processing upgrade for version 1.1.37
2017-10-23 16:05:54,567 | INFO     | upgrader.py          ( 369) | _callUpgradeAgent | 4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA's upgrader calling agent for upgrade
2017-10-23 16:05:54,568 | DEBUG    | node.py              (2374) | sendReplyToClient | Node4 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1508774754234674) to client
2017-10-23 16:05:54,568 | TRACE    | zstack.py            ( 726) | transmitThroughListener | Node4C transmitting b'{"result":{"action":"start","timeout":10,"txnTime":1508774754,"name":"upgrade-1137","schedule":{"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb":"2017-10-17T11:25:00.000000+00:00","4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA":"2017-10-17T11:35:00.000000+00:00","4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc":"2017-10-17T11:40:00.000000+00:00","Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv":"2017-10-17T11:20:00.000000+00:00","DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya":"2017-10-17T11:30:00.000000+00:00"},"justification":null,"rootHash":"DXRK9Y7vAfHsb2GbvkXUFggkmHfnFv2uf54gqUELAcmS","version":"1.1.37","seqNo":1,"type":"109","signature":"3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ","sha256":"f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3","auditPath":[],"reqId":1508774754234674,"identifier":"V4SGRU86Z58d6TV7PBUe6f","force":true},"op":"REPLY"}' to b'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*' through listener socket
2017-10-23 16:05:54,568 | DEBUG    | node.py              (2295) | executeBatch | Node4 storing 3PC key (0, 2) for ledger 2 range (1, 1)
2017-10-23 16:05:54,568 | TRACE    | node.py              (1892) | processOrdered | Node4 got ordered requests from backup replica 1
2017-10-23 16:05:54,568 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:05:54,585 | WARNING  | base_events.py       (1308) | _run_once | Executing <Task pending coro=<Looper.runForever() running at /usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py:218> wait_for=<Future pending cb=[Task._wakeup()] created at /usr/lib/python3.5/asyncio/base_events.py:252> cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.255 seconds
2017-10-23 16:05:54,586 | DEBUG    | base_events.py       ( 681) | create_connection | connect <socket.socket fd=75, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('0.0.0.0', 0)> to ('127.0.0.1', 30003)
2017-10-23 16:05:54,589 | DEBUG    | base_events.py       (1270) | _run_once | poll 5.796 ms took 0.009 ms: 1 events
2017-10-23 16:05:54,590 | DEBUG    | base_events.py       ( 719) | create_connection | <socket.socket fd=75, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('127.0.0.1', 36136), raddr=('127.0.0.1', 30003)> connected to 127.0.0.1:'30003': (<_SelectorSocketTransport fd=75 read=polling write=<idle, bufsize=0>>, <asyncio.streams.StreamReaderProtocol object at 0x7f6779dd1cf8>)
2017-10-23 16:05:54,591 | TRACE    | has_action_queue.py  (  34) | _schedule | 4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA scheduling action partial(_declareTimeoutExceeded) with id 2 to run in 600 seconds
2017-10-23 16:05:55,924 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 65
2017-10-23 16:05:55,925 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:05:55,925 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:05:55,925 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.000780369086301 is acceptable.
2017-10-23 16:05:55,925 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:05:55,926 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:05:55,926 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:05:55,926 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 66 to run in 10 seconds
2017-10-23 16:06:00,811 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:06:00,811 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:06:00,812 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:06:00,812 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:06:05,933 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 66
2017-10-23 16:06:05,933 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:06:05,934 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:06:05,934 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.000780369086301 is acceptable.
2017-10-23 16:06:05,934 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:06:05,934 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:06:05,934 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:06:05,934 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 67 to run in 10 seconds
2017-10-23 16:06:15,821 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:06:15,822 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:06:15,822 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:06:15,831 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:06:15,955 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 67
2017-10-23 16:06:15,955 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:06:15,955 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:06:15,955 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.000780369086301 is acceptable.
2017-10-23 16:06:15,955 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:06:15,955 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:06:15,955 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:06:15,955 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 68 to run in 10 seconds
2017-10-23 16:06:25,974 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 68
2017-10-23 16:06:25,974 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:06:25,974 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:06:25,974 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.000780369086301 is acceptable.
2017-10-23 16:06:25,975 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:06:25,975 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:06:25,975 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:06:25,975 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 69 to run in 10 seconds
2017-10-23 16:06:30,832 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:06:30,832 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:06:30,832 | DEBUG    | kit_zstack.py        (  65) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:06:30,832 | DEBUG    | kit_zstack.py        (  47) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:06:35,630 | TRACE    | remote.py            ( 117) | hasLostConnection | Remote Node3:HA(host='10.0.0.4', port=9705) has monitor events: [512, 4]
2017-10-23 16:06:35,630 | DEBUG    | remote.py            ( 121) | hasLostConnection | Node3:HA(host='10.0.0.4', port=9705) found disconnected event on monitor
2017-10-23 16:06:35,631 | DEBUG    | keep_in_touch.py     (  68) | conns | Node4's connections changed from {'Node2', 'Node3', 'Node1'} to {'Node2', 'Node1'}
2017-10-23 16:06:35,631 | INFO     | keep_in_touch.py     (  92) | _connsChanged | Node4 disconnected from Node3
2017-10-23 16:06:35,631 | DEBUG    | motor.py             (  34) | set_status | Node4 changing status from started to started_hungry
2017-10-23 16:06:35,631 | DEBUG    | node.py              ( 918) | checkInstances | Node4 choosing to start election on the basis of count 3 and nodes {'Node2', 'Node1'}
2017-10-23 16:06:35,983 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action checkPerformance with id 69
2017-10-23 16:06:35,983 | TRACE    | node.py              (2005) | checkPerformance | Node4 checking its performance
2017-10-23 16:06:35,983 | DEBUG    | notifier_plugin_manager.py (  65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:06:35,983 | TRACE    | monitor.py           ( 299) | isMasterThroughputTooLow | Node4 master throughput ratio 1.000780369086301 is acceptable.
2017-10-23 16:06:35,983 | TRACE    | monitor.py           ( 315) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:06:35,984 | TRACE    | monitor.py           ( 345) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:06:35,984 | DEBUG    | node.py              (2022) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:06:35,984 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action checkPerformance with id 70 to run in 10 seconds
2017-10-23 16:06:36,463 | TRACE    | remote.py            ( 117) | hasLostConnection | Remote Node2:HA(host='10.0.0.3', port=9703) has monitor events: [512, 4]
2017-10-23 16:06:36,464 | DEBUG    | remote.py            ( 121) | hasLostConnection | Node2:HA(host='10.0.0.3', port=9703) found disconnected event on monitor
2017-10-23 16:06:36,464 | DEBUG    | keep_in_touch.py     (  68) | conns | Node4's connections changed from {'Node2', 'Node1'} to {'Node1'}
2017-10-23 16:06:36,465 | INFO     | keep_in_touch.py     (  92) | _connsChanged | Node4 disconnected from Node2
2017-10-23 16:06:36,465 | DEBUG    | motor.py             (  34) | set_status | Node4 changing status from started_hungry to starting
2017-10-23 16:06:39,436 | TRACE    | remote.py            ( 117) | hasLostConnection | Remote Node1:HA(host='10.0.0.2', port=9701) has monitor events: [512, 4]
2017-10-23 16:06:39,436 | DEBUG    | remote.py            ( 121) | hasLostConnection | Node1:HA(host='10.0.0.2', port=9701) found disconnected event on monitor
2017-10-23 16:06:39,436 | DEBUG    | keep_in_touch.py     (  68) | conns | Node4's connections changed from {'Node1'} to set()
2017-10-23 16:06:39,436 | INFO     | keep_in_touch.py     (  92) | _connsChanged | Node4 disconnected from Node1
2017-10-23 16:06:39,436 | DEBUG    | node.py              ( 802) | onConnsChanged | Node4 lost connection to primary of master
2017-10-23 16:06:39,436 | DEBUG    | node.py              (2137) | lost_master_primary | Node4 scheduling a view change in 2 sec
2017-10-23 16:06:39,436 | TRACE    | has_action_queue.py  (  34) | _schedule | Node4 scheduling action propose_view_change with id 71 to run in 2 seconds
2017-10-23 16:06:41,443 | TRACE    | has_action_queue.py  (  64) | _serviceActions | Node4 running action propose_view_change with id 71
2017-10-23 16:06:41,443 | DEBUG    | throttler.py         (  30) | acquire | now: 956301.805015142, len(actionsLog): 0
2017-10-23 16:06:41,443 | DEBUG    | throttler.py         (  32) | acquire | after trim, len(actionsLog): 0
2017-10-23 16:06:41,443 | DEBUG    | throttler.py         (  37) | acquire | len(actionsLog) was 0, after append, len(actionsLog): 1
2017-10-23 16:06:41,443 | INFO     | node.py              (2071) | sendInstanceChange | Node4 sending an instance change with view_no 1 since Primary of master protocol instance disconnected
2017-10-23 16:06:41,443 | INFO     | node.py              (2073) | sendInstanceChange | Node4 metrics for monitor: Node4 Monitor metrics:: None
            Delta: 0.4
            Lambda: 60
            Omega: 5
            instances started: [955705.804147449, 955705.804474224]
            ordered request counts: {0: 2, 1: 2}
            ordered request durations: {0: 0.41634283401072025, 1: 0.4166677350876853}
            master request latencies: {}
            client avg request latencies: [{'V4SGRU86Z58d6TV7PBUe6f': (2, 0.20817141700536013)}, {'V4SGRU86Z58d6TV7PBUe6f': (2, 0.20833386754384264)}]
            throughput: {0: 4.80373345383075, 1: 4.799987691821427}
            master throughput: 4.80373345383075
            total requests: 2
            avg backup throughput: 4.799987691821427
            master throughput ratio: 1.000780369086301
2017-10-23 16:06:41,443 | DEBUG    | node.py              (2593) | send | Node4 sending message INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} to all recipients: ['Node2', 'Node3', 'Node1']
2017-10-23 16:06:41,443 | DEBUG    | node.py              (1986) | do_view_change_if_possible | Node4 has no quorum for view 1
2017-10-23 16:06:41,444 | DEBUG    | node.py              (2124) | propose_view_change | Node4 sent view change since was disconnected from primary for too long
2017-10-23 16:06:41,444 | DEBUG    | node.py              (1986) | do_view_change_if_possible | Node4 has no quorum for view 1
2017-10-23 16:06:41,444 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"reason":26,"op":"INSTANCE_CHANGE","viewNo":1}' to Node2
2017-10-23 16:06:41,444 | WARNING  | zstack.py            ( 704) | transmit | Remote Node2 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings
2017-10-23 16:06:41,444 | TRACE    | batched.py           (  85) | flushOutBoxes | Node4 sending msg b'{"reason":26,"op":"INSTANCE_CHANGE","viewNo":1}' to Node2
2017-10-23 16:06:41,444 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"reason":26,"op":"INSTANCE_CHANGE","viewNo":1}' to Node3
2017-10-23 16:06:41,444 | WARNING  | zstack.py            ( 704) | transmit | Remote Node3 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings
2017-10-23 16:06:41,444 | TRACE    | batched.py           (  85) | flushOutBoxes | Node4 sending msg b'{"reason":26,"op":"INSTANCE_CHANGE","viewNo":1}' to Node3
2017-10-23 16:06:41,444 | DEBUG    | zstack.py            ( 699) | transmit | Node4 transmitting message b'{"reason":26,"op":"INSTANCE_CHANGE","viewNo":1}' to Node1
2017-10-23 16:06:41,444 | WARNING  | zstack.py            ( 704) | transmit | Remote Node1 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings
2017-10-23 16:06:41,445 | TRACE    | batched.py           (  85) | flushOutBoxes | Node4 sending msg b'{"reason":26,"op":"INSTANCE_CHANGE","viewNo":1}' to Node1
2017-10-23 16:06:51,356 | DEBUG    | node_runner.py       (  18) | run_node | You can find logs in /home/sovrin/.sovrin/Node4.log
2017-10-23 16:06:51,356 | DEBUG    | node_runner.py       (  21) | run_node | Sovrin related env vars: []
2017-10-23 16:06:52,745 | DEBUG    | __init__.py          (  60) | register | Registered VCS backend: git
2017-10-23 16:06:52,762 | DEBUG    | __init__.py          (  60) | register | Registered VCS backend: hg
2017-10-23 16:06:52,807 | DEBUG    | __init__.py          (  60) | register | Registered VCS backend: svn
2017-10-23 16:06:52,808 | DEBUG    | __init__.py          (  60) | register | Registered VCS backend: bzr
2017-10-23 16:06:53,297 | DEBUG    | selector_events.py   (  53) | __init__ | Using selector: EpollSelector
2017-10-23 16:06:53,297 | DEBUG    | looper.py            ( 125) | __init__ | Setting handler for SIGINT
2017-10-23 16:06:53,340 | DEBUG    | ledger.py            ( 200) | start | Starting ledger...
2017-10-23 16:06:53,362 | DEBUG    | ledger.py            (  72) | recoverTree | Recovering tree from hash store of size 7
2017-10-23 16:06:53,362 | DEBUG    | ledger.py            (  82) | recoverTree | Recovered tree in 0.0004937490448355675 seconds
2017-10-23 16:06:53,421 | DEBUG    | idr_cache.py         (  25) | __init__ | Initializing identity cache Node4
2017-10-23 16:06:53,442 | INFO     | node.py              (2420) | initStateFromLedger | Node4 found state to be empty, recreating from ledger
2017-10-23 16:06:53,528 | DEBUG    | ledger.py            ( 200) | start | Starting ledger...
2017-10-23 16:06:53,550 | DEBUG    | ledger.py            (  72) | recoverTree | Recovering tree from hash store of size 4
2017-10-23 16:06:53,550 | DEBUG    | ledger.py            (  82) | recoverTree | Recovered tree in 0.000504980911500752 seconds
2017-10-23 16:06:53,550 | INFO     | node.py              (2420) | initStateFromLedger | Node4 found state to be empty, recreating from ledger
2017-10-23 16:06:53,590 | DEBUG    | plugin_loader.py     (  96) | _load | skipping plugin plugin_firebase_stats_consumer[class: <class 'plenum.server.plugin.stats_consumer.stats_publisher.StatsPublisher'>] because it does not have a 'pluginType' attribute
2017-10-23 16:06:53,590 | DEBUG    | plugin_loader.py     (  96) | _load | skipping plugin plugin_firebase_stats_consumer[class: typing.Dict<~KT, ~VT>] because it does not have a 'pluginType' attribute
2017-10-23 16:06:53,590 | DEBUG    | plugin_loader.py     (  96) | _load | skipping plugin plugin_firebase_stats_consumer[class: <enum 'Topic'>] because it does not have a 'pluginType' attribute
2017-10-23 16:06:53,590 | DEBUG    | plugin_loader.py     (  96) | _load | skipping plugin plugin_firebase_stats_consumer[class: <class 'plenum.server.stats_consumer.StatsConsumer'>] because it does not have a 'pluginType' attribute
2017-10-23 16:06:53,590 | INFO     | plugin_loader.py     ( 117) | _load | plugin FirebaseStatsConsumer successfully loaded from module plugin_firebase_stats_consumer
2017-10-23 16:06:53,590 | DEBUG    | plugin_loader.py     (  96) | _load | skipping plugin plugin_firebase_stats_consumer[class: <class 'plenum.server.plugin_loader.HasDynamicallyImportedModules'>] because it does not have a 'pluginType' attribute
2017-10-23 16:06:53,590 | DEBUG    | has_action_queue.py  (  79) | startRepeating | checkPerformance will be repeating every 60 seconds
2017-10-23 16:06:53,591 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 1 to run in 60 seconds
2017-10-23 16:06:53,591 | DEBUG    | replica.py           ( 313) | h | Node4:0 set watermarks as 0 300
2017-10-23 16:06:53,591 | DISPLAY  | replicas.py          (  36) | grow | Node4 added replica Node4:0 to instance 0 (master)
2017-10-23 16:06:53,591 | DEBUG    | replica.py           ( 313) | h | Node4:1 set watermarks as 0 300
2017-10-23 16:06:53,591 | DISPLAY  | replicas.py          (  36) | grow | Node4 added replica Node4:1 to instance 1 (backup)
2017-10-23 16:06:53,591 | DEBUG    | has_action_queue.py  (  79) | startRepeating | checkPerformance will be repeating every 10 seconds
2017-10-23 16:06:53,592 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 1 to run in 10 seconds
2017-10-23 16:06:53,592 | DEBUG    | has_action_queue.py  (  79) | startRepeating | checkNodeRequestSpike will be repeating every 60 seconds
2017-10-23 16:06:53,592 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 2 to run in 60 seconds
2017-10-23 16:06:53,592 | DEBUG    | plugin_helper.py     (  24) | loadPlugins | Plugin loading started to load plugins from basedir: /home/sovrin/.sovrin
2017-10-23 16:06:53,592 | DEBUG    | plugin_helper.py     (  68) | loadPlugins | Total plugins loaded from basedir /home/sovrin/.sovrin are : 0
2017-10-23 16:06:53,592 | DEBUG    | node.py              ( 340) | __init__ | total plugins loaded in node: 0
2017-10-23 16:06:53,658 | DEBUG    | ledger.py            ( 200) | start | Starting ledger...
2017-10-23 16:06:53,680 | DEBUG    | ledger.py            (  72) | recoverTree | Recovering tree from hash store of size 1
2017-10-23 16:06:53,681 | DEBUG    | ledger.py            (  82) | recoverTree | Recovered tree in 0.0003930249949917197 seconds
2017-10-23 16:06:53,710 | WARNING  | upgrader.py          ( 126) | check_upgrade_succeeded | Upgrade for node Node4 was not scheduled. Last event is scheduled:2017-10-17 11:35:00+00:00:1.1.37:None
2017-10-23 16:06:53,710 | INFO     | node.py              (2420) | initStateFromLedger | Node4 found state to be empty, recreating from ledger
2017-10-23 16:06:53,710 | DEBUG    | motor.py             (  34) | set_status | Node4 changing status from stopped to starting
2017-10-23 16:06:53,710 | DEBUG    | ledger.py            ( 198) | start | Ledger already started.
2017-10-23 16:06:53,710 | DEBUG    | ledger.py            ( 198) | start | Ledger already started.
2017-10-23 16:06:53,710 | DEBUG    | ledger.py            ( 198) | start | Ledger already started.
2017-10-23 16:06:53,710 | DEBUG    | zstack.py            ( 319) | start | Node4 starting with restricted as True and reSetupAuth as True
2017-10-23 16:06:53,711 | DEBUG    | authenticator.py     (  31) | start | Starting ZAP at inproc://zeromq.zap.1
2017-10-23 16:06:53,711 | DEBUG    | base.py              (  72) | allow | Allowing 0.0.0.0
2017-10-23 16:06:53,711 | DEBUG    | base.py              ( 112) | configure_curve | Configure curve: *[/home/sovrin/.sovrin/Node4/public_keys]
2017-10-23 16:06:53,711 | DEBUG    | zstack.py            ( 347) | open | Node4 will bind its listener at 9707
2017-10-23 16:06:53,712 | INFO     | stacks.py            (  84) | start | CONNECTION: Node4 listening for other nodes at 0.0.0.0:9707
2017-10-23 16:06:53,712 | DEBUG    | zstack.py            ( 319) | start | Node4C starting with restricted as False and reSetupAuth as True
2017-10-23 16:06:53,712 | DEBUG    | authenticator.py     (  31) | start | Starting ZAP at inproc://zeromq.zap.2
2017-10-23 16:06:53,712 | DEBUG    | base.py              (  72) | allow | Allowing 0.0.0.0
2017-10-23 16:06:53,712 | DEBUG    | base.py              ( 112) | configure_curve | Configure curve: *[*]
2017-10-23 16:06:53,712 | DEBUG    | zstack.py            ( 347) | open | Node4C will bind its listener at 9708
2017-10-23 16:06:53,713 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action propose_view_change with id 3 to run in 60 seconds
2017-10-23 16:06:53,713 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 4 to run in 3 seconds
2017-10-23 16:06:53,713 | DEBUG    | has_action_queue.py  (  79) | startRepeating | dump_json_file will be repeating every 60 seconds
2017-10-23 16:06:53,713 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 5 to run in 60 seconds
2017-10-23 16:06:53,713 | INFO     | node.py              ( 621) | start | Node4 first time running...
2017-10-23 16:06:53,714 | DEBUG    | kit_zstack.py        (  97) | connectToMissing | CONNECTION: Node4 found the following missing connections: Node2, Node1, Node3
2017-10-23 16:06:53,714 | TRACE    | remote.py            (  86) | connect | connecting socket 78 45566880 to remote Node2:HA(host='10.0.0.3', port=9703)
2017-10-23 16:06:53,716 | INFO     | zstack.py            ( 590) | connect | CONNECTION: Node4 looking for Node2 at 10.0.0.3:9703
2017-10-23 16:06:53,717 | DEBUG    | zstack.py            ( 645) | sendPingPong | Node4 pinged Node2
2017-10-23 16:06:53,717 | TRACE    | remote.py            (  86) | connect | connecting socket 82 45670656 to remote Node1:HA(host='10.0.0.2', port=9701)
2017-10-23 16:06:53,717 | INFO     | zstack.py            ( 590) | connect | CONNECTION: Node4 looking for Node1 at 10.0.0.2:9701
2017-10-23 16:06:53,717 | DEBUG    | zstack.py            ( 645) | sendPingPong | Node4 pinged Node1
2017-10-23 16:06:53,718 | TRACE    | remote.py            (  86) | connect | connecting socket 83 45707840 to remote Node3:HA(host='10.0.0.4', port=9705)
2017-10-23 16:06:53,718 | INFO     | zstack.py            ( 590) | connect | CONNECTION: Node4 looking for Node3 at 10.0.0.4:9705
2017-10-23 16:06:53,718 | DEBUG    | zstack.py            ( 645) | sendPingPong | Node4 pinged Node3
2017-10-23 16:06:53,718 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:06:53,729 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'pi' to Node2
2017-10-23 16:06:53,729 | TRACE    | batched.py           (  96) | flushOutBoxes | Node4 sending msg b'pi' to Node2
2017-10-23 16:06:53,729 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'pi' to Node1
2017-10-23 16:06:53,729 | TRACE    | batched.py           (  96) | flushOutBoxes | Node4 sending msg b'pi' to Node1
2017-10-23 16:06:53,729 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'pi' to Node3
2017-10-23 16:06:53,729 | TRACE    | batched.py           (  96) | flushOutBoxes | Node4 sending msg b'pi' to Node3
2017-10-23 16:06:53,745 | TRACE    | zstack.py            ( 479) | _receiveFromListener | Node4 got 4 messages through listener
2017-10-23 16:06:53,745 | DEBUG    | zstack.py            ( 663) | handlePingPong | Node4 got ping from Node3
2017-10-23 16:06:53,745 | DEBUG    | zstack.py            ( 645) | sendPingPong | Node4 ponged Node3
2017-10-23 16:06:53,746 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}']}
2017-10-23 16:06:53,746 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}']}, 'Node3')
2017-10-23 16:06:53,746 | DEBUG    | node.py              (1272) | unpackNodeMsg | Node4 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}']}
2017-10-23 16:06:53,746 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}
2017-10-23 16:06:53,746 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}, 'Node3')
2017-10-23 16:06:53,746 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}
2017-10-23 16:06:53,747 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}
2017-10-23 16:06:53,749 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}, 'Node3')
2017-10-23 16:06:53,749 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}
2017-10-23 16:06:53,749 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}']}
2017-10-23 16:06:53,749 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}']}, 'Node3')
2017-10-23 16:06:53,749 | DEBUG    | node.py              (1272) | unpackNodeMsg | Node4 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}']}
2017-10-23 16:06:53,749 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}
2017-10-23 16:06:53,749 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}, 'Node3')
2017-10-23 16:06:53,749 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}
2017-10-23 16:06:53,750 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}
2017-10-23 16:06:53,750 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}, 'Node3')
2017-10-23 16:06:53,750 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}
2017-10-23 16:06:53,750 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}
2017-10-23 16:06:53,750 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1', 'op': 'VIEW_CHANGE_DONE'}, 'Node3')
2017-10-23 16:06:53,750 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}
2017-10-23 16:06:53,751 | DEBUG    | node.py              (2608) | send | Node4 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None}} to 1 recipients: ['Node3']
2017-10-23 16:06:53,751 | DEBUG    | node.py              (2608) | send | Node4 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None}} to 1 recipients: ['Node3']
2017-10-23 16:06:53,751 | DEBUG    | node.py              (2608) | send | Node4 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None}} to 1 recipients: ['Node3']
2017-10-23 16:06:53,752 | DEBUG    | node.py              (2608) | send | Node4 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None}} to 1 recipients: ['Node3']
2017-10-23 16:06:53,755 | DEBUG    | node.py              (1205) | sendToElector | Node4 sending message to elector: (VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}, 'Node3')
2017-10-23 16:06:53,756 | DEBUG    | batched.py           ( 100) | flushOutBoxes | Node4 batching 5 msgs to Node3 into one transmission
2017-10-23 16:06:53,756 | TRACE    | batched.py           ( 101) | flushOutBoxes |     messages: deque([b'po', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null}}'])
2017-10-23 16:06:53,756 | TRACE    | batched.py           ( 110) | flushOutBoxes | Node4 sending payload to Node3: b'{"op":"BATCH","signature":null,"messages":["po","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":1,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":1,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":7,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":7,\\"viewNo\\":null}}"]}'
2017-10-23 16:06:53,756 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"op":"BATCH","signature":null,"messages":["po","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":1,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":1,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":7,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":7,\\"viewNo\\":null}}"]}' to Node3
2017-10-23 16:06:53,756 | DEBUG    | zstack.py            ( 728) | transmit | Remote Node3 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings
2017-10-23 16:06:53,778 | TRACE    | zstack.py            ( 479) | _receiveFromListener | Node4 got 7 messages through listener
2017-10-23 16:06:53,779 | DEBUG    | zstack.py            ( 663) | handlePingPong | Node4 got ping from Node2
2017-10-23 16:06:53,779 | DEBUG    | zstack.py            ( 645) | sendPingPong | Node4 ponged Node2
2017-10-23 16:06:53,779 | DEBUG    | zstack.py            ( 669) | handlePingPong | Node4 got pong from Node3
2017-10-23 16:06:53,779 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":2},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}']}
2017-10-23 16:06:53,779 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"params":{"ledgerId":2},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":2},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}']}, 'Node2')
2017-10-23 16:06:53,779 | DEBUG    | node.py              (1272) | unpackNodeMsg | Node4 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":2},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}']}
2017-10-23 16:06:53,780 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}
2017-10-23 16:06:53,780 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}, 'Node2')
2017-10-23 16:06:53,780 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}
2017-10-23 16:06:53,780 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}
2017-10-23 16:06:53,780 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}, 'Node2')
2017-10-23 16:06:53,780 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}
2017-10-23 16:06:53,780 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: BATCH{'signature': None, 'messages': ['{"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1","viewNo":0}],"viewNo":0,"op":"CURRENT_STATE"}', '{"viewNo":null,"op":"LEDGER_STATUS","merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ledgerId":0,"txnSeqNo":4,"ppSeqNo":null}', '{"viewNo":null,"op":"LEDGER_STATUS","merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ledgerId":1,"txnSeqNo":7,"ppSeqNo":null}', '{"viewNo":null,"op":"LEDGER_STATUS","merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ledgerId":2,"txnSeqNo":1,"ppSeqNo":null}']}
2017-10-23 16:06:53,780 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1","viewNo":0}],"viewNo":0,"op":"CURRENT_STATE"}', '{"viewNo":null,"op":"LEDGER_STATUS","merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ledgerId":0,"txnSeqNo":4,"ppSeqNo":null}', '{"viewNo":null,"op":"LEDGER_STATUS","merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ledgerId":1,"txnSeqNo":7,"ppSeqNo":null}', '{"viewNo":null,"op":"LEDGER_STATUS","merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ledgerId":2,"txnSeqNo":1,"ppSeqNo":null}']}, 'Node3')
2017-10-23 16:06:53,780 | DEBUG    | node.py              (1272) | unpackNodeMsg | Node4 processing a batch BATCH{'signature': None, 'messages': ['{"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1","viewNo":0}],"viewNo":0,"op":"CURRENT_STATE"}', '{"viewNo":null,"op":"LEDGER_STATUS","merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ledgerId":0,"txnSeqNo":4,"ppSeqNo":null}', '{"viewNo":null,"op":"LEDGER_STATUS","merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ledgerId":1,"txnSeqNo":7,"ppSeqNo":null}', '{"viewNo":null,"op":"LEDGER_STATUS","merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ledgerId":2,"txnSeqNo":1,"ppSeqNo":null}']}
2017-10-23 16:06:53,781 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: CURRENT_STATE{'viewNo': 0, 'primary': [{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}]}
2017-10-23 16:06:53,781 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'viewNo': 0, 'op': 'CURRENT_STATE', 'primary': [{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}]}, 'Node3')
2017-10-23 16:06:53,781 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox CURRENT_STATE{'viewNo': 0, 'primary': [{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}]}
2017-10-23 16:06:53,781 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: LEDGER_STATUS{'viewNo': None, 'ledgerId': 0, 'txnSeqNo': 4, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': None}
2017-10-23 16:06:53,781 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'op': 'LEDGER_STATUS', 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'Node3')
2017-10-23 16:06:53,781 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox LEDGER_STATUS{'viewNo': None, 'ledgerId': 0, 'txnSeqNo': 4, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': None}
2017-10-23 16:06:53,782 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: LEDGER_STATUS{'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None}
2017-10-23 16:06:53,782 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'txnSeqNo': 7, 'op': 'LEDGER_STATUS', 'viewNo': None, 'ledgerId': 1, 'ppSeqNo': None}, 'Node3')
2017-10-23 16:06:53,782 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox LEDGER_STATUS{'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None}
2017-10-23 16:06:53,782 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: LEDGER_STATUS{'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None}
2017-10-23 16:06:53,782 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'txnSeqNo': 1, 'op': 'LEDGER_STATUS', 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'Node3')
2017-10-23 16:06:53,782 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox LEDGER_STATUS{'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None}
2017-10-23 16:06:53,782 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":1},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":1},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}']}
2017-10-23 16:06:53,782 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"params":{"ledgerId":1},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":1},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}']}, 'Node2')
2017-10-23 16:06:53,783 | DEBUG    | node.py              (1272) | unpackNodeMsg | Node4 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":1},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":1},"op":"MESSAGE_REQUEST","msg_type":"LEDGER_STATUS"}']}
2017-10-23 16:06:53,783 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}
2017-10-23 16:06:53,783 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}, 'Node2')
2017-10-23 16:06:53,783 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}
2017-10-23 16:06:53,783 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}
2017-10-23 16:06:53,783 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}, 'Node2')
2017-10-23 16:06:53,783 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}
2017-10-23 16:06:53,784 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}
2017-10-23 16:06:53,784 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'VIEW_CHANGE_DONE', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1', 'viewNo': 0}, 'Node2')
2017-10-23 16:06:53,784 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}
2017-10-23 16:06:53,784 | DEBUG    | zstack.py            ( 669) | handlePingPong | Node4 got pong from Node2
2017-10-23 16:06:53,784 | DEBUG    | node.py              (2608) | send | Node4 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None}} to 1 recipients: ['Node2']
2017-10-23 16:06:53,785 | DEBUG    | node.py              (2608) | send | Node4 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None}} to 1 recipients: ['Node2']
2017-10-23 16:06:53,786 | DEBUG    | node.py              ( 907) | process_current_state_message | Node4 processing current state CURRENT_STATE{'viewNo': 0, 'primary': [{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}]} from Node3
2017-10-23 16:06:53,786 | DEBUG    | node.py              (1205) | sendToElector | Node4 sending message to elector: (VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}, 'Node3')
2017-10-23 16:06:53,787 | DEBUG    | ledger_manager.py    ( 246) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'viewNo': None, 'ledgerId': 0, 'txnSeqNo': 4, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': None} from Node3
2017-10-23 16:06:53,789 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 0 of size 4 with 4
2017-10-23 16:06:53,789 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 0 of size 4 with 4
2017-10-23 16:06:53,790 | DEBUG    | ledger_manager.py    ( 246) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None} from Node3
2017-10-23 16:06:53,790 | DEBUG    | ledger_manager.py    (1016) | stashLedgerStatus | Node4 stashing ledger status LEDGER_STATUS{'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None} from Node3
2017-10-23 16:06:53,790 | DEBUG    | ledger_manager.py    ( 246) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None} from Node3
2017-10-23 16:06:53,790 | DEBUG    | ledger_manager.py    (1016) | stashLedgerStatus | Node4 stashing ledger status LEDGER_STATUS{'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None} from Node3
2017-10-23 16:06:53,790 | DEBUG    | node.py              (2608) | send | Node4 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None}} to 1 recipients: ['Node2']
2017-10-23 16:06:53,791 | DEBUG    | node.py              (2608) | send | Node4 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None}} to 1 recipients: ['Node2']
2017-10-23 16:06:53,791 | DEBUG    | node.py              (1205) | sendToElector | Node4 sending message to elector: (VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}, 'Node2')
2017-10-23 16:06:53,791 | DEBUG    | batched.py           ( 100) | flushOutBoxes | Node4 batching 5 msgs to Node2 into one transmission
2017-10-23 16:06:53,791 | TRACE    | batched.py           ( 101) | flushOutBoxes |     messages: deque([b'po', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null}}'])
2017-10-23 16:06:53,791 | TRACE    | batched.py           ( 110) | flushOutBoxes | Node4 sending payload to Node2: b'{"op":"BATCH","signature":null,"messages":["po","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":1,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":1,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":7,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":7,\\"viewNo\\":null}}"]}'
2017-10-23 16:06:53,792 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"op":"BATCH","signature":null,"messages":["po","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":1,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":1,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":7,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":7,\\"viewNo\\":null}}"]}' to Node2
2017-10-23 16:06:53,792 | DEBUG    | keep_in_touch.py     (  68) | conns | Node4's connections changed from set() to {'Node2', 'Node3'}
2017-10-23 16:06:53,792 | INFO     | keep_in_touch.py     (  98) | _connsChanged | CONNECTION: Node4 now connected to Node2
2017-10-23 16:06:53,792 | INFO     | keep_in_touch.py     (  98) | _connsChanged | CONNECTION: Node4 now connected to Node3
2017-10-23 16:06:53,792 | DEBUG    | motor.py             (  34) | set_status | Node4 changing status from starting to started_hungry
2017-10-23 16:06:53,792 | DEBUG    | node.py              ( 941) | checkInstances | Node4 choosing to start election on the basis of count 3 and nodes {'Node2', 'Node3'}
2017-10-23 16:06:53,793 | DEBUG    | primary_selector.py  (  77) | get_msgs_for_lagged_nodes | Node4 has no ViewChangeDone message to send for view 0
2017-10-23 16:06:53,793 | DEBUG    | node.py              ( 902) | send_current_state_to_lagging_node | Node4 sending current state CURRENT_STATE{'viewNo': 0, 'primary': []} to lagged node Node2
2017-10-23 16:06:53,793 | DEBUG    | node.py              (2608) | send | Node4 sending message CURRENT_STATE{'viewNo': 0, 'primary': []} to 1 recipients: ['Node2']
2017-10-23 16:06:53,793 | DEBUG    | primary_selector.py  (  77) | get_msgs_for_lagged_nodes | Node4 has no ViewChangeDone message to send for view 0
2017-10-23 16:06:53,793 | DEBUG    | node.py              ( 902) | send_current_state_to_lagging_node | Node4 sending current state CURRENT_STATE{'viewNo': 0, 'primary': []} to lagged node Node3
2017-10-23 16:06:53,793 | DEBUG    | node.py              (2608) | send | Node4 sending message CURRENT_STATE{'viewNo': 0, 'primary': []} to 1 recipients: ['Node3']
2017-10-23 16:06:53,793 | DEBUG    | node.py              (2608) | send | Node4 sending message LEDGER_STATUS{'viewNo': None, 'ledgerId': 0, 'txnSeqNo': 4, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': None} to 1 recipients: ['Node2']
2017-10-23 16:06:53,793 | DEBUG    | node.py              (2608) | send | Node4 sending message LEDGER_STATUS{'viewNo': None, 'ledgerId': 0, 'txnSeqNo': 4, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': None} to 1 recipients: ['Node3']
2017-10-23 16:06:53,796 | DEBUG    | primary_selector.py  ( 106) | _processViewChangeDoneMessage | Node4's primary selector started processing of ViewChangeDone msg from Node3 : VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}
2017-10-23 16:06:53,796 | DEBUG    | primary_selector.py  ( 190) | _hasViewChangeQuorum | Node4 needs 1 ViewChangeDone messages
2017-10-23 16:06:53,796 | DEBUG    | primary_selector.py  ( 266) | _startSelection | Node4 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages
2017-10-23 16:06:53,796 | DEBUG    | primary_selector.py  ( 106) | _processViewChangeDoneMessage | Node4's primary selector started processing of ViewChangeDone msg from Node3 : VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}
2017-10-23 16:06:53,796 | DEBUG    | primary_selector.py  ( 190) | _hasViewChangeQuorum | Node4 needs 1 ViewChangeDone messages
2017-10-23 16:06:53,796 | DEBUG    | primary_selector.py  ( 266) | _startSelection | Node4 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages
2017-10-23 16:06:53,796 | DEBUG    | primary_selector.py  ( 106) | _processViewChangeDoneMessage | Node4's primary selector started processing of ViewChangeDone msg from Node2 : VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}
2017-10-23 16:06:53,796 | DEBUG    | primary_selector.py  ( 196) | _hasViewChangeQuorum | Node4 got view change quorum (2 >= 2)
2017-10-23 16:06:53,796 | DEBUG    | primary_selector.py  ( 209) | has_view_change_from_primary | Node4 has not received ViewChangeDone from the next primary Node1 (viewNo: 0, totalNodes: 4)
2017-10-23 16:06:53,797 | DEBUG    | primary_selector.py  ( 266) | _startSelection | Node4 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages
2017-10-23 16:06:53,797 | DEBUG    | batched.py           ( 100) | flushOutBoxes | Node4 batching 2 msgs to Node2 into one transmission
2017-10-23 16:06:53,797 | TRACE    | batched.py           ( 101) | flushOutBoxes |     messages: deque([b'{"viewNo":0,"primary":[],"op":"CURRENT_STATE"}', b'{"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","txnSeqNo":4,"viewNo":null,"op":"LEDGER_STATUS","ledgerId":0,"ppSeqNo":null}'])
2017-10-23 16:06:53,797 | TRACE    | batched.py           ( 110) | flushOutBoxes | Node4 sending payload to Node2: b'{"op":"BATCH","signature":null,"messages":["{\\"viewNo\\":0,\\"primary\\":[],\\"op\\":\\"CURRENT_STATE\\"}","{\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"txnSeqNo\\":4,\\"viewNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":0,\\"ppSeqNo\\":null}"]}'
2017-10-23 16:06:53,797 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"op":"BATCH","signature":null,"messages":["{\\"viewNo\\":0,\\"primary\\":[],\\"op\\":\\"CURRENT_STATE\\"}","{\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"txnSeqNo\\":4,\\"viewNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":0,\\"ppSeqNo\\":null}"]}' to Node2
2017-10-23 16:06:53,797 | DEBUG    | batched.py           ( 100) | flushOutBoxes | Node4 batching 2 msgs to Node3 into one transmission
2017-10-23 16:06:53,797 | TRACE    | batched.py           ( 101) | flushOutBoxes |     messages: deque([b'{"viewNo":0,"primary":[],"op":"CURRENT_STATE"}', b'{"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","txnSeqNo":4,"viewNo":null,"op":"LEDGER_STATUS","ledgerId":0,"ppSeqNo":null}'])
2017-10-23 16:06:53,797 | TRACE    | batched.py           ( 110) | flushOutBoxes | Node4 sending payload to Node3: b'{"op":"BATCH","signature":null,"messages":["{\\"viewNo\\":0,\\"primary\\":[],\\"op\\":\\"CURRENT_STATE\\"}","{\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"txnSeqNo\\":4,\\"viewNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":0,\\"ppSeqNo\\":null}"]}'
2017-10-23 16:06:53,797 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"op":"BATCH","signature":null,"messages":["{\\"viewNo\\":0,\\"primary\\":[],\\"op\\":\\"CURRENT_STATE\\"}","{\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"txnSeqNo\\":4,\\"viewNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":0,\\"ppSeqNo\\":null}"]}' to Node3
2017-10-23 16:06:53,809 | TRACE    | zstack.py            ( 479) | _receiveFromListener | Node4 got 4 messages through listener
2017-10-23 16:06:53,809 | DEBUG    | zstack.py            ( 663) | handlePingPong | Node4 got ping from Node1
2017-10-23 16:06:53,810 | DEBUG    | zstack.py            ( 645) | sendPingPong | Node4 ponged Node1
2017-10-23 16:06:53,810 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node1: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}']}
2017-10-23 16:06:53,810 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}']}, 'Node1')
2017-10-23 16:06:53,810 | DEBUG    | node.py              (1272) | unpackNodeMsg | Node4 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}']}
2017-10-23 16:06:53,810 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node1: MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}
2017-10-23 16:06:53,810 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}, 'Node1')
2017-10-23 16:06:53,810 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}
2017-10-23 16:06:53,810 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node1: MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}
2017-10-23 16:06:53,811 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}, 'Node1')
2017-10-23 16:06:53,811 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'}
2017-10-23 16:06:53,811 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node1: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"viewNo":0,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1","op":"VIEW_CHANGE_DONE"}', '{"viewNo":0,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1","op":"VIEW_CHANGE_DONE"}']}
2017-10-23 16:06:53,811 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"viewNo":0,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1","op":"VIEW_CHANGE_DONE"}', '{"viewNo":0,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1","op":"VIEW_CHANGE_DONE"}']}, 'Node1')
2017-10-23 16:06:53,811 | DEBUG    | node.py              (1272) | unpackNodeMsg | Node4 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"viewNo":0,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1","op":"VIEW_CHANGE_DONE"}', '{"viewNo":0,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1","op":"VIEW_CHANGE_DONE"}']}
2017-10-23 16:06:53,811 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node1: MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}
2017-10-23 16:06:53,811 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}, 'Node1')
2017-10-23 16:06:53,811 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}
2017-10-23 16:06:53,812 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node1: MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}
2017-10-23 16:06:53,812 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}, 'Node1')
2017-10-23 16:06:53,812 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'}
2017-10-23 16:06:53,812 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node1: VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}
2017-10-23 16:06:53,812 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1', 'op': 'VIEW_CHANGE_DONE'}, 'Node1')
2017-10-23 16:06:53,812 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}
2017-10-23 16:06:53,812 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node1: VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}
2017-10-23 16:06:53,813 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1', 'op': 'VIEW_CHANGE_DONE'}, 'Node1')
2017-10-23 16:06:53,813 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}
2017-10-23 16:06:53,813 | DEBUG    | zstack.py            ( 669) | handlePingPong | Node4 got pong from Node1
2017-10-23 16:06:53,813 | DEBUG    | node.py              (2608) | send | Node4 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None}} to 1 recipients: ['Node1']
2017-10-23 16:06:53,813 | DEBUG    | node.py              (2608) | send | Node4 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None}} to 1 recipients: ['Node1']
2017-10-23 16:06:53,814 | DEBUG    | node.py              (2608) | send | Node4 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None}} to 1 recipients: ['Node1']
2017-10-23 16:06:53,818 | DEBUG    | node.py              (2608) | send | Node4 sending message MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None}} to 1 recipients: ['Node1']
2017-10-23 16:06:53,820 | DEBUG    | node.py              (1205) | sendToElector | Node4 sending message to elector: (VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}, 'Node1')
2017-10-23 16:06:53,820 | DEBUG    | node.py              (1205) | sendToElector | Node4 sending message to elector: (VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}, 'Node1')
2017-10-23 16:06:53,820 | DEBUG    | primary_selector.py  ( 106) | _processViewChangeDoneMessage | Node4's primary selector started processing of ViewChangeDone msg from Node1 : VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}
2017-10-23 16:06:53,820 | DEBUG    | primary_selector.py  ( 196) | _hasViewChangeQuorum | Node4 got view change quorum (3 >= 2)
2017-10-23 16:06:53,820 | DEBUG    | primary_selector.py  ( 215) | has_view_change_from_primary | Node4 received ViewChangeDone from primary Node1
2017-10-23 16:06:53,821 | DEBUG    | primary_selector.py  ( 238) | has_sufficient_same_view_change_done_messages | Node4 found acceptable primary Node1 and ledger info ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'), (2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF'))
2017-10-23 16:06:53,821 | DEBUG    | primary_selector.py  ( 271) | _startSelection | Node4 cannot start primary selection since mode is 100
2017-10-23 16:06:53,821 | DEBUG    | primary_selector.py  ( 106) | _processViewChangeDoneMessage | Node4's primary selector started processing of ViewChangeDone msg from Node1 : VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}
2017-10-23 16:06:53,821 | DEBUG    | primary_selector.py  ( 271) | _startSelection | Node4 cannot start primary selection since mode is 100
2017-10-23 16:06:53,821 | DEBUG    | batched.py           ( 100) | flushOutBoxes | Node4 batching 5 msgs to Node1 into one transmission
2017-10-23 16:06:53,821 | TRACE    | batched.py           ( 101) | flushOutBoxes |     messages: deque([b'po', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null}}', b'{"op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null}}'])
2017-10-23 16:06:53,821 | TRACE    | batched.py           ( 110) | flushOutBoxes | Node4 sending payload to Node1: b'{"op":"BATCH","signature":null,"messages":["po","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":1,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":1,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":7,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":7,\\"viewNo\\":null}}"]}'
2017-10-23 16:06:53,821 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"op":"BATCH","signature":null,"messages":["po","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":1,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":1,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":7,\\"viewNo\\":null}}","{\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":7,\\"viewNo\\":null}}"]}' to Node1
2017-10-23 16:06:53,822 | DEBUG    | keep_in_touch.py     (  68) | conns | Node4's connections changed from {'Node2', 'Node3'} to {'Node2', 'Node1', 'Node3'}
2017-10-23 16:06:53,823 | INFO     | keep_in_touch.py     (  98) | _connsChanged | CONNECTION: Node4 now connected to Node1
2017-10-23 16:06:53,823 | DEBUG    | motor.py             (  34) | set_status | Node4 changing status from started_hungry to started
2017-10-23 16:06:53,823 | DEBUG    | node.py              ( 941) | checkInstances | Node4 choosing to start election on the basis of count 4 and nodes {'Node2', 'Node1', 'Node3'}
2017-10-23 16:06:53,823 | DEBUG    | node.py              ( 902) | send_current_state_to_lagging_node | Node4 sending current state CURRENT_STATE{'viewNo': 0, 'primary': [VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'), (2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF')), 'name': 'Node1'}]} to lagged node Node1
2017-10-23 16:06:53,823 | DEBUG    | node.py              (2608) | send | Node4 sending message CURRENT_STATE{'viewNo': 0, 'primary': [VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'), (2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF')), 'name': 'Node1'}]} to 1 recipients: ['Node1']
2017-10-23 16:06:53,824 | DEBUG    | node.py              (2608) | send | Node4 sending message LEDGER_STATUS{'viewNo': None, 'ledgerId': 0, 'txnSeqNo': 4, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': None} to 1 recipients: ['Node1']
2017-10-23 16:06:53,824 | TRACE    | zstack.py            ( 479) | _receiveFromListener | Node4 got 1 messages through listener
2017-10-23 16:06:53,824 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: BATCH{'signature': None, 'messages': ['{"viewNo":0,"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1","viewNo":0}],"op":"CURRENT_STATE"}', '{"viewNo":null,"ppSeqNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ledgerId":0,"op":"LEDGER_STATUS","txnSeqNo":4}', '{"viewNo":null,"ppSeqNo":null,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ledgerId":1,"op":"LEDGER_STATUS","txnSeqNo":7}', '{"viewNo":null,"ppSeqNo":null,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ledgerId":2,"op":"LEDGER_STATUS","txnSeqNo":1}']}
2017-10-23 16:06:53,824 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"viewNo":0,"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1","viewNo":0}],"op":"CURRENT_STATE"}', '{"viewNo":null,"ppSeqNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ledgerId":0,"op":"LEDGER_STATUS","txnSeqNo":4}', '{"viewNo":null,"ppSeqNo":null,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ledgerId":1,"op":"LEDGER_STATUS","txnSeqNo":7}', '{"viewNo":null,"ppSeqNo":null,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ledgerId":2,"op":"LEDGER_STATUS","txnSeqNo":1}']}, 'Node2')
2017-10-23 16:06:53,824 | DEBUG    | node.py              (1272) | unpackNodeMsg | Node4 processing a batch BATCH{'signature': None, 'messages': ['{"viewNo":0,"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1","viewNo":0}],"op":"CURRENT_STATE"}', '{"viewNo":null,"ppSeqNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ledgerId":0,"op":"LEDGER_STATUS","txnSeqNo":4}', '{"viewNo":null,"ppSeqNo":null,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ledgerId":1,"op":"LEDGER_STATUS","txnSeqNo":7}', '{"viewNo":null,"ppSeqNo":null,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ledgerId":2,"op":"LEDGER_STATUS","txnSeqNo":1}']}
2017-10-23 16:06:53,825 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: CURRENT_STATE{'viewNo': 0, 'primary': [{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}]}
2017-10-23 16:06:53,825 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'viewNo': 0, 'op': 'CURRENT_STATE', 'primary': [{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}]}, 'Node2')
2017-10-23 16:06:53,825 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox CURRENT_STATE{'viewNo': 0, 'primary': [{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}]}
2017-10-23 16:06:53,825 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: LEDGER_STATUS{'viewNo': None, 'ledgerId': 0, 'txnSeqNo': 4, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': None}
2017-10-23 16:06:53,825 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'op': 'LEDGER_STATUS', 'viewNo': None, 'ledgerId': 0, 'ppSeqNo': None}, 'Node2')
2017-10-23 16:06:53,825 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox LEDGER_STATUS{'viewNo': None, 'ledgerId': 0, 'txnSeqNo': 4, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': None}
2017-10-23 16:06:53,825 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: LEDGER_STATUS{'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None}
2017-10-23 16:06:53,825 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'txnSeqNo': 7, 'op': 'LEDGER_STATUS', 'viewNo': None, 'ledgerId': 1, 'ppSeqNo': None}, 'Node2')
2017-10-23 16:06:53,826 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox LEDGER_STATUS{'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None}
2017-10-23 16:06:53,826 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: LEDGER_STATUS{'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None}
2017-10-23 16:06:53,826 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'txnSeqNo': 1, 'op': 'LEDGER_STATUS', 'viewNo': None, 'ledgerId': 2, 'ppSeqNo': None}, 'Node2')
2017-10-23 16:06:53,826 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox LEDGER_STATUS{'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None}
2017-10-23 16:06:53,826 | DEBUG    | node.py              ( 907) | process_current_state_message | Node4 processing current state CURRENT_STATE{'viewNo': 0, 'primary': [{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}]} from Node2
2017-10-23 16:06:53,826 | DEBUG    | node.py              (1205) | sendToElector | Node4 sending message to elector: (VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}, 'Node2')
2017-10-23 16:06:53,826 | DEBUG    | ledger_manager.py    ( 246) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'viewNo': None, 'ledgerId': 0, 'txnSeqNo': 4, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': None} from Node2
2017-10-23 16:06:53,827 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 0 of size 4 with 4
2017-10-23 16:06:53,827 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 0 of size 4 with 4
2017-10-23 16:06:53,827 | DEBUG    | ledger_manager.py    ( 312) | processLedgerStatus | Node4 found out from {'Node2', 'Node3'} that its ledger of type 0 is latest
2017-10-23 16:06:53,827 | DEBUG    | ledger_manager.py    ( 315) | processLedgerStatus | Node4 found from ledger status LEDGER_STATUS{'viewNo': None, 'ledgerId': 0, 'txnSeqNo': 4, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': None} that it does not need catchup
2017-10-23 16:06:53,827 | DEBUG    | node.py              (1479) | preLedgerCatchUp | Node4 going to process any ordered requests before starting catchup.
2017-10-23 16:06:53,827 | DEBUG    | replica.py           (2259) | _remove_ordered_from_queue | Node4:0 going to remove 0 Ordered messages from outbox
2017-10-23 16:06:53,827 | DEBUG    | node.py              (1888) | force_process_ordered | Node4 processed 0 Ordered batches for instance 0 before starting catch up
2017-10-23 16:06:53,827 | DEBUG    | replica.py           (2259) | _remove_ordered_from_queue | Node4:1 going to remove 0 Ordered messages from outbox
2017-10-23 16:06:53,827 | DEBUG    | node.py              (1888) | force_process_ordered | Node4 processed 0 Ordered batches for instance 1 before starting catch up
2017-10-23 16:06:53,827 | DEBUG    | node.py              (2469) | processStashedOrderedReqs | Node4 processed 0 stashed ordered requests
2017-10-23 16:06:53,827 | DEBUG    | monitor.py           ( 192) | reset | Node4's Monitor being reset
2017-10-23 16:06:53,828 | INFO     | node.py              (1489) | preLedgerCatchUp | Node4 reverted 0 batches before starting catch up for ledger 0
2017-10-23 16:06:53,828 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 0 of size 4 with 4
2017-10-23 16:06:53,828 | DEBUG    | node.py              ( 941) | checkInstances | Node4 choosing to start election on the basis of count 4 and nodes {'Node2', 'Node1', 'Node3'}
2017-10-23 16:06:53,828 | DEBUG    | node.py              (2608) | send | Node4 sending message MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node1']
2017-10-23 16:06:53,828 | DEBUG    | node.py              ( 855) | _ask_for_ledger_status | Node4 asking Node1 for ledger status of ledger 2
2017-10-23 16:06:53,828 | DEBUG    | node.py              (2608) | send | Node4 sending message MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node2']
2017-10-23 16:06:53,828 | DEBUG    | node.py              ( 855) | _ask_for_ledger_status | Node4 asking Node2 for ledger status of ledger 2
2017-10-23 16:06:53,828 | DEBUG    | node.py              (2608) | send | Node4 sending message MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node3']
2017-10-23 16:06:53,828 | DEBUG    | node.py              ( 855) | _ask_for_ledger_status | Node4 asking Node3 for ledger status of ledger 2
2017-10-23 16:06:53,829 | DEBUG    | node.py              (2608) | send | Node4 sending message MESSAGE_REQUEST{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS'} to all recipients: ['Node2', 'Node1', 'Node3']
2017-10-23 16:06:53,829 | DEBUG    | node.py              ( 855) | _ask_for_ledger_status | Node4 asking Node4 for ledger status of ledger 2
2017-10-23 16:06:53,829 | DEBUG    | ledger_manager.py    (1027) | processStashedLedgerStatuses | Node4 going to process 1 stashed ledger statuses for ledger 2
2017-10-23 16:06:53,829 | DEBUG    | ledger_manager.py    ( 246) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None} from Node3
2017-10-23 16:06:53,829 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 2 of size 1 with 1
2017-10-23 16:06:53,829 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 2 of size 1 with 1
2017-10-23 16:06:53,829 | INFO     | ledger_manager.py    ( 848) | catchupCompleted | CATCH-UP: Node4 completed catching up ledger 0, caught up 0 in total
2017-10-23 16:06:53,829 | DEBUG    | ledger_manager.py    ( 246) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None} from Node2
2017-10-23 16:06:53,829 | DEBUG    | ledger_manager.py    (1016) | stashLedgerStatus | Node4 stashing ledger status LEDGER_STATUS{'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None} from Node2
2017-10-23 16:06:53,830 | DEBUG    | ledger_manager.py    ( 246) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None} from Node2
2017-10-23 16:06:53,831 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 2 of size 1 with 1
2017-10-23 16:06:53,831 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 2 of size 1 with 1
2017-10-23 16:06:53,831 | DEBUG    | ledger_manager.py    ( 312) | processLedgerStatus | Node4 found out from {'Node2', 'Node3'} that its ledger of type 2 is latest
2017-10-23 16:06:53,831 | DEBUG    | ledger_manager.py    ( 315) | processLedgerStatus | Node4 found from ledger status LEDGER_STATUS{'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None} that it does not need catchup
2017-10-23 16:06:53,831 | DEBUG    | node.py              (1479) | preLedgerCatchUp | Node4 going to process any ordered requests before starting catchup.
2017-10-23 16:06:53,831 | DEBUG    | replica.py           (2259) | _remove_ordered_from_queue | Node4:0 going to remove 0 Ordered messages from outbox
2017-10-23 16:06:53,831 | DEBUG    | node.py              (1888) | force_process_ordered | Node4 processed 0 Ordered batches for instance 0 before starting catch up
2017-10-23 16:06:53,831 | DEBUG    | replica.py           (2259) | _remove_ordered_from_queue | Node4:1 going to remove 0 Ordered messages from outbox
2017-10-23 16:06:53,831 | DEBUG    | node.py              (1888) | force_process_ordered | Node4 processed 0 Ordered batches for instance 1 before starting catch up
2017-10-23 16:06:53,831 | DEBUG    | node.py              (2469) | processStashedOrderedReqs | Node4 processed 0 stashed ordered requests
2017-10-23 16:06:53,831 | DEBUG    | monitor.py           ( 192) | reset | Node4's Monitor being reset
2017-10-23 16:06:53,832 | INFO     | node.py              (1489) | preLedgerCatchUp | Node4 reverted 0 batches before starting catch up for ledger 2
2017-10-23 16:06:53,832 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 2 of size 1 with 1
2017-10-23 16:06:53,832 | DEBUG    | pool_config.py       (  35) | processLedger | <sovrin_node.server.pool_config.PoolConfig object at 0x7f0005811668> processing config ledger for any POOL_CONFIGs
2017-10-23 16:06:53,832 | DEBUG    | upgrader.py          ( 208) | processLedger | 4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA processing config ledger for any upgrades
2017-10-23 16:06:53,832 | INFO     | upgrader.py          ( 214) | processLedger | 4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA found upgrade START txn OrderedDict([('action', 'start'), ('force', True), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade-1137'), ('reqId', 1508774754234674), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-10-17T11:35:00.000000+00:00'), ('4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', '2017-10-17T11:40:00.000000+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-10-17T11:25:00.000000+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-10-17T11:30:00.000000+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-10-17T11:20:00.000000+00:00')])), ('sha256', 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3'), ('signature', '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ'), ('timeout', 10), ('txnTime', 1508774754), ('type', '109'), ('version', '1.1.37'), ('seqNo', 1)])
2017-10-23 16:06:53,832 | INFO     | upgrader.py          ( 292) | handleUpgradeTxn | Node 'Node4' handles upgrade txn OrderedDict([('action', 'start'), ('force', True), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade-1137'), ('reqId', 1508774754234674), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-10-17T11:35:00.000000+00:00'), ('4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc', '2017-10-17T11:40:00.000000+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-10-17T11:25:00.000000+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-10-17T11:30:00.000000+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-10-17T11:20:00.000000+00:00')])), ('sha256', 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3'), ('signature', '3R1H8sKSqCKxj3VJxoU8RUbFt9RhEzjYniG5afzcxkzxdogseFQ866SgmtmZBvVRzvV6G88JcVmQzf6dMH4XmJzQ'), ('timeout', 10), ('txnTime', 1508774754), ('type', '109'), ('version', '1.1.37'), ('seqNo', 1)])
2017-10-23 16:06:53,832 | DEBUG    | node.py              (2608) | send | Node4 sending message MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node1']
2017-10-23 16:06:53,832 | DEBUG    | node.py              ( 855) | _ask_for_ledger_status | Node4 asking Node1 for ledger status of ledger 1
2017-10-23 16:06:53,832 | DEBUG    | node.py              (2608) | send | Node4 sending message MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node2']
2017-10-23 16:06:53,833 | DEBUG    | node.py              ( 855) | _ask_for_ledger_status | Node4 asking Node2 for ledger status of ledger 1
2017-10-23 16:06:53,833 | DEBUG    | node.py              (2608) | send | Node4 sending message MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} to 1 recipients: ['Node3']
2017-10-23 16:06:53,833 | DEBUG    | node.py              ( 855) | _ask_for_ledger_status | Node4 asking Node3 for ledger status of ledger 1
2017-10-23 16:06:53,833 | DEBUG    | node.py              (2608) | send | Node4 sending message MESSAGE_REQUEST{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS'} to all recipients: ['Node2', 'Node1', 'Node3']
2017-10-23 16:06:53,833 | DEBUG    | node.py              ( 855) | _ask_for_ledger_status | Node4 asking Node4 for ledger status of ledger 1
2017-10-23 16:06:53,835 | DEBUG    | ledger_manager.py    (1027) | processStashedLedgerStatuses | Node4 going to process 2 stashed ledger statuses for ledger 1
2017-10-23 16:06:53,835 | DEBUG    | ledger_manager.py    ( 246) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None} from Node3
2017-10-23 16:06:53,835 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 1 of size 7 with 7
2017-10-23 16:06:53,836 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 1 of size 7 with 7
2017-10-23 16:06:53,836 | DEBUG    | ledger_manager.py    ( 246) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None} from Node2
2017-10-23 16:06:53,836 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 1 of size 7 with 7
2017-10-23 16:06:53,836 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 1 of size 7 with 7
2017-10-23 16:06:53,836 | DEBUG    | ledger_manager.py    ( 312) | processLedgerStatus | Node4 found out from {'Node2', 'Node3'} that its ledger of type 1 is latest
2017-10-23 16:06:53,836 | DEBUG    | ledger_manager.py    ( 315) | processLedgerStatus | Node4 found from ledger status LEDGER_STATUS{'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None} that it does not need catchup
2017-10-23 16:06:53,836 | DEBUG    | node.py              (1479) | preLedgerCatchUp | Node4 going to process any ordered requests before starting catchup.
2017-10-23 16:06:53,836 | DEBUG    | replica.py           (2259) | _remove_ordered_from_queue | Node4:0 going to remove 0 Ordered messages from outbox
2017-10-23 16:06:53,836 | DEBUG    | node.py              (1888) | force_process_ordered | Node4 processed 0 Ordered batches for instance 0 before starting catch up
2017-10-23 16:06:53,836 | DEBUG    | replica.py           (2259) | _remove_ordered_from_queue | Node4:1 going to remove 0 Ordered messages from outbox
2017-10-23 16:06:53,837 | DEBUG    | node.py              (1888) | force_process_ordered | Node4 processed 0 Ordered batches for instance 1 before starting catch up
2017-10-23 16:06:53,837 | DEBUG    | node.py              (2469) | processStashedOrderedReqs | Node4 processed 0 stashed ordered requests
2017-10-23 16:06:53,837 | DEBUG    | monitor.py           ( 192) | reset | Node4's Monitor being reset
2017-10-23 16:06:53,837 | INFO     | node.py              (1489) | preLedgerCatchUp | Node4 reverted 0 batches before starting catch up for ledger 1
2017-10-23 16:06:53,837 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 1 of size 7 with 7
2017-10-23 16:06:53,837 | INFO     | ledger_manager.py    ( 848) | catchupCompleted | CATCH-UP: Node4 completed catching up ledger 1, caught up 0 in total
2017-10-23 16:06:53,837 | DEBUG    | node.py              (1609) | num_txns_caught_up_in_last_catchup | Node4 caught up to 0 txns in the last catchup
2017-10-23 16:06:53,837 | DEBUG    | node.py              (2469) | processStashedOrderedReqs | Node4 processed 0 stashed ordered requests
2017-10-23 16:06:53,837 | DEBUG    | monitor.py           ( 192) | reset | Node4's Monitor being reset
2017-10-23 16:06:53,837 | DEBUG    | primary_selector.py  ( 196) | _hasViewChangeQuorum | Node4 got view change quorum (3 >= 2)
2017-10-23 16:06:53,838 | DEBUG    | node.py              (1549) | is_catchup_needed | Node4 is caught up for the current view 0
2017-10-23 16:06:53,838 | INFO     | node.py              (1537) | allLedgersCaughtUp | CATCH-UP: Node4 does not need any more catchups
2017-10-23 16:06:53,838 | DEBUG    | primary_decider.py   ( 134) | send | Node4's elector sending VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'), (2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF')], 'name': 'Node1'}
2017-10-23 16:06:53,838 | DEBUG    | primary_selector.py  ( 281) | _startSelection | Node4 starting selection
2017-10-23 16:06:53,838 | DISPLAY  | primary_selector.py  ( 291) | _startSelection | PRIMARY SELECTION: Node4:0 selected primary Node1:0 for instance 0 (view 0)
2017-10-23 16:06:53,838 | INFO     | node.py              ( 510) | start_participating | Node4 started participating
2017-10-23 16:06:53,839 | DEBUG    | replica.py           ( 408) | primaryName | Node4:0 setting primaryName for view no 0 to: Node1:0
2017-10-23 16:06:53,839 | DEBUG    | replica.py           (1694) | _gc | Node4:0 cleaning up till (0, 0)
2017-10-23 16:06:53,839 | DEBUG    | replica.py           (1709) | _gc | Node4:0 found 0 3-phase keys to clean
2017-10-23 16:06:53,839 | DEBUG    | replica.py           (1711) | _gc | Node4:0 found 0 request keys to clean
2017-10-23 16:06:53,839 | DEBUG    | replica.py           ( 313) | h | Node4:0 set watermarks as 0 300
2017-10-23 16:06:53,839 | DISPLAY  | primary_selector.py  ( 315) | _startSelection | VIEW CHANGE: Node4:0 declares view change 0 as completed for instance 0, new primary is Node1:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'), (2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF')]
2017-10-23 16:06:53,839 | DISPLAY  | primary_selector.py  ( 291) | _startSelection | PRIMARY SELECTION: Node4:1 selected primary Node2:1 for instance 1 (view 0)
2017-10-23 16:06:53,839 | DEBUG    | replica.py           ( 408) | primaryName | Node4:1 setting primaryName for view no 0 to: Node2:1
2017-10-23 16:06:53,840 | DEBUG    | replica.py           (1694) | _gc | Node4:1 cleaning up till (0, 0)
2017-10-23 16:06:53,840 | DEBUG    | replica.py           (1709) | _gc | Node4:1 found 0 3-phase keys to clean
2017-10-23 16:06:53,840 | DEBUG    | replica.py           (1711) | _gc | Node4:1 found 0 request keys to clean
2017-10-23 16:06:53,840 | DEBUG    | replica.py           ( 313) | h | Node4:1 set watermarks as 0 300
2017-10-23 16:06:53,840 | DEBUG    | replica.py           ( 514) | _setup_for_non_master | Node4:1 Setting last ordered for non-master as (0, 0)
2017-10-23 16:06:53,840 | DEBUG    | replica.py           ( 323) | last_ordered_3pc | Node4:1 set last ordered as (0, 0)
2017-10-23 16:06:53,840 | DISPLAY  | primary_selector.py  ( 315) | _startSelection | VIEW CHANGE: Node4:1 declares view change 0 as completed for instance 1, new primary is Node2:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'), (2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF')]
2017-10-23 16:06:53,840 | INFO     | upgrader.py          ( 150) | should_notify_about_upgrade_result | Node's 'Node4' last upgrade txn is None
2017-10-23 16:06:53,840 | INFO     | ledger_manager.py    ( 848) | catchupCompleted | CATCH-UP: Node4 completed catching up ledger 2, caught up 0 in total
2017-10-23 16:06:53,840 | DEBUG    | node.py              (1609) | num_txns_caught_up_in_last_catchup | Node4 caught up to 0 txns in the last catchup
2017-10-23 16:06:53,840 | DEBUG    | node.py              (2469) | processStashedOrderedReqs | Node4 processed 0 stashed ordered requests
2017-10-23 16:06:53,841 | DEBUG    | monitor.py           ( 192) | reset | Node4's Monitor being reset
2017-10-23 16:06:53,841 | DEBUG    | primary_selector.py  ( 196) | _hasViewChangeQuorum | Node4 got view change quorum (4 >= 2)
2017-10-23 16:06:53,841 | DEBUG    | node.py              (1549) | is_catchup_needed | Node4 is caught up for the current view 0
2017-10-23 16:06:53,842 | INFO     | node.py              (1537) | allLedgersCaughtUp | CATCH-UP: Node4 does not need any more catchups
2017-10-23 16:06:53,842 | DEBUG    | primary_selector.py  ( 281) | _startSelection | Node4 starting selection
2017-10-23 16:06:53,842 | DEBUG    | primary_selector.py  ( 284) | _startSelection | Node4:0 already has a primary
2017-10-23 16:06:53,842 | DEBUG    | primary_selector.py  ( 284) | _startSelection | Node4:1 already has a primary
2017-10-23 16:06:53,842 | INFO     | node.py              (1624) | no_more_catchups_needed | Node4 starting to participate since catchup is done, primaries are selected but mode was not set to participating
2017-10-23 16:06:53,842 | INFO     | node.py              ( 510) | start_participating | Node4 started participating
2017-10-23 16:06:53,842 | TRACE    | zstack.py            ( 479) | _receiveFromListener | Node4C got 1 messages through listener
2017-10-23 16:06:53,843 | DEBUG    | zstack.py            ( 663) | handlePingPong | Node4C got ping from b'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*'
2017-10-23 16:06:53,843 | TRACE    | zstack.py            ( 754) | transmitThroughListener | Node4C transmitting b'po' to b'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*' through listener socket
2017-10-23 16:06:53,843 | DEBUG    | zstack.py            ( 645) | sendPingPong | Node4C ponged b'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*'
2017-10-23 16:06:53,843 | DEBUG    | node.py              (2608) | send | Node4 sending message VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'), (2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF')], 'name': 'Node1'} to all recipients: ['Node2', 'Node1', 'Node3']
2017-10-23 16:06:53,843 | DEBUG    | primary_selector.py  ( 106) | _processViewChangeDoneMessage | Node4's primary selector started processing of ViewChangeDone msg from Node2 : VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}
2017-10-23 16:06:53,843 | DEBUG    | message_processor.py (  29) | discard | Node4 discarding message VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'} because it already decided primary which is Node1:0
2017-10-23 16:06:53,843 | DEBUG    | batched.py           ( 100) | flushOutBoxes | Node4 batching 5 msgs to Node2 into one transmission
2017-10-23 16:06:53,843 | TRACE    | batched.py           ( 101) | flushOutBoxes |     messages: deque([b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', b'{"viewNo":0,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1","op":"VIEW_CHANGE_DONE"}'])
2017-10-23 16:06:53,843 | TRACE    | batched.py           ( 110) | flushOutBoxes | Node4 sending payload to Node2: b'{"op":"BATCH","signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"viewNo\\":0,\\"ledgerInfo\\":[[0,4,\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\"],[1,7,\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\"],[2,1,\\"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF\\"]],\\"name\\":\\"Node1\\",\\"op\\":\\"VIEW_CHANGE_DONE\\"}"]}'
2017-10-23 16:06:53,844 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"op":"BATCH","signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"viewNo\\":0,\\"ledgerInfo\\":[[0,4,\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\"],[1,7,\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\"],[2,1,\\"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF\\"]],\\"name\\":\\"Node1\\",\\"op\\":\\"VIEW_CHANGE_DONE\\"}"]}' to Node2
2017-10-23 16:06:53,844 | DEBUG    | batched.py           ( 100) | flushOutBoxes | Node4 batching 7 msgs to Node1 into one transmission
2017-10-23 16:06:53,846 | TRACE    | batched.py           ( 101) | flushOutBoxes |     messages: deque([b'{"viewNo":0,"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1","viewNo":0}],"op":"CURRENT_STATE"}', b'{"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","txnSeqNo":4,"viewNo":null,"op":"LEDGER_STATUS","ledgerId":0,"ppSeqNo":null}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', b'{"viewNo":0,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1","op":"VIEW_CHANGE_DONE"}'])
2017-10-23 16:06:53,846 | TRACE    | batched.py           ( 110) | flushOutBoxes | Node4 sending payload to Node1: b'{"op":"BATCH","signature":null,"messages":["{\\"viewNo\\":0,\\"primary\\":[{\\"ledgerInfo\\":[[0,4,\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\"],[1,7,\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\"],[2,1,\\"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF\\"]],\\"name\\":\\"Node1\\",\\"viewNo\\":0}],\\"op\\":\\"CURRENT_STATE\\"}","{\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"txnSeqNo\\":4,\\"viewNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":0,\\"ppSeqNo\\":null}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"viewNo\\":0,\\"ledgerInfo\\":[[0,4,\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\"],[1,7,\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\"],[2,1,\\"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF\\"]],\\"name\\":\\"Node1\\",\\"op\\":\\"VIEW_CHANGE_DONE\\"}"]}'
2017-10-23 16:06:53,846 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"op":"BATCH","signature":null,"messages":["{\\"viewNo\\":0,\\"primary\\":[{\\"ledgerInfo\\":[[0,4,\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\"],[1,7,\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\"],[2,1,\\"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF\\"]],\\"name\\":\\"Node1\\",\\"viewNo\\":0}],\\"op\\":\\"CURRENT_STATE\\"}","{\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"txnSeqNo\\":4,\\"viewNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"ledgerId\\":0,\\"ppSeqNo\\":null}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"viewNo\\":0,\\"ledgerInfo\\":[[0,4,\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\"],[1,7,\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\"],[2,1,\\"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF\\"]],\\"name\\":\\"Node1\\",\\"op\\":\\"VIEW_CHANGE_DONE\\"}"]}' to Node1
2017-10-23 16:06:53,846 | DEBUG    | batched.py           ( 100) | flushOutBoxes | Node4 batching 5 msgs to Node3 into one transmission
2017-10-23 16:06:53,846 | TRACE    | batched.py           ( 101) | flushOutBoxes |     messages: deque([b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', b'{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', b'{"viewNo":0,"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1","op":"VIEW_CHANGE_DONE"}'])
2017-10-23 16:06:53,847 | TRACE    | batched.py           ( 110) | flushOutBoxes | Node4 sending payload to Node3: b'{"op":"BATCH","signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"viewNo\\":0,\\"ledgerInfo\\":[[0,4,\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\"],[1,7,\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\"],[2,1,\\"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF\\"]],\\"name\\":\\"Node1\\",\\"op\\":\\"VIEW_CHANGE_DONE\\"}"]}'
2017-10-23 16:06:53,847 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"op":"BATCH","signature":null,"messages":["{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1},\\"msg_type\\":\\"LEDGER_STATUS\\"}","{\\"viewNo\\":0,\\"ledgerInfo\\":[[0,4,\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\"],[1,7,\\"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o\\"],[2,1,\\"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF\\"]],\\"name\\":\\"Node1\\",\\"op\\":\\"VIEW_CHANGE_DONE\\"}"]}' to Node3
2017-10-23 16:06:53,847 | TRACE    | zstack.py            ( 479) | _receiveFromListener | Node4 got 1 messages through listener
2017-10-23 16:06:53,848 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node1: BATCH{'signature': None, 'messages': ['{"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1","viewNo":0}],"viewNo":0,"op":"CURRENT_STATE"}', '{"txnSeqNo":4,"op":"LEDGER_STATUS","ppSeqNo":null,"viewNo":null,"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"}', '{"txnSeqNo":7,"op":"LEDGER_STATUS","ppSeqNo":null,"viewNo":null,"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"}', '{"txnSeqNo":1,"op":"LEDGER_STATUS","ppSeqNo":null,"viewNo":null,"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"}']}
2017-10-23 16:06:53,848 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1","viewNo":0}],"viewNo":0,"op":"CURRENT_STATE"}', '{"txnSeqNo":4,"op":"LEDGER_STATUS","ppSeqNo":null,"viewNo":null,"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"}', '{"txnSeqNo":7,"op":"LEDGER_STATUS","ppSeqNo":null,"viewNo":null,"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"}', '{"txnSeqNo":1,"op":"LEDGER_STATUS","ppSeqNo":null,"viewNo":null,"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"}']}, 'Node1')
2017-10-23 16:06:53,848 | DEBUG    | node.py              (1272) | unpackNodeMsg | Node4 processing a batch BATCH{'signature': None, 'messages': ['{"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,7,"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"],[2,1,"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"]],"name":"Node1","viewNo":0}],"viewNo":0,"op":"CURRENT_STATE"}', '{"txnSeqNo":4,"op":"LEDGER_STATUS","ppSeqNo":null,"viewNo":null,"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"}', '{"txnSeqNo":7,"op":"LEDGER_STATUS","ppSeqNo":null,"viewNo":null,"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o"}', '{"txnSeqNo":1,"op":"LEDGER_STATUS","ppSeqNo":null,"viewNo":null,"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF"}']}
2017-10-23 16:06:53,848 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node1: CURRENT_STATE{'viewNo': 0, 'primary': [{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}]}
2017-10-23 16:06:53,848 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'viewNo': 0, 'op': 'CURRENT_STATE', 'primary': [{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}]}, 'Node1')
2017-10-23 16:06:53,848 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox CURRENT_STATE{'viewNo': 0, 'primary': [{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}]}
2017-10-23 16:06:53,848 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node1: LEDGER_STATUS{'viewNo': None, 'ledgerId': 0, 'txnSeqNo': 4, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': None}
2017-10-23 16:06:53,849 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'viewNo': None, 'op': 'LEDGER_STATUS', 'ledgerId': 0, 'ppSeqNo': None}, 'Node1')
2017-10-23 16:06:53,849 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox LEDGER_STATUS{'viewNo': None, 'ledgerId': 0, 'txnSeqNo': 4, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': None}
2017-10-23 16:06:53,849 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node1: LEDGER_STATUS{'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None}
2017-10-23 16:06:53,849 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'txnSeqNo': 7, 'viewNo': None, 'op': 'LEDGER_STATUS', 'ledgerId': 1, 'ppSeqNo': None}, 'Node1')
2017-10-23 16:06:53,849 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox LEDGER_STATUS{'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None}
2017-10-23 16:06:53,849 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node1: LEDGER_STATUS{'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None}
2017-10-23 16:06:53,849 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'txnSeqNo': 1, 'viewNo': None, 'op': 'LEDGER_STATUS', 'ledgerId': 2, 'ppSeqNo': None}, 'Node1')
2017-10-23 16:06:53,849 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox LEDGER_STATUS{'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None}
2017-10-23 16:06:53,849 | DEBUG    | node.py              ( 907) | process_current_state_message | Node4 processing current state CURRENT_STATE{'viewNo': 0, 'primary': [{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}]} from Node1
2017-10-23 16:06:53,850 | DEBUG    | node.py              (1205) | sendToElector | Node4 sending message to elector: (VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}, 'Node1')
2017-10-23 16:06:53,850 | DEBUG    | ledger_manager.py    ( 246) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'viewNo': None, 'ledgerId': 0, 'txnSeqNo': 4, 'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ppSeqNo': None} from Node1
2017-10-23 16:06:53,850 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 0 of size 4 with 4
2017-10-23 16:06:53,850 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 0 of size 4 with 4
2017-10-23 16:06:53,853 | DEBUG    | ledger_manager.py    ( 246) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None} from Node1
2017-10-23 16:06:53,853 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 1 of size 7 with 7
2017-10-23 16:06:53,853 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 1 of size 7 with 7
2017-10-23 16:06:53,853 | DEBUG    | ledger_manager.py    ( 246) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None} from Node1
2017-10-23 16:06:53,853 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 2 of size 1 with 1
2017-10-23 16:06:53,854 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 2 of size 1 with 1
2017-10-23 16:06:53,854 | DEBUG    | primary_selector.py  ( 106) | _processViewChangeDoneMessage | Node4's primary selector started processing of ViewChangeDone msg from Node1 : VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'}
2017-10-23 16:06:53,854 | DEBUG    | message_processor.py (  29) | discard | Node4 discarding message VIEW_CHANGE_DONE{'viewNo': 0, 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 7, '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o'], [2, 1, 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF']], 'name': 'Node1'} because it already decided primary which is Node1:0
2017-10-23 16:06:53,855 | TRACE    | zstack.py            ( 479) | _receiveFromListener | Node4 got 1 messages through listener
2017-10-23 16:06:53,855 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS"}']}
2017-10-23 16:06:53,855 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS"}']}, 'Node2')
2017-10-23 16:06:53,855 | DEBUG    | node.py              (1272) | unpackNodeMsg | Node4 processing a batch BATCH{'signature': None, 'messages': ['{"params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS"}', '{"params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null},"op":"MESSAGE_RESPONSE","msg_type":"LEDGER_STATUS"}']}
2017-10-23 16:06:53,856 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None}}
2017-10-23 16:06:53,856 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None}}, 'Node2')
2017-10-23 16:06:53,856 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None}}
2017-10-23 16:06:53,856 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None}}
2017-10-23 16:06:53,856 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None}}, 'Node2')
2017-10-23 16:06:53,856 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None}}
2017-10-23 16:06:53,856 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None}}
2017-10-23 16:06:53,856 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None}}, 'Node2')
2017-10-23 16:06:53,857 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None}}
2017-10-23 16:06:53,857 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None}}
2017-10-23 16:06:53,857 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None}}, 'Node2')
2017-10-23 16:06:53,857 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None}}
2017-10-23 16:06:53,857 | DEBUG    | ledger_manager.py    ( 246) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None} from Node2
2017-10-23 16:06:53,857 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 2 of size 1 with 1
2017-10-23 16:06:53,857 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 2 of size 1 with 1
2017-10-23 16:06:53,858 | DEBUG    | ledger_manager.py    ( 312) | processLedgerStatus | Node4 found out from {'Node2', 'Node1'} that its ledger of type 2 is latest
2017-10-23 16:06:53,858 | DEBUG    | ledger_manager.py    ( 246) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None} from Node2
2017-10-23 16:06:53,858 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 2 of size 1 with 1
2017-10-23 16:06:53,858 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 2 of size 1 with 1
2017-10-23 16:06:53,858 | DEBUG    | ledger_manager.py    ( 312) | processLedgerStatus | Node4 found out from {'Node2', 'Node1'} that its ledger of type 2 is latest
2017-10-23 16:06:53,858 | DEBUG    | ledger_manager.py    ( 246) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None} from Node2
2017-10-23 16:06:53,858 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 1 of size 7 with 7
2017-10-23 16:06:53,859 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 1 of size 7 with 7
2017-10-23 16:06:53,859 | DEBUG    | ledger_manager.py    ( 312) | processLedgerStatus | Node4 found out from {'Node2', 'Node1'} that its ledger of type 1 is latest
2017-10-23 16:06:53,859 | DEBUG    | ledger_manager.py    ( 246) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None} from Node2
2017-10-23 16:06:53,859 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 1 of size 7 with 7
2017-10-23 16:06:53,859 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 1 of size 7 with 7
2017-10-23 16:06:53,859 | DEBUG    | ledger_manager.py    ( 312) | processLedgerStatus | Node4 found out from {'Node2', 'Node1'} that its ledger of type 1 is latest
2017-10-23 16:06:53,860 | TRACE    | zstack.py            ( 479) | _receiveFromListener | Node4 got 1 messages through listener
2017-10-23 16:06:53,860 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null}}']}
2017-10-23 16:06:53,861 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null}}']}, 'Node3')
2017-10-23 16:06:53,861 | DEBUG    | node.py              (1272) | unpackNodeMsg | Node4 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null}}']}
2017-10-23 16:06:53,861 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None}}
2017-10-23 16:06:53,861 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None}}, 'Node3')
2017-10-23 16:06:53,861 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None}}
2017-10-23 16:06:53,861 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None}}
2017-10-23 16:06:53,861 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None}}, 'Node3')
2017-10-23 16:06:53,861 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None}}
2017-10-23 16:06:53,862 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None}}
2017-10-23 16:06:53,862 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None}}, 'Node3')
2017-10-23 16:06:53,862 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None}}
2017-10-23 16:06:53,862 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None}}
2017-10-23 16:06:53,862 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None}}, 'Node3')
2017-10-23 16:06:53,862 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None}}
2017-10-23 16:06:53,862 | DEBUG    | ledger_manager.py    ( 246) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None} from Node3
2017-10-23 16:06:53,863 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 2 of size 1 with 1
2017-10-23 16:06:53,863 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 2 of size 1 with 1
2017-10-23 16:06:53,864 | DEBUG    | ledger_manager.py    ( 312) | processLedgerStatus | Node4 found out from {'Node2', 'Node1', 'Node3'} that its ledger of type 2 is latest
2017-10-23 16:06:53,864 | DEBUG    | ledger_manager.py    ( 246) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None} from Node3
2017-10-23 16:06:53,865 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 2 of size 1 with 1
2017-10-23 16:06:53,865 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 2 of size 1 with 1
2017-10-23 16:06:53,865 | DEBUG    | ledger_manager.py    ( 312) | processLedgerStatus | Node4 found out from {'Node2', 'Node1', 'Node3'} that its ledger of type 2 is latest
2017-10-23 16:06:53,865 | DEBUG    | ledger_manager.py    ( 246) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None} from Node3
2017-10-23 16:06:53,865 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 1 of size 7 with 7
2017-10-23 16:06:53,865 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 1 of size 7 with 7
2017-10-23 16:06:53,865 | DEBUG    | ledger_manager.py    ( 312) | processLedgerStatus | Node4 found out from {'Node2', 'Node1', 'Node3'} that its ledger of type 1 is latest
2017-10-23 16:06:53,866 | DEBUG    | ledger_manager.py    ( 246) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None} from Node3
2017-10-23 16:06:53,866 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 1 of size 7 with 7
2017-10-23 16:06:53,866 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 1 of size 7 with 7
2017-10-23 16:06:53,866 | DEBUG    | ledger_manager.py    ( 312) | processLedgerStatus | Node4 found out from {'Node2', 'Node1', 'Node3'} that its ledger of type 1 is latest
2017-10-23 16:06:53,866 | TRACE    | zstack.py            ( 479) | _receiveFromListener | Node4C got 1 messages through listener
2017-10-23 16:06:53,867 | TRACE    | node.py              (1388) | validateClientMsg | Node4C received CLIENT message: LEDGER_STATUS{'viewNo': None, 'ledgerId': 0, 'txnSeqNo': 4, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'ppSeqNo': None}
2017-10-23 16:06:53,867 | DEBUG    | node.py              (1434) | processClientInBox | Node4C processing b'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*' request LEDGER_STATUS{'viewNo': None, 'ledgerId': 0, 'txnSeqNo': 4, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'ppSeqNo': None}
2017-10-23 16:06:53,867 | DEBUG    | ledger_manager.py    ( 246) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'viewNo': None, 'ledgerId': 0, 'txnSeqNo': 4, 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'ppSeqNo': None} from b'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*'
2017-10-23 16:06:53,867 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 0 of size 4 with 4
2017-10-23 16:06:53,867 | TRACE    | zstack.py            ( 754) | transmitThroughListener | Node4C transmitting b'{"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","txnSeqNo":4,"viewNo":null,"op":"LEDGER_STATUS","ledgerId":0,"ppSeqNo":null}' to b'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*' through listener socket
2017-10-23 16:06:53,867 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 0 of size 4 with 4
2017-10-23 16:06:53,867 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 0 of size 4 with 4
2017-10-23 16:06:53,879 | TRACE    | zstack.py            ( 479) | _receiveFromListener | Node4 got 1 messages through listener
2017-10-23 16:06:53,880 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node1: BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}']}
2017-10-23 16:06:53,880 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}']}, 'Node1')
2017-10-23 16:06:53,880 | DEBUG    | node.py              (1272) | unpackNodeMsg | Node4 processing a batch BATCH{'signature': None, 'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF","ppSeqNo":null,"txnSeqNo":1,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o","ppSeqNo":null,"txnSeqNo":7,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}']}
2017-10-23 16:06:53,880 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node1: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None}}
2017-10-23 16:06:53,880 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None}}, 'Node1')
2017-10-23 16:06:53,880 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None}}
2017-10-23 16:06:53,881 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node1: MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None}}
2017-10-23 16:06:53,881 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None}}, 'Node1')
2017-10-23 16:06:53,881 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 2}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None}}
2017-10-23 16:06:53,881 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node1: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None}}
2017-10-23 16:06:53,881 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None}}, 'Node1')
2017-10-23 16:06:53,881 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None}}
2017-10-23 16:06:53,881 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node1: MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None}}
2017-10-23 16:06:53,881 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None}}, 'Node1')
2017-10-23 16:06:53,881 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox MESSAGE_RESPONSE{'params': {'ledgerId': 1}, 'msg_type': 'LEDGER_STATUS', 'msg': {'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None}}
2017-10-23 16:06:53,882 | DEBUG    | ledger_manager.py    ( 246) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None} from Node1
2017-10-23 16:06:53,882 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 2 of size 1 with 1
2017-10-23 16:06:53,882 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 2 of size 1 with 1
2017-10-23 16:06:53,882 | DEBUG    | ledger_manager.py    ( 312) | processLedgerStatus | Node4 found out from {'Node2', 'Node1', 'Node3'} that its ledger of type 2 is latest
2017-10-23 16:06:53,883 | DEBUG    | ledger_manager.py    ( 246) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'viewNo': None, 'ledgerId': 2, 'txnSeqNo': 1, 'merkleRoot': 'J1HcLPgXFVNVgv8hhhD5Nbxg4taz8MEdD61eGZkkEJF', 'ppSeqNo': None} from Node1
2017-10-23 16:06:53,883 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 2 of size 1 with 1
2017-10-23 16:06:53,883 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 2 of size 1 with 1
2017-10-23 16:06:53,883 | DEBUG    | ledger_manager.py    ( 312) | processLedgerStatus | Node4 found out from {'Node2', 'Node1', 'Node3'} that its ledger of type 2 is latest
2017-10-23 16:06:53,883 | DEBUG    | ledger_manager.py    ( 246) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None} from Node1
2017-10-23 16:06:53,884 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 1 of size 7 with 7
2017-10-23 16:06:53,884 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 1 of size 7 with 7
2017-10-23 16:06:53,886 | DEBUG    | ledger_manager.py    ( 312) | processLedgerStatus | Node4 found out from {'Node2', 'Node1', 'Node3'} that its ledger of type 1 is latest
2017-10-23 16:06:53,886 | DEBUG    | ledger_manager.py    ( 246) | processLedgerStatus | Node4 received ledger status: LEDGER_STATUS{'viewNo': None, 'ledgerId': 1, 'txnSeqNo': 7, 'merkleRoot': '6jJH537sAma75cnCHqpB3x5YXuFd3mWNa3cST4Syhr3o', 'ppSeqNo': None} from Node1
2017-10-23 16:06:53,886 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 1 of size 7 with 7
2017-10-23 16:06:53,887 | DEBUG    | ledger_manager.py    ( 984) | _compareLedger | Node4 comparing its ledger 1 of size 7 with 7
2017-10-23 16:06:53,887 | DEBUG    | ledger_manager.py    ( 312) | processLedgerStatus | Node4 found out from {'Node2', 'Node1', 'Node3'} that its ledger of type 1 is latest
2017-10-23 16:06:56,725 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 4
2017-10-23 16:07:03,595 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 1
2017-10-23 16:07:03,596 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:07:03,596 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:07:03,596 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:07:03,596 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:07:03,596 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:07:03,596 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:07:03,597 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 6 to run in 10 seconds
2017-10-23 16:07:08,723 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:07:08,723 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:07:08,723 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:07:08,724 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:07:13,603 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 6
2017-10-23 16:07:13,603 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:07:13,604 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:07:13,604 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:07:13,604 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:07:13,604 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:07:13,604 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:07:13,604 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 7 to run in 10 seconds
2017-10-23 16:07:23,614 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 7
2017-10-23 16:07:23,614 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:07:23,614 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:07:23,614 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:07:23,614 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:07:23,614 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:07:23,615 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:07:23,615 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 8 to run in 10 seconds
2017-10-23 16:07:23,725 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:07:23,725 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:07:23,725 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:07:23,726 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:07:33,619 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 8
2017-10-23 16:07:33,619 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:07:33,619 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:07:33,619 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:07:33,619 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:07:33,620 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:07:33,620 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:07:33,620 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 9 to run in 10 seconds
2017-10-23 16:07:38,728 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:07:38,729 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:07:38,729 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:07:38,729 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:07:43,625 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 9
2017-10-23 16:07:43,626 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:07:43,626 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:07:43,626 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:07:43,626 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:07:43,626 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:07:43,626 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:07:43,626 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 10 to run in 10 seconds
2017-10-23 16:07:52,213 | TRACE    | zstack.py            ( 479) | _receiveFromListener | Node4 got 1 messages through listener
2017-10-23 16:07:52,214 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node1: INSTANCE_CHANGE{'viewNo': 1, 'reason': 26}
2017-10-23 16:07:52,214 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'viewNo': 1, 'reason': 26, 'op': 'INSTANCE_CHANGE'}, 'Node1')
2017-10-23 16:07:52,214 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox INSTANCE_CHANGE{'viewNo': 1, 'reason': 26}
2017-10-23 16:07:52,214 | DEBUG    | node.py              (1915) | processInstanceChange | Node4 received instance change request: INSTANCE_CHANGE{'viewNo': 1, 'reason': 26} from Node1
2017-10-23 16:07:52,214 | DEBUG    | node.py              (1958) | do_view_change_if_possible | Node4 has no quorum for view 1
2017-10-23 16:07:52,215 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:07:52,215 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:07:52,215 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:07:52,215 | DEBUG    | node.py              (1946) | processInstanceChange | Node4 received instance change message INSTANCE_CHANGE{'viewNo': 1, 'reason': 26} but did not find the master to be slow or has already sent an instance change message
2017-10-23 16:07:53,600 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 2
2017-10-23 16:07:53,600 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:07:53,600 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:07:53,600 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 11 to run in 60 seconds
2017-10-23 16:07:53,601 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 1
2017-10-23 16:07:53,601 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:07:53,601 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 2 to run in 60 seconds
2017-10-23 16:07:53,636 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 10
2017-10-23 16:07:53,636 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:07:53,636 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:07:53,636 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:07:53,637 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:07:53,637 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:07:53,637 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:07:53,637 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 12 to run in 10 seconds
2017-10-23 16:07:53,721 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 5
2017-10-23 16:07:53,722 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 13 to run in 60 seconds
2017-10-23 16:07:53,722 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action propose_view_change with id 3
2017-10-23 16:07:53,722 | TRACE    | node.py              (2097) | propose_view_change | The primary is already connected so view change will not be proposed
2017-10-23 16:07:53,735 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:07:53,735 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:07:53,735 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:07:53,735 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:08:03,643 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 12
2017-10-23 16:08:03,643 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:08:03,643 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:08:03,644 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:08:03,644 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:08:03,644 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:08:03,644 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:08:03,644 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 14 to run in 10 seconds
2017-10-23 16:08:08,745 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:08:08,745 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:08:08,745 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:08:08,745 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:08:13,645 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 14
2017-10-23 16:08:13,645 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:08:13,646 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:08:13,646 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:08:13,646 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:08:13,646 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:08:13,647 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:08:13,647 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 15 to run in 10 seconds
2017-10-23 16:08:23,656 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 15
2017-10-23 16:08:23,656 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:08:23,656 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:08:23,656 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:08:23,656 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:08:23,656 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:08:23,656 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:08:23,656 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 16 to run in 10 seconds
2017-10-23 16:08:23,755 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:08:23,756 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:08:23,756 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:08:23,756 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:08:33,664 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 16
2017-10-23 16:08:33,664 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:08:33,664 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:08:33,664 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:08:33,664 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:08:33,665 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:08:33,665 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:08:33,665 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 17 to run in 10 seconds
2017-10-23 16:08:38,756 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:08:38,756 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:08:38,756 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:08:38,756 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:08:43,680 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 17
2017-10-23 16:08:43,680 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:08:43,680 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:08:43,681 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:08:43,681 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:08:43,681 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:08:43,681 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:08:43,681 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 18 to run in 10 seconds
2017-10-23 16:08:53,612 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 11
2017-10-23 16:08:53,613 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:08:53,613 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:08:53,613 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 19 to run in 60 seconds
2017-10-23 16:08:53,613 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 2
2017-10-23 16:08:53,613 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:08:53,613 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 3 to run in 60 seconds
2017-10-23 16:08:53,686 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 18
2017-10-23 16:08:53,686 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:08:53,686 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:08:53,686 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:08:53,686 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:08:53,686 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:08:53,686 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:08:53,686 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 20 to run in 10 seconds
2017-10-23 16:08:53,733 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 13
2017-10-23 16:08:53,734 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 21 to run in 60 seconds
2017-10-23 16:08:53,757 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:08:53,757 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:08:53,758 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:08:53,758 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:09:03,689 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 20
2017-10-23 16:09:03,689 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:09:03,689 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:09:03,689 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:09:03,689 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:09:03,689 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:09:03,689 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:09:03,689 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 22 to run in 10 seconds
2017-10-23 16:09:08,761 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:09:08,761 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:09:08,762 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:09:08,762 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:09:13,694 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 22
2017-10-23 16:09:13,694 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:09:13,694 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:09:13,694 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:09:13,694 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:09:13,694 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:09:13,694 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:09:13,695 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 23 to run in 10 seconds
2017-10-23 16:09:23,704 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 23
2017-10-23 16:09:23,705 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:09:23,705 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:09:23,705 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:09:23,705 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:09:23,705 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:09:23,705 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:09:23,705 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 24 to run in 10 seconds
2017-10-23 16:09:23,764 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:09:23,764 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:09:23,764 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:09:23,765 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:09:33,714 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 24
2017-10-23 16:09:33,714 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:09:33,714 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:09:33,715 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:09:33,715 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:09:33,715 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:09:33,715 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:09:33,715 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 25 to run in 10 seconds
2017-10-23 16:09:38,770 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:09:38,770 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:09:38,770 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:09:38,771 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:09:43,723 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 25
2017-10-23 16:09:43,723 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:09:43,723 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:09:43,724 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:09:43,724 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:09:43,724 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:09:43,724 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:09:43,724 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 26 to run in 10 seconds
2017-10-23 16:09:53,619 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 19
2017-10-23 16:09:53,619 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:09:53,619 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:09:53,620 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 27 to run in 60 seconds
2017-10-23 16:09:53,620 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 3
2017-10-23 16:09:53,620 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:09:53,620 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 4 to run in 60 seconds
2017-10-23 16:09:53,725 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 26
2017-10-23 16:09:53,725 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:09:53,725 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:09:53,725 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:09:53,725 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:09:53,725 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:09:53,726 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:09:53,726 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 28 to run in 10 seconds
2017-10-23 16:09:53,740 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 21
2017-10-23 16:09:53,740 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 29 to run in 60 seconds
2017-10-23 16:09:53,781 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:09:53,782 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:09:53,782 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:09:53,782 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:10:03,726 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 28
2017-10-23 16:10:03,726 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:10:03,727 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:10:03,727 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:10:03,727 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:10:03,727 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:10:03,727 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:10:03,727 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 30 to run in 10 seconds
2017-10-23 16:10:08,782 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:10:08,782 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:10:08,782 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:10:08,783 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:10:13,730 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 30
2017-10-23 16:10:13,730 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:10:13,730 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:10:13,730 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:10:13,730 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:10:13,730 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:10:13,730 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:10:13,730 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 31 to run in 10 seconds
2017-10-23 16:10:23,734 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 31
2017-10-23 16:10:23,734 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:10:23,734 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:10:23,734 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:10:23,734 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:10:23,734 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:10:23,734 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:10:23,735 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 32 to run in 10 seconds
2017-10-23 16:10:23,786 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:10:23,787 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:10:23,787 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:10:23,788 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:10:33,743 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 32
2017-10-23 16:10:33,743 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:10:33,743 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:10:33,743 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:10:33,743 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:10:33,743 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:10:33,743 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:10:33,744 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 33 to run in 10 seconds
2017-10-23 16:10:38,789 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:10:38,790 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:10:38,790 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:10:38,790 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:10:43,755 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 33
2017-10-23 16:10:43,756 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:10:43,756 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:10:43,756 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:10:43,756 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:10:43,757 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:10:43,757 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:10:43,757 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 34 to run in 10 seconds
2017-10-23 16:10:53,632 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 27
2017-10-23 16:10:53,632 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:10:53,632 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:10:53,632 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 35 to run in 60 seconds
2017-10-23 16:10:53,632 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 4
2017-10-23 16:10:53,632 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:10:53,633 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 5 to run in 60 seconds
2017-10-23 16:10:53,753 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 29
2017-10-23 16:10:53,754 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 36 to run in 60 seconds
2017-10-23 16:10:53,768 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 34
2017-10-23 16:10:53,768 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:10:53,768 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:10:53,768 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:10:53,768 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:10:53,768 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:10:53,769 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:10:53,769 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 37 to run in 10 seconds
2017-10-23 16:10:53,795 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:10:53,795 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:10:53,795 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:10:53,796 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:11:03,773 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 37
2017-10-23 16:11:03,774 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:11:03,774 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:11:03,774 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:11:03,774 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:11:03,774 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:11:03,774 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:11:03,774 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 38 to run in 10 seconds
2017-10-23 16:11:08,800 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:11:08,800 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:11:08,800 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:11:08,800 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:11:13,788 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 38
2017-10-23 16:11:13,788 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:11:13,788 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:11:13,788 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:11:13,789 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:11:13,789 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:11:13,789 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:11:13,789 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 39 to run in 10 seconds
2017-10-23 16:11:23,797 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 39
2017-10-23 16:11:23,797 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:11:23,797 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:11:23,798 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:11:23,798 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:11:23,798 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:11:23,798 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:11:23,798 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 40 to run in 10 seconds
2017-10-23 16:11:23,812 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:11:23,812 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:11:23,812 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:11:23,813 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:11:33,802 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 40
2017-10-23 16:11:33,802 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:11:33,803 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:11:33,803 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:11:33,803 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:11:33,803 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:11:33,803 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:11:33,803 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 41 to run in 10 seconds
2017-10-23 16:11:38,820 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:11:38,821 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:11:38,821 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:11:38,821 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:11:43,803 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 41
2017-10-23 16:11:43,804 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:11:43,804 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:11:43,804 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:11:43,804 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:11:43,804 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:11:43,804 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:11:43,804 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 42 to run in 10 seconds
2017-10-23 16:11:53,641 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 35
2017-10-23 16:11:53,641 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:11:53,641 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:11:53,641 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 43 to run in 60 seconds
2017-10-23 16:11:53,641 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 5
2017-10-23 16:11:53,641 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:11:53,641 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 6 to run in 60 seconds
2017-10-23 16:11:53,757 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 36
2017-10-23 16:11:53,758 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 44 to run in 60 seconds
2017-10-23 16:11:53,805 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 42
2017-10-23 16:11:53,806 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:11:53,806 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:11:53,806 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:11:53,806 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:11:53,806 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:11:53,806 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:11:53,806 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 45 to run in 10 seconds
2017-10-23 16:11:53,830 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:11:53,830 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:11:53,830 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:11:53,830 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:12:03,814 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 45
2017-10-23 16:12:03,814 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:12:03,814 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:12:03,814 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:12:03,814 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:12:03,814 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:12:03,815 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:12:03,815 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 46 to run in 10 seconds
2017-10-23 16:12:08,835 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:12:08,836 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:12:08,836 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:12:08,836 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:12:13,815 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 46
2017-10-23 16:12:13,815 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:12:13,815 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:12:13,817 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:12:13,818 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:12:13,818 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:12:13,818 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:12:13,818 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 47 to run in 10 seconds
2017-10-23 16:12:23,824 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 47
2017-10-23 16:12:23,825 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:12:23,825 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:12:23,825 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:12:23,826 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:12:23,826 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:12:23,826 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:12:23,826 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 48 to run in 10 seconds
2017-10-23 16:12:23,839 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:12:23,839 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:12:23,840 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:12:23,840 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:12:33,838 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 48
2017-10-23 16:12:33,838 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:12:33,838 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:12:33,838 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:12:33,838 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:12:33,838 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:12:33,839 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:12:33,839 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 49 to run in 10 seconds
2017-10-23 16:12:38,841 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:12:38,841 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:12:38,841 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:12:38,842 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:12:43,843 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 49
2017-10-23 16:12:43,844 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:12:43,844 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:12:43,844 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:12:43,844 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:12:43,844 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:12:43,844 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:12:43,844 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 50 to run in 10 seconds
2017-10-23 16:12:53,643 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 43
2017-10-23 16:12:53,643 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:12:53,644 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:12:53,644 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 51 to run in 60 seconds
2017-10-23 16:12:53,644 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 6
2017-10-23 16:12:53,644 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:12:53,644 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 7 to run in 60 seconds
2017-10-23 16:12:53,760 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 44
2017-10-23 16:12:53,760 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 52 to run in 60 seconds
2017-10-23 16:12:53,844 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:12:53,844 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:12:53,844 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:12:53,844 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:12:53,855 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 50
2017-10-23 16:12:53,856 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:12:53,856 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:12:53,856 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:12:53,856 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:12:53,856 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:12:53,856 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:12:53,856 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 53 to run in 10 seconds
2017-10-23 16:13:03,863 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 53
2017-10-23 16:13:03,863 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:13:03,863 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:13:03,863 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:13:03,864 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:13:03,864 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:13:03,864 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:13:03,864 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 54 to run in 10 seconds
2017-10-23 16:13:08,855 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:13:08,855 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:13:08,855 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:13:08,856 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:13:13,876 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 54
2017-10-23 16:13:13,876 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:13:13,876 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:13:13,876 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:13:13,876 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:13:13,877 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:13:13,877 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:13:13,877 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 55 to run in 10 seconds
2017-10-23 16:13:23,862 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:13:23,863 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:13:23,863 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:13:23,864 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:13:23,888 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 55
2017-10-23 16:13:23,889 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:13:23,889 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:13:23,889 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:13:23,889 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:13:23,890 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:13:23,890 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:13:23,890 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 56 to run in 10 seconds
2017-10-23 16:13:33,903 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 56
2017-10-23 16:13:33,903 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:13:33,904 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:13:33,904 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:13:33,904 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:13:33,904 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:13:33,904 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:13:33,905 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 57 to run in 10 seconds
2017-10-23 16:13:38,863 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:13:38,863 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:13:38,864 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:13:38,864 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:13:43,918 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 57
2017-10-23 16:13:43,918 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:13:43,918 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:13:43,918 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:13:43,919 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:13:43,919 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:13:43,919 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:13:43,922 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 58 to run in 10 seconds
2017-10-23 16:13:53,655 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 51
2017-10-23 16:13:53,655 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:13:53,655 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:13:53,655 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 59 to run in 60 seconds
2017-10-23 16:13:53,656 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 7
2017-10-23 16:13:53,656 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:13:53,656 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 8 to run in 60 seconds
2017-10-23 16:13:53,770 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 52
2017-10-23 16:13:53,772 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 60 to run in 60 seconds
2017-10-23 16:13:53,867 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:13:53,868 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:13:53,868 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:13:53,869 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:13:53,937 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 58
2017-10-23 16:13:53,937 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:13:53,937 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:13:53,937 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:13:53,938 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:13:53,938 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:13:53,938 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:13:53,938 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 61 to run in 10 seconds
2017-10-23 16:14:03,947 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 61
2017-10-23 16:14:03,947 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:14:03,947 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:14:03,948 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:14:03,948 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:14:03,948 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:14:03,948 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:14:03,948 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 62 to run in 10 seconds
2017-10-23 16:14:08,878 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:14:08,878 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:14:08,879 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:14:08,879 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:14:13,949 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 62
2017-10-23 16:14:13,949 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:14:13,949 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:14:13,950 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:14:13,950 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:14:13,950 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:14:13,950 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:14:13,950 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 63 to run in 10 seconds
2017-10-23 16:14:23,888 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:14:23,888 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:14:23,889 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:14:23,889 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:14:23,951 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 63
2017-10-23 16:14:23,951 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:14:23,951 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:14:23,951 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:14:23,951 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:14:23,951 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:14:23,951 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:14:23,952 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 64 to run in 10 seconds
2017-10-23 16:14:33,954 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 64
2017-10-23 16:14:33,955 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:14:33,955 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:14:33,957 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:14:33,957 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:14:33,957 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:14:33,957 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:14:33,957 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 65 to run in 10 seconds
2017-10-23 16:14:38,895 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:14:38,895 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:14:38,896 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:14:38,896 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:14:43,960 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 65
2017-10-23 16:14:43,960 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:14:43,961 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:14:43,961 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:14:43,961 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:14:43,961 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:14:43,961 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:14:43,962 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 66 to run in 10 seconds
2017-10-23 16:14:53,656 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 59
2017-10-23 16:14:53,657 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:14:53,657 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:14:53,657 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 67 to run in 60 seconds
2017-10-23 16:14:53,657 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 8
2017-10-23 16:14:53,657 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:14:53,658 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 9 to run in 60 seconds
2017-10-23 16:14:53,785 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 60
2017-10-23 16:14:53,786 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 68 to run in 60 seconds
2017-10-23 16:14:53,908 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:14:53,908 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:14:53,908 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:14:53,910 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:14:53,971 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 66
2017-10-23 16:14:53,971 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:14:53,972 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:14:53,972 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:14:53,972 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:14:53,972 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:14:53,972 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:14:53,972 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 69 to run in 10 seconds
2017-10-23 16:15:03,976 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 69
2017-10-23 16:15:03,977 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:15:03,977 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:15:03,977 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:15:03,977 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:15:03,977 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:15:03,978 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:15:03,978 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 70 to run in 10 seconds
2017-10-23 16:15:08,908 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:15:08,908 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:15:08,908 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:15:08,909 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:15:13,979 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 70
2017-10-23 16:15:13,980 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:15:13,980 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:15:13,980 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:15:13,980 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:15:13,980 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:15:13,980 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:15:13,980 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 71 to run in 10 seconds
2017-10-23 16:15:23,914 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:15:23,914 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:15:23,914 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:15:23,915 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:15:23,988 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 71
2017-10-23 16:15:23,989 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:15:23,989 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:15:23,989 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:15:23,989 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:15:23,989 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:15:23,989 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:15:23,990 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 72 to run in 10 seconds
2017-10-23 16:15:34,000 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 72
2017-10-23 16:15:34,001 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:15:34,001 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:15:34,001 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:15:34,001 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:15:34,001 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:15:34,001 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:15:34,001 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 73 to run in 10 seconds
2017-10-23 16:15:38,923 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:15:38,923 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:15:38,923 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:15:38,924 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:15:44,009 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 73
2017-10-23 16:15:44,009 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:15:44,009 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:15:44,009 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:15:44,009 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:15:44,010 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:15:44,010 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:15:44,010 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 74 to run in 10 seconds
2017-10-23 16:15:53,665 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 67
2017-10-23 16:15:53,665 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:15:53,665 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:15:53,665 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 75 to run in 60 seconds
2017-10-23 16:15:53,665 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 9
2017-10-23 16:15:53,665 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:15:53,665 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 10 to run in 60 seconds
2017-10-23 16:15:53,796 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 68
2017-10-23 16:15:53,797 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 76 to run in 60 seconds
2017-10-23 16:15:53,925 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:15:53,925 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:15:53,925 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:15:53,928 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:15:54,017 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 74
2017-10-23 16:15:54,017 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:15:54,017 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:15:54,017 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:15:54,018 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:15:54,018 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:15:54,018 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:15:54,018 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 77 to run in 10 seconds
2017-10-23 16:16:04,030 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 77
2017-10-23 16:16:04,030 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:16:04,030 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:16:04,030 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:16:04,030 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:16:04,031 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:16:04,031 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:16:04,031 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 78 to run in 10 seconds
2017-10-23 16:16:08,927 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:16:08,927 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:16:08,927 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:16:08,928 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:16:14,039 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 78
2017-10-23 16:16:14,039 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:16:14,040 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:16:14,040 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:16:14,040 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:16:14,040 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:16:14,040 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:16:14,040 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 79 to run in 10 seconds
2017-10-23 16:16:23,932 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:16:23,933 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:16:23,933 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:16:23,933 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:16:24,051 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 79
2017-10-23 16:16:24,051 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:16:24,051 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:16:24,051 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:16:24,051 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:16:24,051 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:16:24,052 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:16:24,052 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 80 to run in 10 seconds
2017-10-23 16:16:34,062 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 80
2017-10-23 16:16:34,062 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:16:34,062 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:16:34,062 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:16:34,062 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:16:34,062 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:16:34,062 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:16:34,062 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 81 to run in 10 seconds
2017-10-23 16:16:38,933 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:16:38,933 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:16:38,933 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:16:38,935 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:16:44,070 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 81
2017-10-23 16:16:44,070 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:16:44,072 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:16:44,072 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:16:44,073 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:16:44,073 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:16:44,073 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:16:44,073 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 82 to run in 10 seconds
2017-10-23 16:16:53,666 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 75
2017-10-23 16:16:53,666 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:16:53,666 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:16:53,666 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 83 to run in 60 seconds
2017-10-23 16:16:53,666 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 10
2017-10-23 16:16:53,666 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:16:53,667 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 11 to run in 60 seconds
2017-10-23 16:16:53,802 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 76
2017-10-23 16:16:53,803 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 84 to run in 60 seconds
2017-10-23 16:16:53,933 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:16:53,933 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:16:53,934 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:16:53,934 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:16:54,083 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 82
2017-10-23 16:16:54,083 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:16:54,084 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:16:54,084 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:16:54,084 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:16:54,084 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:16:54,085 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:16:54,085 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 85 to run in 10 seconds
2017-10-23 16:17:04,096 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 85
2017-10-23 16:17:04,097 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:17:04,097 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:17:04,097 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:17:04,097 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:17:04,097 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:17:04,097 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:17:04,098 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 86 to run in 10 seconds
2017-10-23 16:17:08,945 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:17:08,945 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:17:08,948 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:17:08,949 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:17:14,101 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 86
2017-10-23 16:17:14,101 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:17:14,101 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:17:14,101 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:17:14,101 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:17:14,102 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:17:14,102 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:17:14,102 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 87 to run in 10 seconds
2017-10-23 16:17:23,950 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:17:23,950 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:17:23,951 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:17:23,951 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:17:24,114 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 87
2017-10-23 16:17:24,114 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:17:24,114 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:17:24,114 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:17:24,115 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:17:24,115 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:17:24,115 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:17:24,115 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 88 to run in 10 seconds
2017-10-23 16:17:34,120 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 88
2017-10-23 16:17:34,120 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:17:34,121 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:17:34,121 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:17:34,121 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:17:34,121 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:17:34,121 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:17:34,121 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 89 to run in 10 seconds
2017-10-23 16:17:38,957 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:17:38,957 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:17:38,957 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:17:38,957 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:17:44,130 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 89
2017-10-23 16:17:44,130 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:17:44,131 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:17:44,131 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:17:44,131 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:17:44,131 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:17:44,132 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:17:44,132 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 90 to run in 10 seconds
2017-10-23 16:17:53,668 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 83
2017-10-23 16:17:53,668 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:17:53,668 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:17:53,668 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 91 to run in 60 seconds
2017-10-23 16:17:53,669 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 11
2017-10-23 16:17:53,669 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:17:53,669 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 12 to run in 60 seconds
2017-10-23 16:17:53,808 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 84
2017-10-23 16:17:53,809 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 92 to run in 60 seconds
2017-10-23 16:17:53,961 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:17:53,961 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:17:53,961 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:17:53,962 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:17:54,142 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 90
2017-10-23 16:17:54,142 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:17:54,142 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:17:54,142 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:17:54,142 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:17:54,142 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:17:54,142 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:17:54,142 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 93 to run in 10 seconds
2017-10-23 16:18:04,148 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 93
2017-10-23 16:18:04,149 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:18:04,149 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:18:04,149 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:18:04,149 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:18:04,149 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:18:04,149 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:18:04,150 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 94 to run in 10 seconds
2017-10-23 16:18:08,965 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:18:08,966 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:18:08,966 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:18:08,966 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:18:14,166 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 94
2017-10-23 16:18:14,166 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:18:14,166 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:18:14,166 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:18:14,166 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:18:14,167 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:18:14,167 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:18:14,167 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 95 to run in 10 seconds
2017-10-23 16:18:23,978 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:18:23,978 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:18:23,978 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:18:23,979 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:18:24,179 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 95
2017-10-23 16:18:24,180 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:18:24,180 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:18:24,180 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:18:24,180 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:18:24,180 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:18:24,180 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:18:24,180 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 96 to run in 10 seconds
2017-10-23 16:18:34,183 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 96
2017-10-23 16:18:34,184 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:18:34,184 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:18:34,184 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:18:34,184 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:18:34,184 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:18:34,184 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:18:34,184 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 97 to run in 10 seconds
2017-10-23 16:18:38,986 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:18:38,986 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:18:38,986 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:18:38,987 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:18:44,197 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 97
2017-10-23 16:18:44,197 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:18:44,197 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:18:44,197 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:18:44,197 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:18:44,197 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:18:44,197 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:18:44,197 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 98 to run in 10 seconds
2017-10-23 16:18:53,668 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 91
2017-10-23 16:18:53,669 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:18:53,669 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:18:53,669 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 99 to run in 60 seconds
2017-10-23 16:18:53,669 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 12
2017-10-23 16:18:53,670 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:18:53,670 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 13 to run in 60 seconds
2017-10-23 16:18:53,819 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 92
2017-10-23 16:18:53,820 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 100 to run in 60 seconds
2017-10-23 16:18:53,991 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:18:53,992 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:18:53,992 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:18:53,994 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:18:54,205 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 98
2017-10-23 16:18:54,206 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:18:54,206 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:18:54,206 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:18:54,206 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:18:54,207 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:18:54,207 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:18:54,207 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 101 to run in 10 seconds
2017-10-23 16:19:04,216 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 101
2017-10-23 16:19:04,216 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:19:04,216 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:19:04,216 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:19:04,216 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:19:04,217 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:19:04,217 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:19:04,217 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 102 to run in 10 seconds
2017-10-23 16:19:09,000 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:19:09,000 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:19:09,000 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:19:09,001 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:19:14,229 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 102
2017-10-23 16:19:14,229 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:19:14,229 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:19:14,229 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:19:14,229 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:19:14,230 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:19:14,230 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:19:14,230 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 103 to run in 10 seconds
2017-10-23 16:19:24,006 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:19:24,006 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:19:24,006 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:19:24,007 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:19:24,240 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 103
2017-10-23 16:19:24,241 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:19:24,241 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:19:24,241 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:19:24,241 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:19:24,241 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:19:24,241 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:19:24,241 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 104 to run in 10 seconds
2017-10-23 16:19:34,250 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 104
2017-10-23 16:19:34,250 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:19:34,250 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:19:34,251 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:19:34,251 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:19:34,251 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:19:34,251 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:19:34,251 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 105 to run in 10 seconds
2017-10-23 16:19:39,017 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:19:39,017 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:19:39,017 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:19:39,018 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:19:44,253 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 105
2017-10-23 16:19:44,253 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:19:44,253 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:19:44,254 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:19:44,254 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:19:44,254 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:19:44,254 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:19:44,255 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 106 to run in 10 seconds
2017-10-23 16:19:53,678 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 99
2017-10-23 16:19:53,678 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:19:53,678 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:19:53,678 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 107 to run in 60 seconds
2017-10-23 16:19:53,678 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 13
2017-10-23 16:19:53,679 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:19:53,679 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 14 to run in 60 seconds
2017-10-23 16:19:53,833 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 100
2017-10-23 16:19:53,834 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 108 to run in 60 seconds
2017-10-23 16:19:54,021 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:19:54,022 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:19:54,022 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:19:54,022 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:19:54,266 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 106
2017-10-23 16:19:54,266 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:19:54,266 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:19:54,266 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:19:54,266 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:19:54,266 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:19:54,266 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:19:54,266 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 109 to run in 10 seconds
2017-10-23 16:20:04,272 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 109
2017-10-23 16:20:04,272 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:20:04,273 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:20:04,273 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:20:04,273 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:20:04,273 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:20:04,273 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:20:04,273 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 110 to run in 10 seconds
2017-10-23 16:20:09,022 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:20:09,023 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:20:09,023 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:20:09,023 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:20:14,280 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 110
2017-10-23 16:20:14,280 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:20:14,280 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:20:14,281 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:20:14,281 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:20:14,281 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:20:14,281 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:20:14,281 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 111 to run in 10 seconds
2017-10-23 16:20:24,026 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:20:24,026 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:20:24,026 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:20:24,027 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:20:24,294 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 111
2017-10-23 16:20:24,294 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:20:24,294 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:20:24,294 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:20:24,294 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:20:24,294 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:20:24,294 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:20:24,295 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 112 to run in 10 seconds
2017-10-23 16:20:34,300 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 112
2017-10-23 16:20:34,300 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:20:34,300 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:20:34,300 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:20:34,300 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:20:34,300 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:20:34,300 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:20:34,301 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 113 to run in 10 seconds
2017-10-23 16:20:39,032 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:20:39,032 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:20:39,032 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:20:39,032 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:20:44,301 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 113
2017-10-23 16:20:44,301 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:20:44,301 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:20:44,302 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:20:44,302 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:20:44,302 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:20:44,302 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:20:44,302 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 114 to run in 10 seconds
2017-10-23 16:20:53,681 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 107
2017-10-23 16:20:53,681 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:20:53,681 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:20:53,681 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 115 to run in 60 seconds
2017-10-23 16:20:53,681 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 14
2017-10-23 16:20:53,681 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:20:53,681 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 15 to run in 60 seconds
2017-10-23 16:20:53,835 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 108
2017-10-23 16:20:53,836 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 116 to run in 60 seconds
2017-10-23 16:20:54,034 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:20:54,034 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:20:54,034 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:20:54,035 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:20:54,309 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 114
2017-10-23 16:20:54,309 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:20:54,309 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:20:54,309 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:20:54,309 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:20:54,310 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:20:54,310 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:20:54,310 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 117 to run in 10 seconds
2017-10-23 16:21:04,310 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 117
2017-10-23 16:21:04,310 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:21:04,310 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:21:04,310 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:21:04,311 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:21:04,311 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:21:04,311 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:21:04,311 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 118 to run in 10 seconds
2017-10-23 16:21:09,043 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:21:09,043 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:21:09,043 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:21:09,044 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:21:14,316 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 118
2017-10-23 16:21:14,316 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:21:14,316 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike
2017-10-23 16:21:14,316 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:21:14,316 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:21:14,317 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:21:14,317 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:21:14,317 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 119 to run in 10 seconds
2017-10-23 16:21:24,047 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:21:24,048 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:21:24,048 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:21:24,049 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:21:24,319 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 119
2017-10-23 16:21:24,320 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:21:24,320 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:21:24,320 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:21:24,320 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:21:24,320 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:21:24,321 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:21:24,321 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 120 to run in 10 seconds
2017-10-23 16:21:34,322 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 120
2017-10-23 16:21:34,322 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:21:34,322 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:21:34,323 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:21:34,323 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:21:34,323 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:21:34,323 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:21:34,323 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 121 to run in 10 seconds
2017-10-23 16:21:39,057 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:21:39,057 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:21:39,058 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:21:39,058 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:21:44,331 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 121
2017-10-23 16:21:44,331 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:21:44,331 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:21:44,331 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:21:44,332 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:21:44,332 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:21:44,332 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:21:44,332 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 122 to run in 10 seconds
2017-10-23 16:21:53,691 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 115
2017-10-23 16:21:53,691 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:21:53,691 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:21:53,691 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 123 to run in 60 seconds
2017-10-23 16:21:53,691 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 15
2017-10-23 16:21:53,692 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:21:53,692 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 16 to run in 60 seconds
2017-10-23 16:21:53,839 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 116
2017-10-23 16:21:53,840 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 124 to run in 60 seconds
2017-10-23 16:21:54,064 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:21:54,064 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:21:54,064 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:21:54,065 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:21:54,337 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 122
2017-10-23 16:21:54,337 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:21:54,337 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:21:54,337 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:21:54,337 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:21:54,338 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:21:54,338 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:21:54,338 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 125 to run in 10 seconds
2017-10-23 16:22:04,343 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 125
2017-10-23 16:22:04,343 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:22:04,343 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:22:04,343 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:22:04,343 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:22:04,344 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:22:04,344 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:22:04,344 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 126 to run in 10 seconds
2017-10-23 16:22:09,073 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:22:09,074 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:22:09,074 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:22:09,075 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:22:14,349 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 126
2017-10-23 16:22:14,349 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:22:14,350 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:22:14,350 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:22:14,350 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:22:14,350 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:22:14,350 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:22:14,350 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 127 to run in 10 seconds
2017-10-23 16:22:24,077 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:22:24,077 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:22:24,077 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:22:24,078 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:22:24,354 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 127
2017-10-23 16:22:24,354 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:22:24,354 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:22:24,354 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:22:24,354 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:22:24,354 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:22:24,355 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:22:24,355 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 128 to run in 10 seconds
2017-10-23 16:22:34,370 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 128
2017-10-23 16:22:34,370 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:22:34,370 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:22:34,371 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:22:34,371 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:22:34,372 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:22:34,372 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:22:34,372 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 129 to run in 10 seconds
2017-10-23 16:22:39,077 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:22:39,078 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:22:39,078 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:22:39,078 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:22:44,384 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 129
2017-10-23 16:22:44,384 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:22:44,385 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:22:44,385 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:22:44,385 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:22:44,385 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:22:44,385 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:22:44,385 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 130 to run in 10 seconds
2017-10-23 16:22:53,695 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 123
2017-10-23 16:22:53,695 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:22:53,695 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:22:53,695 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 131 to run in 60 seconds
2017-10-23 16:22:53,696 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 16
2017-10-23 16:22:53,696 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:22:53,696 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 17 to run in 60 seconds
2017-10-23 16:22:53,841 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 124
2017-10-23 16:22:53,841 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 132 to run in 60 seconds
2017-10-23 16:22:54,084 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:22:54,084 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:22:54,084 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:22:54,084 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:22:54,395 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 130
2017-10-23 16:22:54,395 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:22:54,395 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:22:54,395 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:22:54,395 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:22:54,395 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:22:54,396 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:22:54,396 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 133 to run in 10 seconds
2017-10-23 16:23:04,404 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 133
2017-10-23 16:23:04,404 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:23:04,404 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:23:04,404 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:23:04,404 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:23:04,405 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:23:04,405 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:23:04,405 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 134 to run in 10 seconds
2017-10-23 16:23:09,093 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:23:09,093 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:23:09,093 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:23:09,094 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:23:14,412 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 134
2017-10-23 16:23:14,412 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:23:14,412 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:23:14,412 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:23:14,413 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:23:14,413 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:23:14,413 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:23:14,413 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 135 to run in 10 seconds
2017-10-23 16:23:24,098 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:23:24,098 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:23:24,098 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:23:24,098 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:23:24,419 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 135
2017-10-23 16:23:24,419 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:23:24,420 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:23:24,420 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:23:24,420 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:23:24,420 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:23:24,420 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:23:24,420 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 136 to run in 10 seconds
2017-10-23 16:23:34,423 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 136
2017-10-23 16:23:34,423 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:23:34,423 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:23:34,423 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:23:34,423 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:23:34,423 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:23:34,424 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:23:34,424 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 137 to run in 10 seconds
2017-10-23 16:23:39,105 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:23:39,106 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:23:39,106 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:23:39,106 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:23:44,431 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 137
2017-10-23 16:23:44,431 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:23:44,431 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:23:44,432 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:23:44,432 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:23:44,432 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:23:44,432 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:23:44,432 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 138 to run in 10 seconds
2017-10-23 16:23:53,703 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 131
2017-10-23 16:23:53,704 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:23:53,704 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:23:53,704 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 139 to run in 60 seconds
2017-10-23 16:23:53,704 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 17
2017-10-23 16:23:53,705 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:23:53,705 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 18 to run in 60 seconds
2017-10-23 16:23:53,855 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 132
2017-10-23 16:23:53,856 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 140 to run in 60 seconds
2017-10-23 16:23:54,115 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:23:54,115 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:23:54,116 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:23:54,116 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:23:54,438 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 138
2017-10-23 16:23:54,438 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:23:54,438 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:23:54,438 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:23:54,439 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:23:54,439 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:23:54,439 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:23:54,439 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 141 to run in 10 seconds
2017-10-23 16:24:04,442 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 141
2017-10-23 16:24:04,442 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:24:04,442 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:24:04,442 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:24:04,442 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:24:04,442 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:24:04,442 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:24:04,443 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 142 to run in 10 seconds
2017-10-23 16:24:09,119 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:24:09,120 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:24:09,120 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:24:09,121 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:24:14,451 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 142
2017-10-23 16:24:14,451 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:24:14,452 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:24:14,452 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:24:14,452 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:24:14,452 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:24:14,452 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:24:14,452 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 143 to run in 10 seconds
2017-10-23 16:24:24,132 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:24:24,132 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:24:24,132 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:24:24,134 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:24:24,453 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 143
2017-10-23 16:24:24,454 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:24:24,454 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:24:24,454 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:24:24,454 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:24:24,454 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:24:24,454 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:24:24,455 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 144 to run in 10 seconds
2017-10-23 16:24:34,466 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 144
2017-10-23 16:24:34,466 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:24:34,466 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:24:34,466 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:24:34,466 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:24:34,466 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:24:34,466 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:24:34,466 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 145 to run in 10 seconds
2017-10-23 16:24:39,140 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:24:39,140 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:24:39,140 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:24:39,140 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:24:44,475 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 145
2017-10-23 16:24:44,475 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:24:44,475 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:24:44,475 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:24:44,475 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:24:44,475 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:24:44,476 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:24:44,476 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 146 to run in 10 seconds
2017-10-23 16:24:53,707 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 139
2017-10-23 16:24:53,707 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:24:53,708 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:24:53,708 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 147 to run in 60 seconds
2017-10-23 16:24:53,708 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 18
2017-10-23 16:24:53,708 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:24:53,708 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 19 to run in 60 seconds
2017-10-23 16:24:53,861 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 140
2017-10-23 16:24:53,863 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 148 to run in 60 seconds
2017-10-23 16:24:54,145 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:24:54,145 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:24:54,146 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:24:54,152 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:24:54,482 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 146
2017-10-23 16:24:54,482 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:24:54,484 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:24:54,484 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:24:54,484 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:24:54,485 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:24:54,485 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:24:54,485 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 149 to run in 10 seconds
2017-10-23 16:25:04,494 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 149
2017-10-23 16:25:04,494 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:25:04,494 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:25:04,495 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:25:04,495 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:25:04,495 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:25:04,495 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:25:04,495 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 150 to run in 10 seconds
2017-10-23 16:25:09,146 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:25:09,147 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:25:09,147 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:25:09,148 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:25:14,501 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 150
2017-10-23 16:25:14,501 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:25:14,501 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:25:14,501 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:25:14,501 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:25:14,501 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:25:14,501 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:25:14,501 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 151 to run in 10 seconds
2017-10-23 16:25:24,151 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:25:24,151 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:25:24,152 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:25:24,152 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:25:24,510 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 151
2017-10-23 16:25:24,511 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:25:24,511 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:25:24,511 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:25:24,511 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:25:24,511 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:25:24,511 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:25:24,511 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 152 to run in 10 seconds
2017-10-23 16:25:34,514 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 152
2017-10-23 16:25:34,514 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:25:34,514 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:25:34,515 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:25:34,515 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:25:34,515 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:25:34,515 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:25:34,515 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 153 to run in 10 seconds
2017-10-23 16:25:39,158 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:25:39,159 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:25:39,159 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:25:39,159 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:25:44,517 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 153
2017-10-23 16:25:44,517 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:25:44,518 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:25:44,518 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:25:44,518 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:25:44,518 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:25:44,518 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:25:44,518 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 154 to run in 10 seconds
2017-10-23 16:25:53,713 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 147
2017-10-23 16:25:53,713 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:25:53,713 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:25:53,713 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 155 to run in 60 seconds
2017-10-23 16:25:53,713 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 19
2017-10-23 16:25:53,714 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:25:53,714 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 20 to run in 60 seconds
2017-10-23 16:25:53,864 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 148
2017-10-23 16:25:53,865 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 156 to run in 60 seconds
2017-10-23 16:25:54,164 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:25:54,164 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:25:54,164 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:25:54,165 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:25:54,528 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 154
2017-10-23 16:25:54,528 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:25:54,528 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:25:54,528 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:25:54,529 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:25:54,529 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:25:54,529 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:25:54,529 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 157 to run in 10 seconds
2017-10-23 16:26:04,533 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 157
2017-10-23 16:26:04,533 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:26:04,533 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:26:04,533 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:26:04,533 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:26:04,533 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:26:04,534 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:26:04,534 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 158 to run in 10 seconds
2017-10-23 16:26:09,172 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:26:09,172 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:26:09,172 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:26:09,172 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:26:14,537 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 158
2017-10-23 16:26:14,538 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:26:14,538 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:26:14,538 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:26:14,538 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:26:14,538 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:26:14,538 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:26:14,539 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 159 to run in 10 seconds
2017-10-23 16:26:24,180 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:26:24,181 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:26:24,181 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:26:24,181 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:26:24,550 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 159
2017-10-23 16:26:24,550 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:26:24,551 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:26:24,551 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:26:24,551 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:26:24,551 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:26:24,552 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:26:24,552 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 160 to run in 10 seconds
2017-10-23 16:26:34,566 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 160
2017-10-23 16:26:34,566 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:26:34,566 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:26:34,566 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:26:34,566 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:26:34,567 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:26:34,567 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:26:34,567 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 161 to run in 10 seconds
2017-10-23 16:26:39,185 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:26:39,186 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:26:39,186 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:26:39,186 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:26:44,569 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 161
2017-10-23 16:26:44,569 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:26:44,569 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:26:44,569 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:26:44,569 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:26:44,569 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:26:44,569 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:26:44,570 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 162 to run in 10 seconds
2017-10-23 16:26:53,724 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 155
2017-10-23 16:26:53,724 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:26:53,724 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:26:53,724 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 163 to run in 60 seconds
2017-10-23 16:26:53,725 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 20
2017-10-23 16:26:53,725 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:26:53,725 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 21 to run in 60 seconds
2017-10-23 16:26:53,866 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 156
2017-10-23 16:26:53,867 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 164 to run in 60 seconds
2017-10-23 16:26:54,188 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:26:54,188 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:26:54,188 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:26:54,188 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:26:54,580 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 162
2017-10-23 16:26:54,580 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:26:54,580 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:26:54,580 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:26:54,581 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:26:54,581 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:26:54,581 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:26:54,581 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 165 to run in 10 seconds
2017-10-23 16:27:04,591 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 165
2017-10-23 16:27:04,591 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:27:04,591 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:27:04,591 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:27:04,591 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:27:04,591 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:27:04,591 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:27:04,592 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 166 to run in 10 seconds
2017-10-23 16:27:09,199 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:27:09,199 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:27:09,199 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:27:09,200 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:27:14,592 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 166
2017-10-23 16:27:14,592 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:27:14,592 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:27:14,592 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:27:14,592 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:27:14,592 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:27:14,593 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:27:14,593 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 167 to run in 10 seconds
2017-10-23 16:27:24,201 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:27:24,201 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:27:24,201 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:27:24,202 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:27:24,605 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 167
2017-10-23 16:27:24,605 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:27:24,605 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:27:24,606 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:27:24,606 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:27:24,606 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:27:24,606 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:27:24,606 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 168 to run in 10 seconds
2017-10-23 16:27:34,611 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 168
2017-10-23 16:27:34,611 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:27:34,611 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:27:34,611 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:27:34,611 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:27:34,611 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:27:34,611 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:27:34,612 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 169 to run in 10 seconds
2017-10-23 16:27:39,210 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:27:39,211 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:27:39,211 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:27:39,211 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:27:44,622 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 169
2017-10-23 16:27:44,622 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:27:44,622 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:27:44,622 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:27:44,623 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:27:44,623 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:27:44,623 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:27:44,623 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 170 to run in 10 seconds
2017-10-23 16:27:53,728 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 163
2017-10-23 16:27:53,729 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:27:53,729 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:27:53,729 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 171 to run in 60 seconds
2017-10-23 16:27:53,729 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 21
2017-10-23 16:27:53,729 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:27:53,730 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 22 to run in 60 seconds
2017-10-23 16:27:53,874 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 164
2017-10-23 16:27:53,875 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 172 to run in 60 seconds
2017-10-23 16:27:54,214 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:27:54,214 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:27:54,214 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:27:54,214 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:27:54,633 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 170
2017-10-23 16:27:54,634 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:27:54,635 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:27:54,635 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:27:54,635 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:27:54,635 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:27:54,635 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:27:54,635 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 173 to run in 10 seconds
2017-10-23 16:28:04,637 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 173
2017-10-23 16:28:04,637 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:28:04,637 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:28:04,637 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:28:04,637 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:28:04,637 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:28:04,637 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:28:04,637 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 174 to run in 10 seconds
2017-10-23 16:28:09,225 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:28:09,225 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:28:09,225 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:28:09,225 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:28:14,640 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 174
2017-10-23 16:28:14,640 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:28:14,640 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:28:14,640 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:28:14,641 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:28:14,641 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:28:14,641 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:28:14,641 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 175 to run in 10 seconds
2017-10-23 16:28:24,231 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:28:24,232 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:28:24,232 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:28:24,232 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:28:24,653 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 175
2017-10-23 16:28:24,653 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:28:24,654 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:28:24,654 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:28:24,654 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:28:24,654 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:28:24,654 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:28:24,655 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 176 to run in 10 seconds
2017-10-23 16:28:34,662 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 176
2017-10-23 16:28:34,662 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:28:34,662 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:28:34,662 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:28:34,662 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:28:34,663 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:28:34,663 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:28:34,663 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 177 to run in 10 seconds
2017-10-23 16:28:39,233 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:28:39,233 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:28:39,233 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:28:39,234 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:28:44,671 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 177
2017-10-23 16:28:44,672 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:28:44,672 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:28:44,672 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:28:44,673 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:28:44,673 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:28:44,673 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:28:44,673 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 178 to run in 10 seconds
2017-10-23 16:28:53,731 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 171
2017-10-23 16:28:53,732 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:28:53,735 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:28:53,735 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 179 to run in 60 seconds
2017-10-23 16:28:53,735 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 22
2017-10-23 16:28:53,736 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:28:53,736 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 23 to run in 60 seconds
2017-10-23 16:28:53,875 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 172
2017-10-23 16:28:53,876 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 180 to run in 60 seconds
2017-10-23 16:28:54,235 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:28:54,236 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:28:54,236 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:28:54,236 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:28:54,680 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 178
2017-10-23 16:28:54,682 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:28:54,682 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:28:54,682 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:28:54,682 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:28:54,682 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:28:54,683 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:28:54,683 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 181 to run in 10 seconds
2017-10-23 16:29:04,689 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 181
2017-10-23 16:29:04,690 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:29:04,692 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:29:04,692 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:29:04,692 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:29:04,693 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:29:04,693 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:29:04,693 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 182 to run in 10 seconds
2017-10-23 16:29:09,239 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:29:09,239 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:29:09,239 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:29:09,240 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:29:14,702 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 182
2017-10-23 16:29:14,702 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:29:14,703 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:29:14,703 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:29:14,703 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:29:14,703 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:29:14,703 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:29:14,703 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 183 to run in 10 seconds
2017-10-23 16:29:24,239 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:29:24,240 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:29:24,240 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:29:24,241 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:29:24,705 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 183
2017-10-23 16:29:24,705 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:29:24,705 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:29:24,706 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:29:24,706 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:29:24,706 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:29:24,706 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:29:24,706 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 184 to run in 10 seconds
2017-10-23 16:29:34,718 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 184
2017-10-23 16:29:34,718 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:29:34,718 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:29:34,718 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:29:34,718 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:29:34,718 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:29:34,718 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:29:34,718 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 185 to run in 10 seconds
2017-10-23 16:29:39,246 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:29:39,247 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:29:39,247 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:29:39,248 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:29:44,729 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 185
2017-10-23 16:29:44,730 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:29:44,730 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:29:44,730 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:29:44,730 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:29:44,730 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:29:44,730 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:29:44,730 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 186 to run in 10 seconds
2017-10-23 16:29:53,735 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 179
2017-10-23 16:29:53,736 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:29:53,736 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:29:53,736 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 187 to run in 60 seconds
2017-10-23 16:29:53,736 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 23
2017-10-23 16:29:53,736 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:29:53,737 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 24 to run in 60 seconds
2017-10-23 16:29:53,880 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 180
2017-10-23 16:29:53,881 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 188 to run in 60 seconds
2017-10-23 16:29:54,257 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:29:54,258 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:29:54,258 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:29:54,258 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:29:54,742 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 186
2017-10-23 16:29:54,742 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:29:54,743 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:29:54,743 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:29:54,743 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:29:54,743 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:29:54,743 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:29:54,744 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 189 to run in 10 seconds
2017-10-23 16:30:04,753 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 189
2017-10-23 16:30:04,754 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:30:04,754 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:30:04,754 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:30:04,754 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:30:04,755 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:30:04,755 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:30:04,755 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 190 to run in 10 seconds
2017-10-23 16:30:09,263 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:30:09,263 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:30:09,264 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:30:09,264 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:30:14,758 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 190
2017-10-23 16:30:14,759 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:30:14,759 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:30:14,759 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:30:14,759 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:30:14,760 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:30:14,760 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:30:14,760 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 191 to run in 10 seconds
2017-10-23 16:30:24,276 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:30:24,276 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:30:24,277 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:30:24,278 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:30:24,772 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 191
2017-10-23 16:30:24,772 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:30:24,772 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:30:24,772 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:30:24,773 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:30:24,773 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:30:24,773 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:30:24,773 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 192 to run in 10 seconds
2017-10-23 16:30:34,773 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 192
2017-10-23 16:30:34,773 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:30:34,774 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:30:34,774 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:30:34,774 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:30:34,774 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:30:34,774 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:30:34,774 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 193 to run in 10 seconds
2017-10-23 16:30:39,287 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:30:39,288 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:30:39,288 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:30:39,288 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:30:44,787 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 193
2017-10-23 16:30:44,787 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:30:44,787 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:30:44,787 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:30:44,787 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:30:44,788 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:30:44,788 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:30:44,788 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 194 to run in 10 seconds
2017-10-23 16:30:53,743 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 187
2017-10-23 16:30:53,743 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:30:53,743 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:30:53,744 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 195 to run in 60 seconds
2017-10-23 16:30:53,744 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 24
2017-10-23 16:30:53,744 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:30:53,744 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 25 to run in 60 seconds
2017-10-23 16:30:53,890 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 188
2017-10-23 16:30:53,891 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 196 to run in 60 seconds
2017-10-23 16:30:54,297 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:30:54,297 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:30:54,297 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:30:54,297 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:30:54,794 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 194
2017-10-23 16:30:54,794 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:30:54,796 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:30:54,796 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:30:54,796 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:30:54,796 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:30:54,796 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:30:54,796 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 197 to run in 10 seconds
2017-10-23 16:31:04,797 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 197
2017-10-23 16:31:04,797 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:31:04,797 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:31:04,797 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:31:04,797 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:31:04,797 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:31:04,798 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:31:04,798 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 198 to run in 10 seconds
2017-10-23 16:31:09,306 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:31:09,307 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:31:09,307 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:31:09,308 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:31:14,800 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 198
2017-10-23 16:31:14,801 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:31:14,801 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:31:14,801 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:31:14,802 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:31:14,802 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:31:14,802 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:31:14,802 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 199 to run in 10 seconds
2017-10-23 16:31:24,310 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:31:24,310 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:31:24,310 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:31:24,310 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:31:24,810 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 199
2017-10-23 16:31:24,810 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:31:24,810 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:31:24,810 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:31:24,810 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:31:24,810 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:31:24,810 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:31:24,810 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 200 to run in 10 seconds
2017-10-23 16:31:34,822 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 200
2017-10-23 16:31:34,822 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:31:34,822 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:31:34,822 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:31:34,822 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:31:34,823 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:31:34,823 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:31:34,823 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 201 to run in 10 seconds
2017-10-23 16:31:39,316 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:31:39,316 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:31:39,316 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:31:39,317 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:31:44,833 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 201
2017-10-23 16:31:44,833 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:31:44,833 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:31:44,833 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:31:44,834 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:31:44,834 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:31:44,834 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:31:44,834 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 202 to run in 10 seconds
2017-10-23 16:31:53,744 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 195
2017-10-23 16:31:53,744 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:31:53,745 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:31:53,745 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 203 to run in 60 seconds
2017-10-23 16:31:53,745 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 25
2017-10-23 16:31:53,745 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:31:53,745 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 26 to run in 60 seconds
2017-10-23 16:31:53,897 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 196
2017-10-23 16:31:53,898 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 204 to run in 60 seconds
2017-10-23 16:31:54,317 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:31:54,317 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:31:54,317 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:31:54,317 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:31:54,842 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 202
2017-10-23 16:31:54,842 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:31:54,842 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:31:54,842 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:31:54,843 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:31:54,843 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:31:54,843 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:31:54,843 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 205 to run in 10 seconds
2017-10-23 16:32:04,844 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 205
2017-10-23 16:32:04,844 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:32:04,844 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:32:04,844 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:32:04,844 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:32:04,845 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:32:04,845 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:32:04,845 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 206 to run in 10 seconds
2017-10-23 16:32:09,321 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:32:09,321 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:32:09,321 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:32:09,322 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:32:14,849 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 206
2017-10-23 16:32:14,849 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:32:14,849 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:32:14,849 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:32:14,849 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:32:14,849 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:32:14,849 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:32:14,849 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 207 to run in 10 seconds
2017-10-23 16:32:24,322 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:32:24,322 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:32:24,322 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:32:24,323 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:32:24,857 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 207
2017-10-23 16:32:24,857 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:32:24,857 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:32:24,857 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:32:24,857 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:32:24,858 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:32:24,858 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:32:24,858 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 208 to run in 10 seconds
2017-10-23 16:32:34,869 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 208
2017-10-23 16:32:34,869 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:32:34,869 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:32:34,870 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:32:34,870 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:32:34,870 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:32:34,870 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:32:34,870 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 209 to run in 10 seconds
2017-10-23 16:32:39,332 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:32:39,332 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:32:39,332 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:32:39,333 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:32:44,879 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 209
2017-10-23 16:32:44,879 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:32:44,880 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:32:44,880 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:32:44,880 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:32:44,880 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:32:44,880 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:32:44,880 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 210 to run in 10 seconds
2017-10-23 16:32:53,747 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 203
2017-10-23 16:32:53,747 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:32:53,747 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:32:53,748 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 211 to run in 60 seconds
2017-10-23 16:32:53,748 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 26
2017-10-23 16:32:53,748 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:32:53,748 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 27 to run in 60 seconds
2017-10-23 16:32:53,902 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 204
2017-10-23 16:32:53,904 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 212 to run in 60 seconds
2017-10-23 16:32:54,338 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:32:54,339 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:32:54,339 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:32:54,340 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:32:54,890 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 210
2017-10-23 16:32:54,890 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:32:54,890 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:32:54,891 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:32:54,891 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:32:54,891 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:32:54,891 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:32:54,891 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 213 to run in 10 seconds
2017-10-23 16:33:04,898 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 213
2017-10-23 16:33:04,898 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:33:04,899 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:33:04,899 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:33:04,899 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:33:04,899 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:33:04,899 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:33:04,899 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 214 to run in 10 seconds
2017-10-23 16:33:09,342 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:33:09,342 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:33:09,342 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:33:09,343 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:33:14,915 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 214
2017-10-23 16:33:14,915 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:33:14,916 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:33:14,916 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:33:14,916 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:33:14,916 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:33:14,916 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:33:14,916 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 215 to run in 10 seconds
2017-10-23 16:33:24,350 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:33:24,350 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:33:24,350 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:33:24,351 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:33:24,928 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 215
2017-10-23 16:33:24,928 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:33:24,928 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:33:24,928 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:33:24,929 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:33:24,929 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:33:24,929 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:33:24,929 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 216 to run in 10 seconds
2017-10-23 16:33:34,939 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 216
2017-10-23 16:33:34,939 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:33:34,940 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:33:34,940 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:33:34,940 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:33:34,940 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:33:34,940 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:33:34,940 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 217 to run in 10 seconds
2017-10-23 16:33:39,357 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:33:39,357 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:33:39,357 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:33:39,358 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:33:44,949 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 217
2017-10-23 16:33:44,949 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:33:44,949 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:33:44,949 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:33:44,950 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:33:44,950 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:33:44,950 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:33:44,950 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 218 to run in 10 seconds
2017-10-23 16:33:53,749 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 211
2017-10-23 16:33:53,749 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:33:53,750 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:33:53,750 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 219 to run in 60 seconds
2017-10-23 16:33:53,750 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 27
2017-10-23 16:33:53,750 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:33:53,750 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 28 to run in 60 seconds
2017-10-23 16:33:53,919 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 212
2017-10-23 16:33:53,920 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 220 to run in 60 seconds
2017-10-23 16:33:54,364 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:33:54,365 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:33:54,365 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:33:54,365 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:33:54,954 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 218
2017-10-23 16:33:54,954 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:33:54,955 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:33:54,955 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:33:54,955 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:33:54,955 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:33:54,957 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:33:54,957 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 221 to run in 10 seconds
2017-10-23 16:34:04,967 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 221
2017-10-23 16:34:04,968 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:34:04,968 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:34:04,968 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:34:04,968 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:34:04,968 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:34:04,968 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:34:04,969 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 222 to run in 10 seconds
2017-10-23 16:34:09,369 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:34:09,369 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:34:09,369 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:34:09,370 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:34:14,976 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 222
2017-10-23 16:34:14,976 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:34:14,976 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:34:14,977 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:34:14,977 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:34:14,977 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:34:14,977 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:34:14,977 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 223 to run in 10 seconds
2017-10-23 16:34:24,370 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:34:24,370 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:34:24,370 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:34:24,371 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:34:24,977 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 223
2017-10-23 16:34:24,978 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:34:24,978 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:34:24,978 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:34:24,978 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:34:24,978 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:34:24,979 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:34:24,979 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 224 to run in 10 seconds
2017-10-23 16:34:34,986 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 224
2017-10-23 16:34:34,986 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:34:34,986 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:34:34,986 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:34:34,986 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:34:34,987 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:34:34,987 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:34:34,987 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 225 to run in 10 seconds
2017-10-23 16:34:39,375 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:34:39,375 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:34:39,375 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:34:39,377 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:34:44,994 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 225
2017-10-23 16:34:44,994 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:34:44,994 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:34:44,994 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:34:44,995 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:34:44,995 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:34:44,995 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:34:44,995 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 226 to run in 10 seconds
2017-10-23 16:34:53,756 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 219
2017-10-23 16:34:53,756 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:34:53,757 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:34:53,757 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 227 to run in 60 seconds
2017-10-23 16:34:53,757 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 28
2017-10-23 16:34:53,757 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:34:53,757 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 29 to run in 60 seconds
2017-10-23 16:34:53,923 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 220
2017-10-23 16:34:53,924 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 228 to run in 60 seconds
2017-10-23 16:34:54,383 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:34:54,383 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:34:54,383 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:34:54,384 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:34:55,005 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 226
2017-10-23 16:34:55,005 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:34:55,005 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:34:55,006 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:34:55,006 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:34:55,006 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:34:55,006 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:34:55,006 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 229 to run in 10 seconds
2017-10-23 16:35:05,015 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 229
2017-10-23 16:35:05,015 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:35:05,015 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:35:05,015 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:35:05,015 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:35:05,015 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:35:05,015 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:35:05,015 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 230 to run in 10 seconds
2017-10-23 16:35:09,390 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:35:09,391 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:35:09,391 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:35:09,391 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:35:15,016 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 230
2017-10-23 16:35:15,016 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:35:15,016 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:35:15,016 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:35:15,016 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:35:15,016 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:35:15,017 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:35:15,017 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 231 to run in 10 seconds
2017-10-23 16:35:24,393 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:35:24,393 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:35:24,393 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:35:24,393 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:35:25,018 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 231
2017-10-23 16:35:25,019 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:35:25,019 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:35:25,019 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:35:25,019 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:35:25,019 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:35:25,019 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:35:25,019 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 232 to run in 10 seconds
2017-10-23 16:35:35,028 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 232
2017-10-23 16:35:35,028 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:35:35,028 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:35:35,029 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:35:35,029 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:35:35,029 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:35:35,029 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:35:35,029 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 233 to run in 10 seconds
2017-10-23 16:35:39,399 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:35:39,399 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:35:39,399 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:35:39,400 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:35:45,039 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 233
2017-10-23 16:35:45,039 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:35:45,039 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:35:45,039 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:35:45,040 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:35:45,040 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:35:45,040 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:35:45,040 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 234 to run in 10 seconds
2017-10-23 16:35:53,760 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 227
2017-10-23 16:35:53,761 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:35:53,761 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:35:53,761 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 235 to run in 60 seconds
2017-10-23 16:35:53,761 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 29
2017-10-23 16:35:53,761 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:35:53,761 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 30 to run in 60 seconds
2017-10-23 16:35:53,935 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 228
2017-10-23 16:35:53,936 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 236 to run in 60 seconds
2017-10-23 16:35:54,401 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:35:54,401 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:35:54,401 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:35:54,402 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:35:55,046 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 234
2017-10-23 16:35:55,047 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:35:55,047 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:35:55,047 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:35:55,047 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:35:55,048 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:35:55,048 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:35:55,048 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 237 to run in 10 seconds
2017-10-23 16:36:05,057 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 237
2017-10-23 16:36:05,057 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:36:05,058 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:36:05,058 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:36:05,058 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:36:05,058 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:36:05,058 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:36:05,058 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 238 to run in 10 seconds
2017-10-23 16:36:09,413 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:36:09,413 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:36:09,413 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:36:09,413 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:36:15,062 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 238
2017-10-23 16:36:15,062 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:36:15,062 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:36:15,063 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:36:15,063 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:36:15,063 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:36:15,063 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:36:15,063 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 239 to run in 10 seconds
2017-10-23 16:36:24,424 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:36:24,424 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:36:24,424 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:36:24,425 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:36:25,077 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 239
2017-10-23 16:36:25,078 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:36:25,078 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:36:25,078 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:36:25,078 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:36:25,078 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:36:25,078 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:36:25,079 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 240 to run in 10 seconds
2017-10-23 16:36:35,079 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 240
2017-10-23 16:36:35,079 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:36:35,079 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:36:35,079 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:36:35,079 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:36:35,080 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:36:35,080 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:36:35,080 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 241 to run in 10 seconds
2017-10-23 16:36:39,425 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:36:39,425 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:36:39,425 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:36:39,425 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:36:45,082 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 241
2017-10-23 16:36:45,082 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:36:45,083 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:36:45,083 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:36:45,083 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:36:45,083 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:36:45,084 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:36:45,084 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 242 to run in 10 seconds
2017-10-23 16:36:53,762 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 235
2017-10-23 16:36:53,762 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:36:53,762 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:36:53,762 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 243 to run in 60 seconds
2017-10-23 16:36:53,762 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 30
2017-10-23 16:36:53,762 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:36:53,762 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 31 to run in 60 seconds
2017-10-23 16:36:53,944 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 236
2017-10-23 16:36:53,944 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 244 to run in 60 seconds
2017-10-23 16:36:54,430 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:36:54,431 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:36:54,431 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:36:54,431 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:36:55,087 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 242
2017-10-23 16:36:55,088 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:36:55,088 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:36:55,088 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:36:55,088 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:36:55,088 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:36:55,088 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:36:55,089 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 245 to run in 10 seconds
2017-10-23 16:37:05,098 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 245
2017-10-23 16:37:05,098 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:37:05,098 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:37:05,098 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:37:05,098 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:37:05,098 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:37:05,098 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:37:05,098 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 246 to run in 10 seconds
2017-10-23 16:37:09,430 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:37:09,431 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:37:09,431 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:37:09,431 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:37:15,109 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 246
2017-10-23 16:37:15,109 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:37:15,109 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:37:15,109 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:37:15,109 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:37:15,109 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:37:15,109 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:37:15,110 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 247 to run in 10 seconds
2017-10-23 16:37:24,432 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:37:24,432 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:37:24,432 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:37:24,433 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:37:25,116 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 247
2017-10-23 16:37:25,116 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:37:25,116 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:37:25,116 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:37:25,117 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:37:25,117 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:37:25,117 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:37:25,117 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 248 to run in 10 seconds
2017-10-23 16:37:35,127 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 248
2017-10-23 16:37:35,127 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:37:35,128 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:37:35,128 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:37:35,128 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:37:35,128 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:37:35,128 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:37:35,128 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 249 to run in 10 seconds
2017-10-23 16:37:39,433 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:37:39,434 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:37:39,434 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:37:39,435 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:37:45,138 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 249
2017-10-23 16:37:45,138 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:37:45,138 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:37:45,139 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:37:45,139 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:37:45,139 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:37:45,139 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:37:45,139 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 250 to run in 10 seconds
2017-10-23 16:37:53,765 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 243
2017-10-23 16:37:53,765 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:37:53,765 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:37:53,765 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 251 to run in 60 seconds
2017-10-23 16:37:53,765 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 31
2017-10-23 16:37:53,765 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:37:53,765 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 32 to run in 60 seconds
2017-10-23 16:37:53,952 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 244
2017-10-23 16:37:53,953 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 252 to run in 60 seconds
2017-10-23 16:37:54,440 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:37:54,440 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:37:54,440 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:37:54,441 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:37:55,148 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 250
2017-10-23 16:37:55,148 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:37:55,149 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:37:55,149 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:37:55,149 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:37:55,149 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:37:55,150 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:37:55,150 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 253 to run in 10 seconds
2017-10-23 16:38:05,162 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 253
2017-10-23 16:38:05,163 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:38:05,163 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:38:05,163 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:38:05,163 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:38:05,163 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:38:05,163 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:38:05,163 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 254 to run in 10 seconds
2017-10-23 16:38:09,447 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:38:09,448 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:38:09,448 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:38:09,448 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:38:15,175 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 254
2017-10-23 16:38:15,175 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:38:15,176 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:38:15,176 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:38:15,176 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:38:15,176 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:38:15,176 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:38:15,176 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 255 to run in 10 seconds
2017-10-23 16:38:24,454 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:38:24,455 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:38:24,455 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:38:24,455 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:38:25,178 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 255
2017-10-23 16:38:25,178 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:38:25,179 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:38:25,179 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:38:25,179 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:38:25,179 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:38:25,180 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:38:25,180 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 256 to run in 10 seconds
2017-10-23 16:38:35,191 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 256
2017-10-23 16:38:35,191 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:38:35,192 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:38:35,192 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:38:35,192 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:38:35,192 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:38:35,192 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:38:35,193 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 257 to run in 10 seconds
2017-10-23 16:38:39,462 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:38:39,462 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:38:39,463 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:38:39,463 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:38:45,196 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 257
2017-10-23 16:38:45,196 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:38:45,196 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:38:45,197 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:38:45,197 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:38:45,197 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:38:45,197 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:38:45,197 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 258 to run in 10 seconds
2017-10-23 16:38:53,765 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkNodeRequestSpike with id 251
2017-10-23 16:38:53,765 | DEBUG    | node.py              (1999) | checkNodeRequestSpike | Node4 checking its request amount
2017-10-23 16:38:53,765 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:38:53,765 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkNodeRequestSpike with id 259 to run in 60 seconds
2017-10-23 16:38:53,765 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 32
2017-10-23 16:38:53,766 | DEBUG    | notifier_plugin_manager.py (  74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike
2017-10-23 16:38:53,766 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 33 to run in 60 seconds
2017-10-23 16:38:53,953 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action dump_json_file with id 252
2017-10-23 16:38:53,954 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action dump_json_file with id 260 to run in 60 seconds
2017-10-23 16:38:54,469 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:38:54,469 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:38:54,469 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:38:54,469 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:38:55,208 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 258
2017-10-23 16:38:55,209 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:38:55,209 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:38:55,209 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:38:55,209 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:38:55,209 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:38:55,209 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:38:55,209 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 261 to run in 10 seconds
2017-10-23 16:39:05,210 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 261
2017-10-23 16:39:05,210 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:39:05,210 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:39:05,218 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:39:05,218 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:39:05,218 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:39:05,219 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:39:05,219 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 262 to run in 10 seconds
2017-10-23 16:39:09,471 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:39:09,474 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:39:09,474 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:39:09,475 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:39:15,223 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 262
2017-10-23 16:39:15,224 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:39:15,224 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:39:15,224 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:39:15,224 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:39:15,225 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:39:15,225 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:39:15,225 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 263 to run in 10 seconds
2017-10-23 16:39:24,476 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:39:24,477 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:39:24,477 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:39:24,477 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:39:25,235 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 263
2017-10-23 16:39:25,236 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:39:25,236 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:39:25,236 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:39:25,236 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:39:25,236 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:39:25,236 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:39:25,236 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 264 to run in 10 seconds
2017-10-23 16:39:35,238 | TRACE    | has_action_queue.py  (  66) | _serviceActions | Node4 running action checkPerformance with id 264
2017-10-23 16:39:35,239 | TRACE    | node.py              (1978) | checkPerformance | Node4 checking its performance
2017-10-23 16:39:35,239 | DEBUG    | notifier_plugin_manager.py (  80) | sendMessageUponSuspiciousSpike | NodeRequestSuspiciousSpike: New value 0 is within bounds. Average: 0.0
2017-10-23 16:39:35,239 | DEBUG    | monitor.py           ( 335) | isMasterThroughputTooLow | Node4 master throughput is not measurable.
2017-10-23 16:39:35,239 | TRACE    | monitor.py           ( 361) | isMasterReqLatencyTooHigh | Node4 found master's latency to be lower than the threshold for all requests.
2017-10-23 16:39:35,239 | TRACE    | monitor.py           ( 391) | isMasterAvgReqLatencyTooHigh | Node4 found difference between master and backups avg latencies to be acceptable
2017-10-23 16:39:35,240 | DEBUG    | node.py              (1995) | checkPerformance | Node4's master has higher performance than backups
2017-10-23 16:39:35,240 | TRACE    | has_action_queue.py  (  36) | _schedule | Node4 scheduling action checkPerformance with id 265 to run in 10 seconds
2017-10-23 16:39:39,486 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node2 HA(host='10.0.0.3', port=9703)
2017-10-23 16:39:39,486 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node1 HA(host='10.0.0.2', port=9701)
2017-10-23 16:39:39,486 | DEBUG    | kit_zstack.py        (  67) | reconcileNodeReg | Node4 matched remote Node3 HA(host='10.0.0.4', port=9705)
2017-10-23 16:39:39,487 | DEBUG    | kit_zstack.py        (  50) | maintainConnections | Node4 next check for retries in 15.00 seconds
2017-10-23 16:39:43,272 | TRACE    | zstack.py            ( 479) | _receiveFromListener | Node4C got 1 messages through listener
2017-10-23 16:39:43,288 | DEBUG    | node.py              (2236) | verifySignature | Node4 authenticated V4SGRU86Z58d6TV7PBUe6f signature on  request 1508776783249274
2017-10-23 16:39:43,289 | TRACE    | node.py              (1388) | validateClientMsg | Node4C received CLIENT message: SafeRequest: {'operation': {'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00'}, 'timeout': 10, 'action': 'start', 'version': '1.1.43', 'type': '109', 'force': True, 'justification': None, 'name': 'upgrade-1143', 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3'}, 'reqId': 1508776783249274, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'EejcE22puEWaVFYPWeEEm89YhBeBTvcwStJbP7Wsdenfxc9n5FCew1GFYwUnJsFSPW7YatjmJPYuamAYfagBQrN'}
2017-10-23 16:39:43,289 | DEBUG    | node.py              (1434) | processClientInBox | Node4C processing b'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*' request SafeRequest: {'operation': {'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00'}, 'timeout': 10, 'action': 'start', 'version': '1.1.43', 'type': '109', 'force': True, 'justification': None, 'name': 'upgrade-1143', 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3'}, 'reqId': 1508776783249274, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'EejcE22puEWaVFYPWeEEm89YhBeBTvcwStJbP7Wsdenfxc9n5FCew1GFYwUnJsFSPW7YatjmJPYuamAYfagBQrN'}
2017-10-23 16:39:43,289 | INFO     | upgrader.py          ( 292) | handleUpgradeTxn | Node 'Node4' handles upgrade txn {'timeout': 10, 'version': '1.1.43', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00'}, 'type': '109', 'name': 'upgrade-1143', 'txnTime': None, 'reqId': 1508776783249274, 'force': True, 'justification': None, 'signature': 'EejcE22puEWaVFYPWeEEm89YhBeBTvcwStJbP7Wsdenfxc9n5FCew1GFYwUnJsFSPW7YatjmJPYuamAYfagBQrN', 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3', 'action': 'start'}
2017-10-23 16:39:43,289 | INFO     | upgrader.py          ( 321) | handleUpgradeTxn | Node 'Node4' schedules upgrade to 1.1.43
2017-10-23 16:39:43,289 | INFO     | upgrader.py          ( 359) | _scheduleUpgrade | 4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA's upgrader processing upgrade for version 1.1.43
2017-10-23 16:39:43,290 | TRACE    | has_action_queue.py  (  40) | _schedule | 4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA scheduling action partial(_callUpgradeAgent) with id 1 to run now
2017-10-23 16:39:43,290 | DEBUG    | node.py              (1708) | processRequest | Node4 received client request: SafeRequest: {'operation': {'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00'}, 'timeout': 10, 'action': 'start', 'version': '1.1.43', 'type': '109', 'force': True, 'justification': None, 'name': 'upgrade-1143', 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3'}, 'reqId': 1508776783249274, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'EejcE22puEWaVFYPWeEEm89YhBeBTvcwStJbP7Wsdenfxc9n5FCew1GFYwUnJsFSPW7YatjmJPYuamAYfagBQrN'} from b'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*'
2017-10-23 16:39:43,290 | TRACE    | propagator.py        ( 165) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00'}, 'timeout': 10, 'action': 'start', 'version': '1.1.43', 'type': '109', 'force': True, 'justification': None, 'name': 'upgrade-1143', 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3'}, 'reqId': 1508776783249274, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'EejcE22puEWaVFYPWeEEm89YhBeBTvcwStJbP7Wsdenfxc9n5FCew1GFYwUnJsFSPW7YatjmJPYuamAYfagBQrN'}
2017-10-23 16:39:43,291 | INFO     | propagator.py        ( 148) | propagate | Node4 propagating request ('V4SGRU86Z58d6TV7PBUe6f', 1508776783249274) from client b'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*'
2017-10-23 16:39:43,291 | DEBUG    | node.py              (2608) | send | Node4 sending message PROPAGATE{'request': {'operation': {'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00'}, 'timeout': 10, 'action': 'start', 'version': '1.1.43', 'type': '109', 'force': True, 'justification': None, 'name': 'upgrade-1143', 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3'}, 'reqId': 1508776783249274, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'EejcE22puEWaVFYPWeEEm89YhBeBTvcwStJbP7Wsdenfxc9n5FCew1GFYwUnJsFSPW7YatjmJPYuamAYfagBQrN'}, 'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*'} to all recipients: ['Node2', 'Node1', 'Node3']
2017-10-23 16:39:43,291 | DEBUG    | propagator.py        ( 246) | tryForwarding | Node4 not forwarding request SafeRequest: {'operation': {'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00'}, 'timeout': 10, 'action': 'start', 'version': '1.1.43', 'type': '109', 'force': True, 'justification': None, 'name': 'upgrade-1143', 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3'}, 'reqId': 1508776783249274, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'EejcE22puEWaVFYPWeEEm89YhBeBTvcwStJbP7Wsdenfxc9n5FCew1GFYwUnJsFSPW7YatjmJPYuamAYfagBQrN'} to its replicas since not finalised
2017-10-23 16:39:43,291 | TRACE    | zstack.py            ( 754) | transmitThroughListener | Node4C transmitting b'{"op":"REQACK","reqId":1508776783249274,"identifier":"V4SGRU86Z58d6TV7PBUe6f"}' to b'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*' through listener socket
2017-10-23 16:39:43,291 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"request":{"operation":{"schedule":{"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv":"2017-10-17T11:20:00.000000+00:00","DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya":"2017-10-17T11:30:00.000000+00:00","4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA":"2017-10-17T11:35:00.000000+00:00","4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc":"2017-10-17T11:40:00.000000+00:00","8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb":"2017-10-17T11:25:00.000000+00:00"},"timeout":10,"action":"start","version":"1.1.43","type":"109","force":true,"justification":null,"name":"upgrade-1143","sha256":"f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3"},"reqId":1508776783249274,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"EejcE22puEWaVFYPWeEEm89YhBeBTvcwStJbP7Wsdenfxc9n5FCew1GFYwUnJsFSPW7YatjmJPYuamAYfagBQrN"},"op":"PROPAGATE","senderClient":"xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE\\/:%0nV5Wx*"}' to Node2
2017-10-23 16:39:43,292 | TRACE    | batched.py           (  96) | flushOutBoxes | Node4 sending msg b'{"request":{"operation":{"schedule":{"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv":"2017-10-17T11:20:00.000000+00:00","DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya":"2017-10-17T11:30:00.000000+00:00","4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA":"2017-10-17T11:35:00.000000+00:00","4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc":"2017-10-17T11:40:00.000000+00:00","8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb":"2017-10-17T11:25:00.000000+00:00"},"timeout":10,"action":"start","version":"1.1.43","type":"109","force":true,"justification":null,"name":"upgrade-1143","sha256":"f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3"},"reqId":1508776783249274,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"EejcE22puEWaVFYPWeEEm89YhBeBTvcwStJbP7Wsdenfxc9n5FCew1GFYwUnJsFSPW7YatjmJPYuamAYfagBQrN"},"op":"PROPAGATE","senderClient":"xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE\\/:%0nV5Wx*"}' to Node2
2017-10-23 16:39:43,293 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"request":{"operation":{"schedule":{"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv":"2017-10-17T11:20:00.000000+00:00","DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya":"2017-10-17T11:30:00.000000+00:00","4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA":"2017-10-17T11:35:00.000000+00:00","4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc":"2017-10-17T11:40:00.000000+00:00","8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb":"2017-10-17T11:25:00.000000+00:00"},"timeout":10,"action":"start","version":"1.1.43","type":"109","force":true,"justification":null,"name":"upgrade-1143","sha256":"f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3"},"reqId":1508776783249274,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"EejcE22puEWaVFYPWeEEm89YhBeBTvcwStJbP7Wsdenfxc9n5FCew1GFYwUnJsFSPW7YatjmJPYuamAYfagBQrN"},"op":"PROPAGATE","senderClient":"xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE\\/:%0nV5Wx*"}' to Node1
2017-10-23 16:39:43,293 | TRACE    | batched.py           (  96) | flushOutBoxes | Node4 sending msg b'{"request":{"operation":{"schedule":{"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv":"2017-10-17T11:20:00.000000+00:00","DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya":"2017-10-17T11:30:00.000000+00:00","4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA":"2017-10-17T11:35:00.000000+00:00","4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc":"2017-10-17T11:40:00.000000+00:00","8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb":"2017-10-17T11:25:00.000000+00:00"},"timeout":10,"action":"start","version":"1.1.43","type":"109","force":true,"justification":null,"name":"upgrade-1143","sha256":"f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3"},"reqId":1508776783249274,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"EejcE22puEWaVFYPWeEEm89YhBeBTvcwStJbP7Wsdenfxc9n5FCew1GFYwUnJsFSPW7YatjmJPYuamAYfagBQrN"},"op":"PROPAGATE","senderClient":"xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE\\/:%0nV5Wx*"}' to Node1
2017-10-23 16:39:43,293 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"request":{"operation":{"schedule":{"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv":"2017-10-17T11:20:00.000000+00:00","DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya":"2017-10-17T11:30:00.000000+00:00","4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA":"2017-10-17T11:35:00.000000+00:00","4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc":"2017-10-17T11:40:00.000000+00:00","8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb":"2017-10-17T11:25:00.000000+00:00"},"timeout":10,"action":"start","version":"1.1.43","type":"109","force":true,"justification":null,"name":"upgrade-1143","sha256":"f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3"},"reqId":1508776783249274,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"EejcE22puEWaVFYPWeEEm89YhBeBTvcwStJbP7Wsdenfxc9n5FCew1GFYwUnJsFSPW7YatjmJPYuamAYfagBQrN"},"op":"PROPAGATE","senderClient":"xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE\\/:%0nV5Wx*"}' to Node3
2017-10-23 16:39:43,293 | TRACE    | batched.py           (  96) | flushOutBoxes | Node4 sending msg b'{"request":{"operation":{"schedule":{"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv":"2017-10-17T11:20:00.000000+00:00","DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya":"2017-10-17T11:30:00.000000+00:00","4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA":"2017-10-17T11:35:00.000000+00:00","4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc":"2017-10-17T11:40:00.000000+00:00","8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb":"2017-10-17T11:25:00.000000+00:00"},"timeout":10,"action":"start","version":"1.1.43","type":"109","force":true,"justification":null,"name":"upgrade-1143","sha256":"f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3"},"reqId":1508776783249274,"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"EejcE22puEWaVFYPWeEEm89YhBeBTvcwStJbP7Wsdenfxc9n5FCew1GFYwUnJsFSPW7YatjmJPYuamAYfagBQrN"},"op":"PROPAGATE","senderClient":"xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE\\/:%0nV5Wx*"}' to Node3
2017-10-23 16:39:43,293 | TRACE    | has_action_queue.py  (  66) | _serviceActions | 4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA running action partial(_callUpgradeAgent) with id 1
2017-10-23 16:39:43,294 | INFO     | upgrader.py          ( 421) | _callUpgradeAgent | 4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA's upgrader calling agent for upgrade
2017-10-23 16:39:43,294 | INFO     | node.py              ( 254) | notify_upgrade_start | Node4 is about to be upgraded, sending NODE_UPGRADE
2017-10-23 16:39:43,297 | DEBUG    | node.py              (2608) | send | Node4 sending message Request: {'operation': {'type': '110', 'signature': '5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'reqId': 1508776783297473, 'identifier': '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 'signature': '491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm'} to all recipients: ['Node2', 'Node1', 'Node3']
2017-10-23 16:39:43,298 | TRACE    | zstack.py            ( 479) | _receiveFromListener | Node4 got 4 messages through listener
2017-10-23 16:39:43,299 | DEBUG    | node.py              (2236) | verifySignature | Node4 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate  request 1508776783249274
2017-10-23 16:39:43,299 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node1: PROPAGATE{'request': {'operation': {'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00'}, 'timeout': 10, 'action': 'start', 'version': '1.1.43', 'justification': None, 'force': True, 'type': '109', 'name': 'upgrade-1143', 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3'}, 'reqId': 1508776783249274, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'EejcE22puEWaVFYPWeEEm89YhBeBTvcwStJbP7Wsdenfxc9n5FCew1GFYwUnJsFSPW7YatjmJPYuamAYfagBQrN'}, 'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*'}
2017-10-23 16:39:43,299 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'request': {'operation': {'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00'}, 'timeout': 10, 'action': 'start', 'version': '1.1.43', 'justification': None, 'force': True, 'type': '109', 'name': 'upgrade-1143', 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3'}, 'reqId': 1508776783249274, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'EejcE22puEWaVFYPWeEEm89YhBeBTvcwStJbP7Wsdenfxc9n5FCew1GFYwUnJsFSPW7YatjmJPYuamAYfagBQrN'}, 'op': 'PROPAGATE', 'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*'}, 'Node1')
2017-10-23 16:39:43,300 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox PROPAGATE{'request': {'operation': {'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00'}, 'timeout': 10, 'action': 'start', 'version': '1.1.43', 'justification': None, 'force': True, 'type': '109', 'name': 'upgrade-1143', 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3'}, 'reqId': 1508776783249274, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'EejcE22puEWaVFYPWeEEm89YhBeBTvcwStJbP7Wsdenfxc9n5FCew1GFYwUnJsFSPW7YatjmJPYuamAYfagBQrN'}, 'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*'}
2017-10-23 16:39:43,300 | DEBUG    | node.py              (2236) | verifySignature | Node4 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate  request 1508776783249274
2017-10-23 16:39:43,300 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: PROPAGATE{'request': {'operation': {'force': True, 'timeout': 10, 'action': 'start', 'version': '1.1.43', 'justification': None, 'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00'}, 'type': '109', 'name': 'upgrade-1143', 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3'}, 'signature': 'EejcE22puEWaVFYPWeEEm89YhBeBTvcwStJbP7Wsdenfxc9n5FCew1GFYwUnJsFSPW7YatjmJPYuamAYfagBQrN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1508776783249274}, 'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*'}
2017-10-23 16:39:43,300 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'request': {'operation': {'force': True, 'timeout': 10, 'action': 'start', 'version': '1.1.43', 'justification': None, 'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00'}, 'type': '109', 'name': 'upgrade-1143', 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3'}, 'signature': 'EejcE22puEWaVFYPWeEEm89YhBeBTvcwStJbP7Wsdenfxc9n5FCew1GFYwUnJsFSPW7YatjmJPYuamAYfagBQrN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1508776783249274}, 'op': 'PROPAGATE', 'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*'}, 'Node2')
2017-10-23 16:39:43,300 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox PROPAGATE{'request': {'operation': {'force': True, 'timeout': 10, 'action': 'start', 'version': '1.1.43', 'justification': None, 'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00'}, 'type': '109', 'name': 'upgrade-1143', 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3'}, 'signature': 'EejcE22puEWaVFYPWeEEm89YhBeBTvcwStJbP7Wsdenfxc9n5FCew1GFYwUnJsFSPW7YatjmJPYuamAYfagBQrN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1508776783249274}, 'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*'}
2017-10-23 16:39:43,301 | DEBUG    | node.py              (2236) | verifySignature | Node4 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate  request 1508776783249274
2017-10-23 16:39:43,301 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: PROPAGATE{'request': {'operation': {'force': True, 'timeout': 10, 'action': 'start', 'version': '1.1.43', 'justification': None, 'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00'}, 'type': '109', 'name': 'upgrade-1143', 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3'}, 'reqId': 1508776783249274, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'EejcE22puEWaVFYPWeEEm89YhBeBTvcwStJbP7Wsdenfxc9n5FCew1GFYwUnJsFSPW7YatjmJPYuamAYfagBQrN'}, 'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*'}
2017-10-23 16:39:43,301 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'request': {'operation': {'force': True, 'timeout': 10, 'action': 'start', 'version': '1.1.43', 'justification': None, 'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00'}, 'type': '109', 'name': 'upgrade-1143', 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3'}, 'reqId': 1508776783249274, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'EejcE22puEWaVFYPWeEEm89YhBeBTvcwStJbP7Wsdenfxc9n5FCew1GFYwUnJsFSPW7YatjmJPYuamAYfagBQrN'}, 'op': 'PROPAGATE', 'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*'}, 'Node3')
2017-10-23 16:39:43,301 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox PROPAGATE{'request': {'operation': {'force': True, 'timeout': 10, 'action': 'start', 'version': '1.1.43', 'justification': None, 'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00'}, 'type': '109', 'name': 'upgrade-1143', 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3'}, 'reqId': 1508776783249274, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'EejcE22puEWaVFYPWeEEm89YhBeBTvcwStJbP7Wsdenfxc9n5FCew1GFYwUnJsFSPW7YatjmJPYuamAYfagBQrN'}, 'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*'}
2017-10-23 16:39:43,302 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst', 'type': '110'}, 'reqId': 1508776783276850, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ'}, 'Node1')
2017-10-23 16:39:43,302 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox SafeRequest: {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst', 'type': '110'}, 'reqId': 1508776783276850, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ'}
2017-10-23 16:39:43,302 | DEBUG    | node.py              (1757) | processPropagate | Node Node4 received propagated request: PROPAGATE{'request': {'operation': {'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00'}, 'timeout': 10, 'action': 'start', 'version': '1.1.43', 'justification': None, 'force': True, 'type': '109', 'name': 'upgrade-1143', 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3'}, 'reqId': 1508776783249274, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'EejcE22puEWaVFYPWeEEm89YhBeBTvcwStJbP7Wsdenfxc9n5FCew1GFYwUnJsFSPW7YatjmJPYuamAYfagBQrN'}, 'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*'}
2017-10-23 16:39:43,303 | TRACE    | propagator.py        ( 141) | propagate | Node4 already propagated SafeRequest: {'operation': {'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00'}, 'timeout': 10, 'action': 'start', 'version': '1.1.43', 'justification': None, 'force': True, 'type': '109', 'name': 'upgrade-1143', 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3'}, 'reqId': 1508776783249274, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'EejcE22puEWaVFYPWeEEm89YhBeBTvcwStJbP7Wsdenfxc9n5FCew1GFYwUnJsFSPW7YatjmJPYuamAYfagBQrN'}
2017-10-23 16:39:43,303 | DEBUG    | propagator.py        ( 214) | forward | Node4 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1508776783249274) to 0 replicas
2017-10-23 16:39:43,303 | DEBUG    | node.py              (1757) | processPropagate | Node Node4 received propagated request: PROPAGATE{'request': {'operation': {'force': True, 'timeout': 10, 'action': 'start', 'version': '1.1.43', 'justification': None, 'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00'}, 'type': '109', 'name': 'upgrade-1143', 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3'}, 'signature': 'EejcE22puEWaVFYPWeEEm89YhBeBTvcwStJbP7Wsdenfxc9n5FCew1GFYwUnJsFSPW7YatjmJPYuamAYfagBQrN', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1508776783249274}, 'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*'}
2017-10-23 16:39:43,303 | TRACE    | propagator.py        ( 141) | propagate | Node4 already propagated SafeRequest: {'operation': {'force': True, 'timeout': 10, 'action': 'start', 'version': '1.1.43', 'justification': None, 'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00'}, 'type': '109', 'name': 'upgrade-1143', 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3'}, 'reqId': 1508776783249274, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'EejcE22puEWaVFYPWeEEm89YhBeBTvcwStJbP7Wsdenfxc9n5FCew1GFYwUnJsFSPW7YatjmJPYuamAYfagBQrN'}
2017-10-23 16:39:43,304 | DEBUG    | propagator.py        ( 246) | tryForwarding | Node4 not forwarding request SafeRequest: {'operation': {'force': True, 'timeout': 10, 'action': 'start', 'version': '1.1.43', 'justification': None, 'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00'}, 'type': '109', 'name': 'upgrade-1143', 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3'}, 'reqId': 1508776783249274, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'EejcE22puEWaVFYPWeEEm89YhBeBTvcwStJbP7Wsdenfxc9n5FCew1GFYwUnJsFSPW7YatjmJPYuamAYfagBQrN'} to its replicas since already forwarded
2017-10-23 16:39:43,304 | DEBUG    | node.py              (1757) | processPropagate | Node Node4 received propagated request: PROPAGATE{'request': {'operation': {'force': True, 'timeout': 10, 'action': 'start', 'version': '1.1.43', 'justification': None, 'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00'}, 'type': '109', 'name': 'upgrade-1143', 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3'}, 'reqId': 1508776783249274, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'EejcE22puEWaVFYPWeEEm89YhBeBTvcwStJbP7Wsdenfxc9n5FCew1GFYwUnJsFSPW7YatjmJPYuamAYfagBQrN'}, 'senderClient': 'xo5JUY.S$6PWbpmz5XzA<f)uJEc2oE/:%0nV5Wx*'}
2017-10-23 16:39:43,304 | TRACE    | propagator.py        ( 141) | propagate | Node4 already propagated SafeRequest: {'operation': {'force': True, 'timeout': 10, 'action': 'start', 'version': '1.1.43', 'justification': None, 'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00'}, 'type': '109', 'name': 'upgrade-1143', 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3'}, 'reqId': 1508776783249274, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'EejcE22puEWaVFYPWeEEm89YhBeBTvcwStJbP7Wsdenfxc9n5FCew1GFYwUnJsFSPW7YatjmJPYuamAYfagBQrN'}
2017-10-23 16:39:43,304 | DEBUG    | propagator.py        ( 246) | tryForwarding | Node4 not forwarding request SafeRequest: {'operation': {'force': True, 'timeout': 10, 'action': 'start', 'version': '1.1.43', 'justification': None, 'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-17T11:20:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-17T11:30:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-17T11:35:00.000000+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-17T11:25:00.000000+00:00', '4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc': '2017-10-17T11:40:00.000000+00:00'}, 'type': '109', 'name': 'upgrade-1143', 'sha256': 'f6f2ea8f45d8a057c9566a33f99474da2e5c6a6604d736121650e2730c6fb0a3'}, 'reqId': 1508776783249274, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'EejcE22puEWaVFYPWeEEm89YhBeBTvcwStJbP7Wsdenfxc9n5FCew1GFYwUnJsFSPW7YatjmJPYuamAYfagBQrN'} to its replicas since already forwarded
2017-10-23 16:39:43,305 | TRACE    | propagator.py        ( 165) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst', 'type': '110'}, 'reqId': 1508776783276850, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ'}
2017-10-23 16:39:43,305 | INFO     | propagator.py        ( 148) | propagate | Node4 propagating request ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1508776783276850) from client Node1
2017-10-23 16:39:43,305 | DEBUG    | node.py              (2608) | send | Node4 sending message PROPAGATE{'request': {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst', 'type': '110'}, 'reqId': 1508776783276850, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ'}, 'senderClient': 'Node1'} to all recipients: ['Node2', 'Node1', 'Node3']
2017-10-23 16:39:43,305 | DEBUG    | propagator.py        ( 246) | tryForwarding | Node4 not forwarding request SafeRequest: {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst', 'type': '110'}, 'reqId': 1508776783276850, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ'} to its replicas since not finalised
2017-10-23 16:39:43,305 | DEBUG    | batched.py           ( 100) | flushOutBoxes | Node4 batching 2 msgs to Node2 into one transmission
2017-10-23 16:39:43,305 | TRACE    | batched.py           ( 101) | flushOutBoxes |     messages: deque([b'{"operation":{"type":"110","signature":"5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1","data":{"action":"in_progress","version":"1.1.43"}},"reqId":1508776783297473,"identifier":"4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA","signature":"491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm"}', b'{"request":{"operation":{"data":{"action":"in_progress","version":"1.1.43"},"signature":"2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst","type":"110"},"reqId":1508776783276850,"identifier":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv","signature":"5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ"},"op":"PROPAGATE","senderClient":"Node1"}'])
2017-10-23 16:39:43,306 | TRACE    | batched.py           ( 110) | flushOutBoxes | Node4 sending payload to Node2: b'{"op":"BATCH","signature":null,"messages":["{\\"operation\\":{\\"type\\":\\"110\\",\\"signature\\":\\"5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1\\",\\"data\\":{\\"action\\":\\"in_progress\\",\\"version\\":\\"1.1.43\\"}},\\"reqId\\":1508776783297473,\\"identifier\\":\\"4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA\\",\\"signature\\":\\"491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm\\"}","{\\"request\\":{\\"operation\\":{\\"data\\":{\\"action\\":\\"in_progress\\",\\"version\\":\\"1.1.43\\"},\\"signature\\":\\"2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst\\",\\"type\\":\\"110\\"},\\"reqId\\":1508776783276850,\\"identifier\\":\\"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv\\",\\"signature\\":\\"5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"Node1\\"}"]}'
2017-10-23 16:39:43,306 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"op":"BATCH","signature":null,"messages":["{\\"operation\\":{\\"type\\":\\"110\\",\\"signature\\":\\"5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1\\",\\"data\\":{\\"action\\":\\"in_progress\\",\\"version\\":\\"1.1.43\\"}},\\"reqId\\":1508776783297473,\\"identifier\\":\\"4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA\\",\\"signature\\":\\"491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm\\"}","{\\"request\\":{\\"operation\\":{\\"data\\":{\\"action\\":\\"in_progress\\",\\"version\\":\\"1.1.43\\"},\\"signature\\":\\"2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst\\",\\"type\\":\\"110\\"},\\"reqId\\":1508776783276850,\\"identifier\\":\\"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv\\",\\"signature\\":\\"5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"Node1\\"}"]}' to Node2
2017-10-23 16:39:43,310 | DEBUG    | batched.py           ( 100) | flushOutBoxes | Node4 batching 2 msgs to Node1 into one transmission
2017-10-23 16:39:43,310 | TRACE    | batched.py           ( 101) | flushOutBoxes |     messages: deque([b'{"operation":{"type":"110","signature":"5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1","data":{"action":"in_progress","version":"1.1.43"}},"reqId":1508776783297473,"identifier":"4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA","signature":"491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm"}', b'{"request":{"operation":{"data":{"action":"in_progress","version":"1.1.43"},"signature":"2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst","type":"110"},"reqId":1508776783276850,"identifier":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv","signature":"5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ"},"op":"PROPAGATE","senderClient":"Node1"}'])
2017-10-23 16:39:43,310 | TRACE    | batched.py           ( 110) | flushOutBoxes | Node4 sending payload to Node1: b'{"op":"BATCH","signature":null,"messages":["{\\"operation\\":{\\"type\\":\\"110\\",\\"signature\\":\\"5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1\\",\\"data\\":{\\"action\\":\\"in_progress\\",\\"version\\":\\"1.1.43\\"}},\\"reqId\\":1508776783297473,\\"identifier\\":\\"4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA\\",\\"signature\\":\\"491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm\\"}","{\\"request\\":{\\"operation\\":{\\"data\\":{\\"action\\":\\"in_progress\\",\\"version\\":\\"1.1.43\\"},\\"signature\\":\\"2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst\\",\\"type\\":\\"110\\"},\\"reqId\\":1508776783276850,\\"identifier\\":\\"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv\\",\\"signature\\":\\"5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"Node1\\"}"]}'
2017-10-23 16:39:43,310 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"op":"BATCH","signature":null,"messages":["{\\"operation\\":{\\"type\\":\\"110\\",\\"signature\\":\\"5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1\\",\\"data\\":{\\"action\\":\\"in_progress\\",\\"version\\":\\"1.1.43\\"}},\\"reqId\\":1508776783297473,\\"identifier\\":\\"4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA\\",\\"signature\\":\\"491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm\\"}","{\\"request\\":{\\"operation\\":{\\"data\\":{\\"action\\":\\"in_progress\\",\\"version\\":\\"1.1.43\\"},\\"signature\\":\\"2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst\\",\\"type\\":\\"110\\"},\\"reqId\\":1508776783276850,\\"identifier\\":\\"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv\\",\\"signature\\":\\"5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"Node1\\"}"]}' to Node1
2017-10-23 16:39:43,311 | DEBUG    | batched.py           ( 100) | flushOutBoxes | Node4 batching 2 msgs to Node3 into one transmission
2017-10-23 16:39:43,311 | TRACE    | batched.py           ( 101) | flushOutBoxes |     messages: deque([b'{"operation":{"type":"110","signature":"5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1","data":{"action":"in_progress","version":"1.1.43"}},"reqId":1508776783297473,"identifier":"4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA","signature":"491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm"}', b'{"request":{"operation":{"data":{"action":"in_progress","version":"1.1.43"},"signature":"2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst","type":"110"},"reqId":1508776783276850,"identifier":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv","signature":"5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ"},"op":"PROPAGATE","senderClient":"Node1"}'])
2017-10-23 16:39:43,311 | TRACE    | batched.py           ( 110) | flushOutBoxes | Node4 sending payload to Node3: b'{"op":"BATCH","signature":null,"messages":["{\\"operation\\":{\\"type\\":\\"110\\",\\"signature\\":\\"5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1\\",\\"data\\":{\\"action\\":\\"in_progress\\",\\"version\\":\\"1.1.43\\"}},\\"reqId\\":1508776783297473,\\"identifier\\":\\"4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA\\",\\"signature\\":\\"491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm\\"}","{\\"request\\":{\\"operation\\":{\\"data\\":{\\"action\\":\\"in_progress\\",\\"version\\":\\"1.1.43\\"},\\"signature\\":\\"2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst\\",\\"type\\":\\"110\\"},\\"reqId\\":1508776783276850,\\"identifier\\":\\"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv\\",\\"signature\\":\\"5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"Node1\\"}"]}'
2017-10-23 16:39:43,311 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"op":"BATCH","signature":null,"messages":["{\\"operation\\":{\\"type\\":\\"110\\",\\"signature\\":\\"5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1\\",\\"data\\":{\\"action\\":\\"in_progress\\",\\"version\\":\\"1.1.43\\"}},\\"reqId\\":1508776783297473,\\"identifier\\":\\"4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA\\",\\"signature\\":\\"491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm\\"}","{\\"request\\":{\\"operation\\":{\\"data\\":{\\"action\\":\\"in_progress\\",\\"version\\":\\"1.1.43\\"},\\"signature\\":\\"2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst\\",\\"type\\":\\"110\\"},\\"reqId\\":1508776783276850,\\"identifier\\":\\"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv\\",\\"signature\\":\\"5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"Node1\\"}"]}' to Node3
2017-10-23 16:39:43,312 | TRACE    | zstack.py            ( 479) | _receiveFromListener | Node4 got 3 messages through listener
2017-10-23 16:39:43,313 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': 'Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t', 'type': '110'}, 'signature': '2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9', 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'reqId': 1508776783283567}, 'Node2')
2017-10-23 16:39:43,313 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox SafeRequest: {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': 'Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t', 'type': '110'}, 'reqId': 1508776783283567, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9'}
2017-10-23 16:39:43,313 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: BATCH{'signature': None, 'messages': ['{"reqId":1508776783291152,"identifier":"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya","operation":{"signature":"3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye","data":{"action":"in_progress","version":"1.1.43"},"type":"110"},"signature":"2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ"}', '{"request":{"reqId":1508776783276850,"identifier":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv","operation":{"type":"110","data":{"action":"in_progress","version":"1.1.43"},"signature":"2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst"},"signature":"5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ"},"op":"PROPAGATE","senderClient":"Node1"}']}
2017-10-23 16:39:43,313 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"reqId":1508776783291152,"identifier":"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya","operation":{"signature":"3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye","data":{"action":"in_progress","version":"1.1.43"},"type":"110"},"signature":"2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ"}', '{"request":{"reqId":1508776783276850,"identifier":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv","operation":{"type":"110","data":{"action":"in_progress","version":"1.1.43"},"signature":"2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst"},"signature":"5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ"},"op":"PROPAGATE","senderClient":"Node1"}']}, 'Node3')
2017-10-23 16:39:43,313 | DEBUG    | node.py              (1272) | unpackNodeMsg | Node4 processing a batch BATCH{'signature': None, 'messages': ['{"reqId":1508776783291152,"identifier":"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya","operation":{"signature":"3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye","data":{"action":"in_progress","version":"1.1.43"},"type":"110"},"signature":"2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ"}', '{"request":{"reqId":1508776783276850,"identifier":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv","operation":{"type":"110","data":{"action":"in_progress","version":"1.1.43"},"signature":"2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst"},"signature":"5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ"},"op":"PROPAGATE","senderClient":"Node1"}']}
2017-10-23 16:39:43,313 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye', 'type': '110'}, 'reqId': 1508776783291152, 'identifier': 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 'signature': '2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ'}, 'Node3')
2017-10-23 16:39:43,313 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox SafeRequest: {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye', 'type': '110'}, 'reqId': 1508776783291152, 'identifier': 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 'signature': '2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ'}
2017-10-23 16:39:43,317 | DEBUG    | node.py              (2236) | verifySignature | Node4 authenticated Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv signature on propagate  request 1508776783276850
2017-10-23 16:39:43,317 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: PROPAGATE{'request': {'operation': {'type': '110', 'signature': '2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'reqId': 1508776783276850, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ'}, 'senderClient': 'Node1'}
2017-10-23 16:39:43,317 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'request': {'operation': {'type': '110', 'signature': '2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'reqId': 1508776783276850, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ'}, 'op': 'PROPAGATE', 'senderClient': 'Node1'}, 'Node3')
2017-10-23 16:39:43,317 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox PROPAGATE{'request': {'operation': {'type': '110', 'signature': '2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'reqId': 1508776783276850, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ'}, 'senderClient': 'Node1'}
2017-10-23 16:39:43,317 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: BATCH{'signature': None, 'messages': ['{"op":"PREPREPARE","discarded":1,"stateRootHash":null,"viewNo":0,"ppTime":1508776783,"digest":"ba63b43e76c787646532438003bbbef000ca7dee04421bed8a9d59d3f3e09bd8","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1508776783249274]],"ppSeqNo":1,"ledgerId":2,"txnRootHash":null,"instId":1}', '{"request":{"identifier":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv","signature":"5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ","operation":{"type":"110","data":{"version":"1.1.43","action":"in_progress"},"signature":"2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst"},"reqId":1508776783276850},"senderClient":"Node1","op":"PROPAGATE"}']}
2017-10-23 16:39:43,317 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"op":"PREPREPARE","discarded":1,"stateRootHash":null,"viewNo":0,"ppTime":1508776783,"digest":"ba63b43e76c787646532438003bbbef000ca7dee04421bed8a9d59d3f3e09bd8","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1508776783249274]],"ppSeqNo":1,"ledgerId":2,"txnRootHash":null,"instId":1}', '{"request":{"identifier":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv","signature":"5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ","operation":{"type":"110","data":{"version":"1.1.43","action":"in_progress"},"signature":"2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst"},"reqId":1508776783276850},"senderClient":"Node1","op":"PROPAGATE"}']}, 'Node2')
2017-10-23 16:39:43,317 | DEBUG    | node.py              (1272) | unpackNodeMsg | Node4 processing a batch BATCH{'signature': None, 'messages': ['{"op":"PREPREPARE","discarded":1,"stateRootHash":null,"viewNo":0,"ppTime":1508776783,"digest":"ba63b43e76c787646532438003bbbef000ca7dee04421bed8a9d59d3f3e09bd8","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1508776783249274]],"ppSeqNo":1,"ledgerId":2,"txnRootHash":null,"instId":1}', '{"request":{"identifier":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv","signature":"5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ","operation":{"type":"110","data":{"version":"1.1.43","action":"in_progress"},"signature":"2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst"},"reqId":1508776783276850},"senderClient":"Node1","op":"PROPAGATE"}']}
2017-10-23 16:39:43,318 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: PREPREPARE{'instId': 1, 'txnRootHash': None, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1508776783249274]], 'digest': 'ba63b43e76c787646532438003bbbef000ca7dee04421bed8a9d59d3f3e09bd8', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ledgerId': 2, 'ppSeqNo': 1, 'discarded': 1}
2017-10-23 16:39:43,318 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'instId': 1, 'digest': 'ba63b43e76c787646532438003bbbef000ca7dee04421bed8a9d59d3f3e09bd8', 'txnRootHash': None, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1508776783249274]], 'viewNo': 0, 'op': 'PREPREPARE', 'ppTime': 1508776783, 'stateRootHash': None, 'ledgerId': 2, 'ppSeqNo': 1, 'discarded': 1}, 'Node2')
2017-10-23 16:39:43,318 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox PREPREPARE{'instId': 1, 'txnRootHash': None, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1508776783249274]], 'digest': 'ba63b43e76c787646532438003bbbef000ca7dee04421bed8a9d59d3f3e09bd8', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ledgerId': 2, 'ppSeqNo': 1, 'discarded': 1}
2017-10-23 16:39:43,319 | DEBUG    | node.py              (2236) | verifySignature | Node4 authenticated Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv signature on propagate  request 1508776783276850
2017-10-23 16:39:43,319 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: PROPAGATE{'request': {'operation': {'type': '110', 'signature': '2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'signature': '5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ', 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'reqId': 1508776783276850}, 'senderClient': 'Node1'}
2017-10-23 16:39:43,319 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'request': {'operation': {'type': '110', 'signature': '2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'signature': '5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ', 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'reqId': 1508776783276850}, 'op': 'PROPAGATE', 'senderClient': 'Node1'}, 'Node2')
2017-10-23 16:39:43,319 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox PROPAGATE{'request': {'operation': {'type': '110', 'signature': '2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'signature': '5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ', 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'reqId': 1508776783276850}, 'senderClient': 'Node1'}
2017-10-23 16:39:43,320 | TRACE    | propagator.py        ( 165) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': 'Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t', 'type': '110'}, 'reqId': 1508776783283567, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9'}
2017-10-23 16:39:43,320 | INFO     | propagator.py        ( 148) | propagate | Node4 propagating request ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1508776783283567) from client Node2
2017-10-23 16:39:43,320 | DEBUG    | node.py              (2608) | send | Node4 sending message PROPAGATE{'request': {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': 'Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t', 'type': '110'}, 'reqId': 1508776783283567, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9'}, 'senderClient': 'Node2'} to all recipients: ['Node2', 'Node1', 'Node3']
2017-10-23 16:39:43,320 | DEBUG    | propagator.py        ( 246) | tryForwarding | Node4 not forwarding request SafeRequest: {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': 'Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t', 'type': '110'}, 'reqId': 1508776783283567, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9'} to its replicas since not finalised
2017-10-23 16:39:43,321 | TRACE    | propagator.py        ( 165) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye', 'type': '110'}, 'reqId': 1508776783291152, 'identifier': 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 'signature': '2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ'}
2017-10-23 16:39:43,321 | INFO     | propagator.py        ( 148) | propagate | Node4 propagating request ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1508776783291152) from client Node3
2017-10-23 16:39:43,321 | DEBUG    | node.py              (2608) | send | Node4 sending message PROPAGATE{'request': {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye', 'type': '110'}, 'reqId': 1508776783291152, 'identifier': 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 'signature': '2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ'}, 'senderClient': 'Node3'} to all recipients: ['Node2', 'Node1', 'Node3']
2017-10-23 16:39:43,321 | DEBUG    | propagator.py        ( 246) | tryForwarding | Node4 not forwarding request SafeRequest: {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye', 'type': '110'}, 'reqId': 1508776783291152, 'identifier': 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 'signature': '2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ'} to its replicas since not finalised
2017-10-23 16:39:43,321 | DEBUG    | node.py              (1757) | processPropagate | Node Node4 received propagated request: PROPAGATE{'request': {'operation': {'type': '110', 'signature': '2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'reqId': 1508776783276850, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ'}, 'senderClient': 'Node1'}
2017-10-23 16:39:43,322 | TRACE    | propagator.py        ( 141) | propagate | Node4 already propagated SafeRequest: {'operation': {'type': '110', 'signature': '2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'reqId': 1508776783276850, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ'}
2017-10-23 16:39:43,322 | DEBUG    | propagator.py        ( 214) | forward | Node4 forwarding request ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1508776783276850) to 0 replicas
2017-10-23 16:39:43,322 | DEBUG    | node.py              (1757) | processPropagate | Node Node4 received propagated request: PROPAGATE{'request': {'operation': {'type': '110', 'signature': '2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'signature': '5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ', 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'reqId': 1508776783276850}, 'senderClient': 'Node1'}
2017-10-23 16:39:43,322 | TRACE    | propagator.py        ( 141) | propagate | Node4 already propagated SafeRequest: {'operation': {'type': '110', 'signature': '2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'reqId': 1508776783276850, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ'}
2017-10-23 16:39:43,322 | DEBUG    | propagator.py        ( 246) | tryForwarding | Node4 not forwarding request SafeRequest: {'operation': {'type': '110', 'signature': '2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'reqId': 1508776783276850, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ'} to its replicas since already forwarded
2017-10-23 16:39:43,323 | DEBUG    | batched.py           ( 100) | flushOutBoxes | Node4 batching 2 msgs to Node2 into one transmission
2017-10-23 16:39:43,323 | TRACE    | batched.py           ( 101) | flushOutBoxes |     messages: deque([b'{"request":{"operation":{"data":{"action":"in_progress","version":"1.1.43"},"signature":"Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t","type":"110"},"reqId":1508776783283567,"identifier":"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb","signature":"2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9"},"op":"PROPAGATE","senderClient":"Node2"}', b'{"request":{"operation":{"data":{"action":"in_progress","version":"1.1.43"},"signature":"3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye","type":"110"},"reqId":1508776783291152,"identifier":"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya","signature":"2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ"},"op":"PROPAGATE","senderClient":"Node3"}'])
2017-10-23 16:39:43,323 | TRACE    | batched.py           ( 110) | flushOutBoxes | Node4 sending payload to Node2: b'{"op":"BATCH","signature":null,"messages":["{\\"request\\":{\\"operation\\":{\\"data\\":{\\"action\\":\\"in_progress\\",\\"version\\":\\"1.1.43\\"},\\"signature\\":\\"Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t\\",\\"type\\":\\"110\\"},\\"reqId\\":1508776783283567,\\"identifier\\":\\"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb\\",\\"signature\\":\\"2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"Node2\\"}","{\\"request\\":{\\"operation\\":{\\"data\\":{\\"action\\":\\"in_progress\\",\\"version\\":\\"1.1.43\\"},\\"signature\\":\\"3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye\\",\\"type\\":\\"110\\"},\\"reqId\\":1508776783291152,\\"identifier\\":\\"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya\\",\\"signature\\":\\"2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"Node3\\"}"]}'
2017-10-23 16:39:43,323 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"op":"BATCH","signature":null,"messages":["{\\"request\\":{\\"operation\\":{\\"data\\":{\\"action\\":\\"in_progress\\",\\"version\\":\\"1.1.43\\"},\\"signature\\":\\"Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t\\",\\"type\\":\\"110\\"},\\"reqId\\":1508776783283567,\\"identifier\\":\\"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb\\",\\"signature\\":\\"2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"Node2\\"}","{\\"request\\":{\\"operation\\":{\\"data\\":{\\"action\\":\\"in_progress\\",\\"version\\":\\"1.1.43\\"},\\"signature\\":\\"3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye\\",\\"type\\":\\"110\\"},\\"reqId\\":1508776783291152,\\"identifier\\":\\"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya\\",\\"signature\\":\\"2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"Node3\\"}"]}' to Node2
2017-10-23 16:39:43,326 | DEBUG    | batched.py           ( 100) | flushOutBoxes | Node4 batching 2 msgs to Node1 into one transmission
2017-10-23 16:39:43,327 | TRACE    | batched.py           ( 101) | flushOutBoxes |     messages: deque([b'{"request":{"operation":{"data":{"action":"in_progress","version":"1.1.43"},"signature":"Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t","type":"110"},"reqId":1508776783283567,"identifier":"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb","signature":"2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9"},"op":"PROPAGATE","senderClient":"Node2"}', b'{"request":{"operation":{"data":{"action":"in_progress","version":"1.1.43"},"signature":"3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye","type":"110"},"reqId":1508776783291152,"identifier":"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya","signature":"2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ"},"op":"PROPAGATE","senderClient":"Node3"}'])
2017-10-23 16:39:43,327 | TRACE    | batched.py           ( 110) | flushOutBoxes | Node4 sending payload to Node1: b'{"op":"BATCH","signature":null,"messages":["{\\"request\\":{\\"operation\\":{\\"data\\":{\\"action\\":\\"in_progress\\",\\"version\\":\\"1.1.43\\"},\\"signature\\":\\"Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t\\",\\"type\\":\\"110\\"},\\"reqId\\":1508776783283567,\\"identifier\\":\\"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb\\",\\"signature\\":\\"2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"Node2\\"}","{\\"request\\":{\\"operation\\":{\\"data\\":{\\"action\\":\\"in_progress\\",\\"version\\":\\"1.1.43\\"},\\"signature\\":\\"3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye\\",\\"type\\":\\"110\\"},\\"reqId\\":1508776783291152,\\"identifier\\":\\"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya\\",\\"signature\\":\\"2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"Node3\\"}"]}'
2017-10-23 16:39:43,327 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"op":"BATCH","signature":null,"messages":["{\\"request\\":{\\"operation\\":{\\"data\\":{\\"action\\":\\"in_progress\\",\\"version\\":\\"1.1.43\\"},\\"signature\\":\\"Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t\\",\\"type\\":\\"110\\"},\\"reqId\\":1508776783283567,\\"identifier\\":\\"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb\\",\\"signature\\":\\"2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"Node2\\"}","{\\"request\\":{\\"operation\\":{\\"data\\":{\\"action\\":\\"in_progress\\",\\"version\\":\\"1.1.43\\"},\\"signature\\":\\"3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye\\",\\"type\\":\\"110\\"},\\"reqId\\":1508776783291152,\\"identifier\\":\\"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya\\",\\"signature\\":\\"2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"Node3\\"}"]}' to Node1
2017-10-23 16:39:43,327 | DEBUG    | batched.py           ( 100) | flushOutBoxes | Node4 batching 2 msgs to Node3 into one transmission
2017-10-23 16:39:43,327 | TRACE    | batched.py           ( 101) | flushOutBoxes |     messages: deque([b'{"request":{"operation":{"data":{"action":"in_progress","version":"1.1.43"},"signature":"Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t","type":"110"},"reqId":1508776783283567,"identifier":"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb","signature":"2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9"},"op":"PROPAGATE","senderClient":"Node2"}', b'{"request":{"operation":{"data":{"action":"in_progress","version":"1.1.43"},"signature":"3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye","type":"110"},"reqId":1508776783291152,"identifier":"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya","signature":"2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ"},"op":"PROPAGATE","senderClient":"Node3"}'])
2017-10-23 16:39:43,327 | TRACE    | batched.py           ( 110) | flushOutBoxes | Node4 sending payload to Node3: b'{"op":"BATCH","signature":null,"messages":["{\\"request\\":{\\"operation\\":{\\"data\\":{\\"action\\":\\"in_progress\\",\\"version\\":\\"1.1.43\\"},\\"signature\\":\\"Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t\\",\\"type\\":\\"110\\"},\\"reqId\\":1508776783283567,\\"identifier\\":\\"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb\\",\\"signature\\":\\"2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"Node2\\"}","{\\"request\\":{\\"operation\\":{\\"data\\":{\\"action\\":\\"in_progress\\",\\"version\\":\\"1.1.43\\"},\\"signature\\":\\"3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye\\",\\"type\\":\\"110\\"},\\"reqId\\":1508776783291152,\\"identifier\\":\\"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya\\",\\"signature\\":\\"2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"Node3\\"}"]}'
2017-10-23 16:39:43,327 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"op":"BATCH","signature":null,"messages":["{\\"request\\":{\\"operation\\":{\\"data\\":{\\"action\\":\\"in_progress\\",\\"version\\":\\"1.1.43\\"},\\"signature\\":\\"Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t\\",\\"type\\":\\"110\\"},\\"reqId\\":1508776783283567,\\"identifier\\":\\"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb\\",\\"signature\\":\\"2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"Node2\\"}","{\\"request\\":{\\"operation\\":{\\"data\\":{\\"action\\":\\"in_progress\\",\\"version\\":\\"1.1.43\\"},\\"signature\\":\\"3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye\\",\\"type\\":\\"110\\"},\\"reqId\\":1508776783291152,\\"identifier\\":\\"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya\\",\\"signature\\":\\"2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ\\"},\\"op\\":\\"PROPAGATE\\",\\"senderClient\\":\\"Node3\\"}"]}' to Node3
2017-10-23 16:39:43,328 | DEBUG    | replica.py           ( 823) | processPrePrepare | Node4:1 received PRE-PREPARE(0, 1) from Node2:1 at 958283.690374019
2017-10-23 16:39:43,328 | DEBUG    | replica.py           ( 934) | doPrepare | Node4:1 Sending PREPARE(0, 1) at 958283.69069497
2017-10-23 16:39:43,329 | DEBUG    | replica.py           (2193) | send | Node4:1 sending Prepare
2017-10-23 16:39:43,329 | TRACE    | replica.py           (2194) | send | Node4:1 sending PREPARE{'instId': 1, 'txnRootHash': None, 'digest': 'ba63b43e76c787646532438003bbbef000ca7dee04421bed8a9d59d3f3e09bd8', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ppSeqNo': 1}
2017-10-23 16:39:43,329 | DEBUG    | replica.py           ( 917) | tryCommit | Node4:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 1, 'txnRootHash': None, 'digest': 'ba63b43e76c787646532438003bbbef000ca7dee04421bed8a9d59d3f3e09bd8', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ppSeqNo': 1}
2017-10-23 16:39:43,329 | DEBUG    | replica.py           ( 614) | trackBatches | Node4:1 tracking batch for PREPREPARE{'instId': 1, 'txnRootHash': None, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1508776783249274)], 'digest': 'ba63b43e76c787646532438003bbbef000ca7dee04421bed8a9d59d3f3e09bd8', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ledgerId': 2, 'ppSeqNo': 1, 'discarded': 1} with state root None
2017-10-23 16:39:43,329 | DEBUG    | replica.py           ( 842) | processPrePrepare | Node4:1 processed incoming PRE-PREPARE(0, 1)
2017-10-23 16:39:43,329 | DEBUG    | node.py              (2608) | send | Node4 sending message PREPARE{'instId': 1, 'txnRootHash': None, 'digest': 'ba63b43e76c787646532438003bbbef000ca7dee04421bed8a9d59d3f3e09bd8', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ppSeqNo': 1} to all recipients: ['Node2', 'Node1', 'Node3']
2017-10-23 16:39:43,330 | TRACE    | zstack.py            ( 479) | _receiveFromListener | Node4 got 1 messages through listener
2017-10-23 16:39:43,333 | DEBUG    | node.py              (2236) | verifySignature | Node4 authenticated 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb signature on propagate  request 1508776783283567
2017-10-23 16:39:43,333 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: PROPAGATE{'request': {'operation': {'type': '110', 'signature': 'Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'reqId': 1508776783283567, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9'}, 'senderClient': 'Node2'}
2017-10-23 16:39:43,334 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'request': {'operation': {'type': '110', 'signature': 'Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'reqId': 1508776783283567, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9'}, 'op': 'PROPAGATE', 'senderClient': 'Node2'}, 'Node3')
2017-10-23 16:39:43,334 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox PROPAGATE{'request': {'operation': {'type': '110', 'signature': 'Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'reqId': 1508776783283567, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9'}, 'senderClient': 'Node2'}
2017-10-23 16:39:43,334 | DEBUG    | node.py              (1757) | processPropagate | Node Node4 received propagated request: PROPAGATE{'request': {'operation': {'type': '110', 'signature': 'Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'reqId': 1508776783283567, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9'}, 'senderClient': 'Node2'}
2017-10-23 16:39:43,334 | TRACE    | propagator.py        ( 141) | propagate | Node4 already propagated SafeRequest: {'operation': {'type': '110', 'signature': 'Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'reqId': 1508776783283567, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9'}
2017-10-23 16:39:43,334 | DEBUG    | propagator.py        ( 214) | forward | Node4 forwarding request ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1508776783283567) to 0 replicas
2017-10-23 16:39:43,335 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"instId":1,"txnRootHash":null,"digest":"ba63b43e76c787646532438003bbbef000ca7dee04421bed8a9d59d3f3e09bd8","viewNo":0,"ppTime":1508776783,"stateRootHash":null,"ppSeqNo":1,"op":"PREPARE"}' to Node2
2017-10-23 16:39:43,335 | TRACE    | batched.py           (  96) | flushOutBoxes | Node4 sending msg b'{"instId":1,"txnRootHash":null,"digest":"ba63b43e76c787646532438003bbbef000ca7dee04421bed8a9d59d3f3e09bd8","viewNo":0,"ppTime":1508776783,"stateRootHash":null,"ppSeqNo":1,"op":"PREPARE"}' to Node2
2017-10-23 16:39:43,335 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"instId":1,"txnRootHash":null,"digest":"ba63b43e76c787646532438003bbbef000ca7dee04421bed8a9d59d3f3e09bd8","viewNo":0,"ppTime":1508776783,"stateRootHash":null,"ppSeqNo":1,"op":"PREPARE"}' to Node1
2017-10-23 16:39:43,336 | TRACE    | batched.py           (  96) | flushOutBoxes | Node4 sending msg b'{"instId":1,"txnRootHash":null,"digest":"ba63b43e76c787646532438003bbbef000ca7dee04421bed8a9d59d3f3e09bd8","viewNo":0,"ppTime":1508776783,"stateRootHash":null,"ppSeqNo":1,"op":"PREPARE"}' to Node1
2017-10-23 16:39:43,336 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"instId":1,"txnRootHash":null,"digest":"ba63b43e76c787646532438003bbbef000ca7dee04421bed8a9d59d3f3e09bd8","viewNo":0,"ppTime":1508776783,"stateRootHash":null,"ppSeqNo":1,"op":"PREPARE"}' to Node3
2017-10-23 16:39:43,336 | TRACE    | batched.py           (  96) | flushOutBoxes | Node4 sending msg b'{"instId":1,"txnRootHash":null,"digest":"ba63b43e76c787646532438003bbbef000ca7dee04421bed8a9d59d3f3e09bd8","viewNo":0,"ppTime":1508776783,"stateRootHash":null,"ppSeqNo":1,"op":"PREPARE"}' to Node3
2017-10-23 16:39:43,337 | TRACE    | zstack.py            ( 479) | _receiveFromListener | Node4 got 1 messages through listener
2017-10-23 16:39:43,337 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: BATCH{'signature': None, 'messages': ['{"request":{"identifier":"4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA","signature":"491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm","operation":{"signature":"5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1","type":"110","data":{"version":"1.1.43","action":"in_progress"}},"reqId":1508776783297473},"senderClient":"Node4","op":"PROPAGATE"}', '{"request":{"identifier":"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya","signature":"2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ","operation":{"type":"110","data":{"version":"1.1.43","action":"in_progress"},"signature":"3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye"},"reqId":1508776783291152},"senderClient":"Node3","op":"PROPAGATE"}']}
2017-10-23 16:39:43,337 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"request":{"identifier":"4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA","signature":"491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm","operation":{"signature":"5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1","type":"110","data":{"version":"1.1.43","action":"in_progress"}},"reqId":1508776783297473},"senderClient":"Node4","op":"PROPAGATE"}', '{"request":{"identifier":"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya","signature":"2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ","operation":{"type":"110","data":{"version":"1.1.43","action":"in_progress"},"signature":"3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye"},"reqId":1508776783291152},"senderClient":"Node3","op":"PROPAGATE"}']}, 'Node2')
2017-10-23 16:39:43,337 | DEBUG    | node.py              (1272) | unpackNodeMsg | Node4 processing a batch BATCH{'signature': None, 'messages': ['{"request":{"identifier":"4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA","signature":"491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm","operation":{"signature":"5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1","type":"110","data":{"version":"1.1.43","action":"in_progress"}},"reqId":1508776783297473},"senderClient":"Node4","op":"PROPAGATE"}', '{"request":{"identifier":"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya","signature":"2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ","operation":{"type":"110","data":{"version":"1.1.43","action":"in_progress"},"signature":"3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye"},"reqId":1508776783291152},"senderClient":"Node3","op":"PROPAGATE"}']}
2017-10-23 16:39:43,338 | DEBUG    | node.py              (2236) | verifySignature | Node4 authenticated 4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA signature on propagate  request 1508776783297473
2017-10-23 16:39:43,338 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: PROPAGATE{'request': {'operation': {'type': '110', 'signature': '5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'signature': '491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm', 'identifier': '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 'reqId': 1508776783297473}, 'senderClient': 'Node4'}
2017-10-23 16:39:43,338 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'request': {'operation': {'type': '110', 'signature': '5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'signature': '491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm', 'identifier': '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 'reqId': 1508776783297473}, 'op': 'PROPAGATE', 'senderClient': 'Node4'}, 'Node2')
2017-10-23 16:39:43,338 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox PROPAGATE{'request': {'operation': {'type': '110', 'signature': '5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'signature': '491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm', 'identifier': '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 'reqId': 1508776783297473}, 'senderClient': 'Node4'}
2017-10-23 16:39:43,339 | DEBUG    | node.py              (2236) | verifySignature | Node4 authenticated DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya signature on propagate  request 1508776783291152
2017-10-23 16:39:43,339 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: PROPAGATE{'request': {'operation': {'type': '110', 'signature': '3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'signature': '2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ', 'identifier': 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 'reqId': 1508776783291152}, 'senderClient': 'Node3'}
2017-10-23 16:39:43,339 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'request': {'operation': {'type': '110', 'signature': '3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'signature': '2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ', 'identifier': 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 'reqId': 1508776783291152}, 'op': 'PROPAGATE', 'senderClient': 'Node3'}, 'Node2')
2017-10-23 16:39:43,340 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox PROPAGATE{'request': {'operation': {'type': '110', 'signature': '3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'signature': '2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ', 'identifier': 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 'reqId': 1508776783291152}, 'senderClient': 'Node3'}
2017-10-23 16:39:43,340 | DEBUG    | node.py              (1757) | processPropagate | Node Node4 received propagated request: PROPAGATE{'request': {'operation': {'type': '110', 'signature': '5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'signature': '491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm', 'identifier': '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 'reqId': 1508776783297473}, 'senderClient': 'Node4'}
2017-10-23 16:39:43,340 | TRACE    | propagator.py        ( 165) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'type': '110', 'signature': '5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'reqId': 1508776783297473, 'identifier': '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 'signature': '491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm'}
2017-10-23 16:39:43,340 | INFO     | propagator.py        ( 148) | propagate | Node4 propagating request ('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 1508776783297473) from client Node4
2017-10-23 16:39:43,340 | DEBUG    | node.py              (2608) | send | Node4 sending message PROPAGATE{'request': {'operation': {'type': '110', 'signature': '5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'reqId': 1508776783297473, 'identifier': '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 'signature': '491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm'}, 'senderClient': 'Node4'} to all recipients: ['Node2', 'Node1', 'Node3']
2017-10-23 16:39:43,340 | DEBUG    | propagator.py        ( 214) | forward | Node4 forwarding request ('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 1508776783297473) to 0 replicas
2017-10-23 16:39:43,340 | DEBUG    | node.py              (1757) | processPropagate | Node Node4 received propagated request: PROPAGATE{'request': {'operation': {'type': '110', 'signature': '3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'signature': '2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ', 'identifier': 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 'reqId': 1508776783291152}, 'senderClient': 'Node3'}
2017-10-23 16:39:43,341 | TRACE    | propagator.py        ( 141) | propagate | Node4 already propagated SafeRequest: {'operation': {'type': '110', 'signature': '3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'reqId': 1508776783291152, 'identifier': 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 'signature': '2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ'}
2017-10-23 16:39:43,341 | DEBUG    | propagator.py        ( 214) | forward | Node4 forwarding request ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1508776783291152) to 2 replicas
2017-10-23 16:39:43,341 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"request":{"operation":{"type":"110","signature":"5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1","data":{"action":"in_progress","version":"1.1.43"}},"reqId":1508776783297473,"identifier":"4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA","signature":"491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm"},"op":"PROPAGATE","senderClient":"Node4"}' to Node2
2017-10-23 16:39:43,342 | TRACE    | batched.py           (  96) | flushOutBoxes | Node4 sending msg b'{"request":{"operation":{"type":"110","signature":"5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1","data":{"action":"in_progress","version":"1.1.43"}},"reqId":1508776783297473,"identifier":"4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA","signature":"491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm"},"op":"PROPAGATE","senderClient":"Node4"}' to Node2
2017-10-23 16:39:43,342 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"request":{"operation":{"type":"110","signature":"5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1","data":{"action":"in_progress","version":"1.1.43"}},"reqId":1508776783297473,"identifier":"4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA","signature":"491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm"},"op":"PROPAGATE","senderClient":"Node4"}' to Node1
2017-10-23 16:39:43,343 | TRACE    | batched.py           (  96) | flushOutBoxes | Node4 sending msg b'{"request":{"operation":{"type":"110","signature":"5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1","data":{"action":"in_progress","version":"1.1.43"}},"reqId":1508776783297473,"identifier":"4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA","signature":"491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm"},"op":"PROPAGATE","senderClient":"Node4"}' to Node1
2017-10-23 16:39:43,343 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"request":{"operation":{"type":"110","signature":"5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1","data":{"action":"in_progress","version":"1.1.43"}},"reqId":1508776783297473,"identifier":"4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA","signature":"491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm"},"op":"PROPAGATE","senderClient":"Node4"}' to Node3
2017-10-23 16:39:43,343 | TRACE    | batched.py           (  96) | flushOutBoxes | Node4 sending msg b'{"request":{"operation":{"type":"110","signature":"5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1","data":{"action":"in_progress","version":"1.1.43"}},"reqId":1508776783297473,"identifier":"4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA","signature":"491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm"},"op":"PROPAGATE","senderClient":"Node4"}' to Node3
2017-10-23 16:39:43,344 | TRACE    | zstack.py            ( 479) | _receiveFromListener | Node4 got 1 messages through listener
2017-10-23 16:39:43,345 | DEBUG    | node.py              (2236) | verifySignature | Node4 authenticated 4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA signature on propagate  request 1508776783297473
2017-10-23 16:39:43,345 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: PROPAGATE{'request': {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1', 'type': '110'}, 'reqId': 1508776783297473, 'identifier': '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 'signature': '491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm'}, 'senderClient': 'Node4'}
2017-10-23 16:39:43,345 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'request': {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1', 'type': '110'}, 'reqId': 1508776783297473, 'identifier': '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 'signature': '491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm'}, 'op': 'PROPAGATE', 'senderClient': 'Node4'}, 'Node3')
2017-10-23 16:39:43,345 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox PROPAGATE{'request': {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1', 'type': '110'}, 'reqId': 1508776783297473, 'identifier': '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 'signature': '491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm'}, 'senderClient': 'Node4'}
2017-10-23 16:39:43,345 | DEBUG    | node.py              (1757) | processPropagate | Node Node4 received propagated request: PROPAGATE{'request': {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1', 'type': '110'}, 'reqId': 1508776783297473, 'identifier': '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 'signature': '491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm'}, 'senderClient': 'Node4'}
2017-10-23 16:39:43,346 | TRACE    | propagator.py        ( 141) | propagate | Node4 already propagated SafeRequest: {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1', 'type': '110'}, 'reqId': 1508776783297473, 'identifier': '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 'signature': '491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm'}
2017-10-23 16:39:43,346 | DEBUG    | propagator.py        ( 246) | tryForwarding | Node4 not forwarding request SafeRequest: {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1', 'type': '110'}, 'reqId': 1508776783297473, 'identifier': '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 'signature': '491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm'} to its replicas since already forwarded
2017-10-23 16:39:43,352 | INFO     | upgrader.py          ( 433) | _sendUpdateRequest | Sending message to control tool: {"version": "1.1.43"}
2017-10-23 16:39:43,352 | INFO     | upgrader.py          ( 452) | _sendUpdateRequest | Waiting 10 minutes for upgrade to be performed
2017-10-23 16:39:43,353 | TRACE    | has_action_queue.py  (  36) | _schedule | 4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA scheduling action partial(_declareTimeoutExceeded) with id 2 to run in 600 seconds
2017-10-23 16:39:43,378 | TRACE    | zstack.py            ( 479) | _receiveFromListener | Node4 got 1 messages through listener
2017-10-23 16:39:43,379 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: BATCH{'signature': None, 'messages': ['{"op":"PREPREPARE","discarded":1,"stateRootHash":null,"viewNo":0,"ppTime":1508776783,"digest":"ff05236fae1ab1810aecdf0ca8845a2d462e6ce45103c3175a158df60390b4a1","reqIdr":[["Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv",1508776783276850]],"ppSeqNo":2,"ledgerId":2,"txnRootHash":null,"instId":1}', '{"request":{"identifier":"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb","signature":"2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9","operation":{"data":{"version":"1.1.43","action":"in_progress"},"signature":"Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t","type":"110"},"reqId":1508776783283567},"senderClient":"Node2","op":"PROPAGATE"}']}
2017-10-23 16:39:43,379 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"op":"PREPREPARE","discarded":1,"stateRootHash":null,"viewNo":0,"ppTime":1508776783,"digest":"ff05236fae1ab1810aecdf0ca8845a2d462e6ce45103c3175a158df60390b4a1","reqIdr":[["Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv",1508776783276850]],"ppSeqNo":2,"ledgerId":2,"txnRootHash":null,"instId":1}', '{"request":{"identifier":"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb","signature":"2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9","operation":{"data":{"version":"1.1.43","action":"in_progress"},"signature":"Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t","type":"110"},"reqId":1508776783283567},"senderClient":"Node2","op":"PROPAGATE"}']}, 'Node2')
2017-10-23 16:39:43,379 | DEBUG    | node.py              (1272) | unpackNodeMsg | Node4 processing a batch BATCH{'signature': None, 'messages': ['{"op":"PREPREPARE","discarded":1,"stateRootHash":null,"viewNo":0,"ppTime":1508776783,"digest":"ff05236fae1ab1810aecdf0ca8845a2d462e6ce45103c3175a158df60390b4a1","reqIdr":[["Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv",1508776783276850]],"ppSeqNo":2,"ledgerId":2,"txnRootHash":null,"instId":1}', '{"request":{"identifier":"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb","signature":"2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9","operation":{"data":{"version":"1.1.43","action":"in_progress"},"signature":"Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t","type":"110"},"reqId":1508776783283567},"senderClient":"Node2","op":"PROPAGATE"}']}
2017-10-23 16:39:43,379 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: PREPREPARE{'instId': 1, 'txnRootHash': None, 'reqIdr': [['Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1508776783276850]], 'digest': 'ff05236fae1ab1810aecdf0ca8845a2d462e6ce45103c3175a158df60390b4a1', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ledgerId': 2, 'ppSeqNo': 2, 'discarded': 1}
2017-10-23 16:39:43,380 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'instId': 1, 'digest': 'ff05236fae1ab1810aecdf0ca8845a2d462e6ce45103c3175a158df60390b4a1', 'txnRootHash': None, 'reqIdr': [['Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1508776783276850]], 'viewNo': 0, 'op': 'PREPREPARE', 'ppTime': 1508776783, 'stateRootHash': None, 'ledgerId': 2, 'ppSeqNo': 2, 'discarded': 1}, 'Node2')
2017-10-23 16:39:43,380 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox PREPREPARE{'instId': 1, 'txnRootHash': None, 'reqIdr': [['Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1508776783276850]], 'digest': 'ff05236fae1ab1810aecdf0ca8845a2d462e6ce45103c3175a158df60390b4a1', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ledgerId': 2, 'ppSeqNo': 2, 'discarded': 1}
2017-10-23 16:39:43,381 | DEBUG    | node.py              (2236) | verifySignature | Node4 authenticated 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb signature on propagate  request 1508776783283567
2017-10-23 16:39:43,381 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: PROPAGATE{'request': {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': 'Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t', 'type': '110'}, 'signature': '2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9', 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'reqId': 1508776783283567}, 'senderClient': 'Node2'}
2017-10-23 16:39:43,381 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'request': {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': 'Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t', 'type': '110'}, 'signature': '2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9', 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'reqId': 1508776783283567}, 'op': 'PROPAGATE', 'senderClient': 'Node2'}, 'Node2')
2017-10-23 16:39:43,381 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox PROPAGATE{'request': {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': 'Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t', 'type': '110'}, 'signature': '2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9', 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'reqId': 1508776783283567}, 'senderClient': 'Node2'}
2017-10-23 16:39:43,381 | DEBUG    | node.py              (1757) | processPropagate | Node Node4 received propagated request: PROPAGATE{'request': {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': 'Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t', 'type': '110'}, 'signature': '2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9', 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'reqId': 1508776783283567}, 'senderClient': 'Node2'}
2017-10-23 16:39:43,381 | TRACE    | propagator.py        ( 141) | propagate | Node4 already propagated SafeRequest: {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': 'Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t', 'type': '110'}, 'reqId': 1508776783283567, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9'}
2017-10-23 16:39:43,382 | DEBUG    | propagator.py        ( 246) | tryForwarding | Node4 not forwarding request SafeRequest: {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': 'Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t', 'type': '110'}, 'reqId': 1508776783283567, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9'} to its replicas since already forwarded
2017-10-23 16:39:43,382 | DEBUG    | replica.py           ( 823) | processPrePrepare | Node4:1 received PRE-PREPARE(0, 2) from Node2:1 at 958283.744481475
2017-10-23 16:39:43,383 | DEBUG    | replica.py           ( 934) | doPrepare | Node4:1 Sending PREPARE(0, 2) at 958283.745065493
2017-10-23 16:39:43,383 | DEBUG    | replica.py           (2193) | send | Node4:1 sending Prepare
2017-10-23 16:39:43,383 | TRACE    | replica.py           (2194) | send | Node4:1 sending PREPARE{'instId': 1, 'txnRootHash': None, 'digest': 'ff05236fae1ab1810aecdf0ca8845a2d462e6ce45103c3175a158df60390b4a1', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ppSeqNo': 2}
2017-10-23 16:39:43,383 | DEBUG    | replica.py           ( 917) | tryCommit | Node4:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 1, 'txnRootHash': None, 'digest': 'ff05236fae1ab1810aecdf0ca8845a2d462e6ce45103c3175a158df60390b4a1', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ppSeqNo': 2}
2017-10-23 16:39:43,383 | DEBUG    | replica.py           ( 614) | trackBatches | Node4:1 tracking batch for PREPREPARE{'instId': 1, 'txnRootHash': None, 'reqIdr': [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1508776783276850)], 'digest': 'ff05236fae1ab1810aecdf0ca8845a2d462e6ce45103c3175a158df60390b4a1', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ledgerId': 2, 'ppSeqNo': 2, 'discarded': 1} with state root None
2017-10-23 16:39:43,383 | DEBUG    | replica.py           ( 842) | processPrePrepare | Node4:1 processed incoming PRE-PREPARE(0, 2)
2017-10-23 16:39:43,383 | DEBUG    | node.py              (2608) | send | Node4 sending message PREPARE{'instId': 1, 'txnRootHash': None, 'digest': 'ff05236fae1ab1810aecdf0ca8845a2d462e6ce45103c3175a158df60390b4a1', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ppSeqNo': 2} to all recipients: ['Node2', 'Node1', 'Node3']
2017-10-23 16:39:43,384 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"instId":1,"txnRootHash":null,"digest":"ff05236fae1ab1810aecdf0ca8845a2d462e6ce45103c3175a158df60390b4a1","viewNo":0,"ppTime":1508776783,"stateRootHash":null,"ppSeqNo":2,"op":"PREPARE"}' to Node2
2017-10-23 16:39:43,384 | TRACE    | batched.py           (  96) | flushOutBoxes | Node4 sending msg b'{"instId":1,"txnRootHash":null,"digest":"ff05236fae1ab1810aecdf0ca8845a2d462e6ce45103c3175a158df60390b4a1","viewNo":0,"ppTime":1508776783,"stateRootHash":null,"ppSeqNo":2,"op":"PREPARE"}' to Node2
2017-10-23 16:39:43,384 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"instId":1,"txnRootHash":null,"digest":"ff05236fae1ab1810aecdf0ca8845a2d462e6ce45103c3175a158df60390b4a1","viewNo":0,"ppTime":1508776783,"stateRootHash":null,"ppSeqNo":2,"op":"PREPARE"}' to Node1
2017-10-23 16:39:43,384 | TRACE    | batched.py           (  96) | flushOutBoxes | Node4 sending msg b'{"instId":1,"txnRootHash":null,"digest":"ff05236fae1ab1810aecdf0ca8845a2d462e6ce45103c3175a158df60390b4a1","viewNo":0,"ppTime":1508776783,"stateRootHash":null,"ppSeqNo":2,"op":"PREPARE"}' to Node1
2017-10-23 16:39:43,385 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"instId":1,"txnRootHash":null,"digest":"ff05236fae1ab1810aecdf0ca8845a2d462e6ce45103c3175a158df60390b4a1","viewNo":0,"ppTime":1508776783,"stateRootHash":null,"ppSeqNo":2,"op":"PREPARE"}' to Node3
2017-10-23 16:39:43,385 | TRACE    | batched.py           (  96) | flushOutBoxes | Node4 sending msg b'{"instId":1,"txnRootHash":null,"digest":"ff05236fae1ab1810aecdf0ca8845a2d462e6ce45103c3175a158df60390b4a1","viewNo":0,"ppTime":1508776783,"stateRootHash":null,"ppSeqNo":2,"op":"PREPARE"}' to Node3
2017-10-23 16:39:43,406 | TRACE    | zstack.py            ( 479) | _receiveFromListener | Node4 got 3 messages through listener
2017-10-23 16:39:43,407 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: PREPREPARE{'instId': 1, 'txnRootHash': None, 'reqIdr': [['8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1508776783283567], ['DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1508776783291152]], 'digest': 'a0486a08c5c70381395ab1b2c037173865d93144b55f0acf8018472e0852c94f', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ledgerId': 2, 'ppSeqNo': 3, 'discarded': 2}
2017-10-23 16:39:43,407 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'instId': 1, 'digest': 'a0486a08c5c70381395ab1b2c037173865d93144b55f0acf8018472e0852c94f', 'txnRootHash': None, 'reqIdr': [['8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1508776783283567], ['DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1508776783291152]], 'viewNo': 0, 'op': 'PREPREPARE', 'ppTime': 1508776783, 'stateRootHash': None, 'ledgerId': 2, 'ppSeqNo': 3, 'discarded': 2}, 'Node2')
2017-10-23 16:39:43,407 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox PREPREPARE{'instId': 1, 'txnRootHash': None, 'reqIdr': [['8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1508776783283567], ['DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1508776783291152]], 'digest': 'a0486a08c5c70381395ab1b2c037173865d93144b55f0acf8018472e0852c94f', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ledgerId': 2, 'ppSeqNo': 3, 'discarded': 2}
2017-10-23 16:39:43,407 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: BATCH{'signature': None, 'messages': ['{"viewNo":0,"stateRootHash":null,"instId":1,"digest":"ba63b43e76c787646532438003bbbef000ca7dee04421bed8a9d59d3f3e09bd8","ppTime":1508776783,"ppSeqNo":1,"txnRootHash":null,"op":"PREPARE"}', '{"request":{"reqId":1508776783291152,"identifier":"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya","operation":{"type":"110","data":{"action":"in_progress","version":"1.1.43"},"signature":"3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye"},"signature":"2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ"},"op":"PROPAGATE","senderClient":"Node3"}']}
2017-10-23 16:39:43,407 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"viewNo":0,"stateRootHash":null,"instId":1,"digest":"ba63b43e76c787646532438003bbbef000ca7dee04421bed8a9d59d3f3e09bd8","ppTime":1508776783,"ppSeqNo":1,"txnRootHash":null,"op":"PREPARE"}', '{"request":{"reqId":1508776783291152,"identifier":"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya","operation":{"type":"110","data":{"action":"in_progress","version":"1.1.43"},"signature":"3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye"},"signature":"2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ"},"op":"PROPAGATE","senderClient":"Node3"}']}, 'Node3')
2017-10-23 16:39:43,408 | DEBUG    | node.py              (1272) | unpackNodeMsg | Node4 processing a batch BATCH{'signature': None, 'messages': ['{"viewNo":0,"stateRootHash":null,"instId":1,"digest":"ba63b43e76c787646532438003bbbef000ca7dee04421bed8a9d59d3f3e09bd8","ppTime":1508776783,"ppSeqNo":1,"txnRootHash":null,"op":"PREPARE"}', '{"request":{"reqId":1508776783291152,"identifier":"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya","operation":{"type":"110","data":{"action":"in_progress","version":"1.1.43"},"signature":"3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye"},"signature":"2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ"},"op":"PROPAGATE","senderClient":"Node3"}']}
2017-10-23 16:39:43,408 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: PREPARE{'instId': 1, 'txnRootHash': None, 'digest': 'ba63b43e76c787646532438003bbbef000ca7dee04421bed8a9d59d3f3e09bd8', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ppSeqNo': 1}
2017-10-23 16:39:43,408 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'instId': 1, 'txnRootHash': None, 'digest': 'ba63b43e76c787646532438003bbbef000ca7dee04421bed8a9d59d3f3e09bd8', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ppSeqNo': 1, 'op': 'PREPARE'}, 'Node3')
2017-10-23 16:39:43,408 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox PREPARE{'instId': 1, 'txnRootHash': None, 'digest': 'ba63b43e76c787646532438003bbbef000ca7dee04421bed8a9d59d3f3e09bd8', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ppSeqNo': 1}
2017-10-23 16:39:43,409 | DEBUG    | node.py              (2236) | verifySignature | Node4 authenticated DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya signature on propagate  request 1508776783291152
2017-10-23 16:39:43,409 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: PROPAGATE{'request': {'operation': {'type': '110', 'signature': '3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'reqId': 1508776783291152, 'identifier': 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 'signature': '2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ'}, 'senderClient': 'Node3'}
2017-10-23 16:39:43,409 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'request': {'operation': {'type': '110', 'signature': '3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'reqId': 1508776783291152, 'identifier': 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 'signature': '2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ'}, 'op': 'PROPAGATE', 'senderClient': 'Node3'}, 'Node3')
2017-10-23 16:39:43,409 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox PROPAGATE{'request': {'operation': {'type': '110', 'signature': '3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'reqId': 1508776783291152, 'identifier': 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 'signature': '2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ'}, 'senderClient': 'Node3'}
2017-10-23 16:39:43,410 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: PREPREPARE{'instId': 1, 'txnRootHash': None, 'reqIdr': [['4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 1508776783297473]], 'digest': 'ecaad1f67e5ea4e31550422968a6afe13989bce1235dfd542c1487e6209fedac', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ledgerId': 2, 'ppSeqNo': 4, 'discarded': 1}
2017-10-23 16:39:43,410 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'instId': 1, 'digest': 'ecaad1f67e5ea4e31550422968a6afe13989bce1235dfd542c1487e6209fedac', 'txnRootHash': None, 'reqIdr': [['4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 1508776783297473]], 'viewNo': 0, 'op': 'PREPREPARE', 'ppTime': 1508776783, 'stateRootHash': None, 'ledgerId': 2, 'ppSeqNo': 4, 'discarded': 1}, 'Node2')
2017-10-23 16:39:43,410 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox PREPREPARE{'instId': 1, 'txnRootHash': None, 'reqIdr': [['4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 1508776783297473]], 'digest': 'ecaad1f67e5ea4e31550422968a6afe13989bce1235dfd542c1487e6209fedac', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ledgerId': 2, 'ppSeqNo': 4, 'discarded': 1}
2017-10-23 16:39:43,410 | DEBUG    | node.py              (1757) | processPropagate | Node Node4 received propagated request: PROPAGATE{'request': {'operation': {'type': '110', 'signature': '3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'reqId': 1508776783291152, 'identifier': 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 'signature': '2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ'}, 'senderClient': 'Node3'}
2017-10-23 16:39:43,414 | TRACE    | propagator.py        ( 141) | propagate | Node4 already propagated SafeRequest: {'operation': {'type': '110', 'signature': '3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'reqId': 1508776783291152, 'identifier': 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 'signature': '2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ'}
2017-10-23 16:39:43,414 | DEBUG    | propagator.py        ( 246) | tryForwarding | Node4 not forwarding request SafeRequest: {'operation': {'type': '110', 'signature': '3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye', 'data': {'action': 'in_progress', 'version': '1.1.43'}}, 'reqId': 1508776783291152, 'identifier': 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 'signature': '2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ'} to its replicas since already forwarded
2017-10-23 16:39:43,415 | DEBUG    | replica.py           ( 823) | processPrePrepare | Node4:1 received PRE-PREPARE(0, 3) from Node2:1 at 958283.777075669
2017-10-23 16:39:43,415 | DEBUG    | replica.py           ( 934) | doPrepare | Node4:1 Sending PREPARE(0, 3) at 958283.777472758
2017-10-23 16:39:43,415 | DEBUG    | replica.py           (2193) | send | Node4:1 sending Prepare
2017-10-23 16:39:43,415 | TRACE    | replica.py           (2194) | send | Node4:1 sending PREPARE{'instId': 1, 'txnRootHash': None, 'digest': 'a0486a08c5c70381395ab1b2c037173865d93144b55f0acf8018472e0852c94f', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ppSeqNo': 3}
2017-10-23 16:39:43,416 | DEBUG    | replica.py           ( 917) | tryCommit | Node4:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 1, 'txnRootHash': None, 'digest': 'a0486a08c5c70381395ab1b2c037173865d93144b55f0acf8018472e0852c94f', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ppSeqNo': 3}
2017-10-23 16:39:43,416 | DEBUG    | replica.py           ( 614) | trackBatches | Node4:1 tracking batch for PREPREPARE{'instId': 1, 'txnRootHash': None, 'reqIdr': [('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1508776783283567), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1508776783291152)], 'digest': 'a0486a08c5c70381395ab1b2c037173865d93144b55f0acf8018472e0852c94f', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ledgerId': 2, 'ppSeqNo': 3, 'discarded': 2} with state root None
2017-10-23 16:39:43,416 | DEBUG    | replica.py           ( 842) | processPrePrepare | Node4:1 processed incoming PRE-PREPARE(0, 3)
2017-10-23 16:39:43,416 | DEBUG    | replica.py           ( 953) | doCommit | Node4:1 Sending COMMIT(0, 1) at 958283.778184031
2017-10-23 16:39:43,416 | DEBUG    | replica.py           (2193) | send | Node4:1 sending Commit
2017-10-23 16:39:43,416 | TRACE    | replica.py           (2194) | send | Node4:1 sending COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 1}
2017-10-23 16:39:43,416 | DEBUG    | replica.py           ( 929) | tryOrder | Node4:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 1} commits where f is 1
2017-10-23 16:39:43,416 | DEBUG    | replica.py           ( 876) | processPrepare | Node4:1 processed incoming PREPARE (0, 1)
2017-10-23 16:39:43,416 | DEBUG    | replica.py           ( 823) | processPrePrepare | Node4:1 received PRE-PREPARE(0, 4) from Node2:1 at 958283.778708814
2017-10-23 16:39:43,417 | DEBUG    | replica.py           ( 934) | doPrepare | Node4:1 Sending PREPARE(0, 4) at 958283.778957582
2017-10-23 16:39:43,417 | DEBUG    | replica.py           (2193) | send | Node4:1 sending Prepare
2017-10-23 16:39:43,417 | TRACE    | replica.py           (2194) | send | Node4:1 sending PREPARE{'instId': 1, 'txnRootHash': None, 'digest': 'ecaad1f67e5ea4e31550422968a6afe13989bce1235dfd542c1487e6209fedac', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ppSeqNo': 4}
2017-10-23 16:39:43,417 | DEBUG    | replica.py           ( 917) | tryCommit | Node4:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 1, 'txnRootHash': None, 'digest': 'ecaad1f67e5ea4e31550422968a6afe13989bce1235dfd542c1487e6209fedac', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ppSeqNo': 4}
2017-10-23 16:39:43,417 | DEBUG    | replica.py           ( 614) | trackBatches | Node4:1 tracking batch for PREPREPARE{'instId': 1, 'txnRootHash': None, 'reqIdr': [('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 1508776783297473)], 'digest': 'ecaad1f67e5ea4e31550422968a6afe13989bce1235dfd542c1487e6209fedac', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ledgerId': 2, 'ppSeqNo': 4, 'discarded': 1} with state root None
2017-10-23 16:39:43,417 | DEBUG    | replica.py           ( 842) | processPrePrepare | Node4:1 processed incoming PRE-PREPARE(0, 4)
2017-10-23 16:39:43,417 | DEBUG    | node.py              (2608) | send | Node4 sending message PREPARE{'instId': 1, 'txnRootHash': None, 'digest': 'a0486a08c5c70381395ab1b2c037173865d93144b55f0acf8018472e0852c94f', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ppSeqNo': 3} to all recipients: ['Node2', 'Node1', 'Node3']
2017-10-23 16:39:43,418 | DEBUG    | node.py              (2608) | send | Node4 sending message COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 1} to all recipients: ['Node2', 'Node1', 'Node3']
2017-10-23 16:39:43,418 | DEBUG    | node.py              (2608) | send | Node4 sending message PREPARE{'instId': 1, 'txnRootHash': None, 'digest': 'ecaad1f67e5ea4e31550422968a6afe13989bce1235dfd542c1487e6209fedac', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ppSeqNo': 4} to all recipients: ['Node2', 'Node1', 'Node3']
2017-10-23 16:39:43,439 | DEBUG    | batched.py           ( 100) | flushOutBoxes | Node4 batching 3 msgs to Node2 into one transmission
2017-10-23 16:39:43,439 | TRACE    | batched.py           ( 101) | flushOutBoxes |     messages: deque([b'{"instId":1,"txnRootHash":null,"digest":"a0486a08c5c70381395ab1b2c037173865d93144b55f0acf8018472e0852c94f","viewNo":0,"ppTime":1508776783,"stateRootHash":null,"ppSeqNo":3,"op":"PREPARE"}', b'{"instId":1,"viewNo":0,"ppSeqNo":1,"op":"COMMIT"}', b'{"instId":1,"txnRootHash":null,"digest":"ecaad1f67e5ea4e31550422968a6afe13989bce1235dfd542c1487e6209fedac","viewNo":0,"ppTime":1508776783,"stateRootHash":null,"ppSeqNo":4,"op":"PREPARE"}'])
2017-10-23 16:39:43,439 | TRACE    | batched.py           ( 110) | flushOutBoxes | Node4 sending payload to Node2: b'{"op":"BATCH","signature":null,"messages":["{\\"instId\\":1,\\"txnRootHash\\":null,\\"digest\\":\\"a0486a08c5c70381395ab1b2c037173865d93144b55f0acf8018472e0852c94f\\",\\"viewNo\\":0,\\"ppTime\\":1508776783,\\"stateRootHash\\":null,\\"ppSeqNo\\":3,\\"op\\":\\"PREPARE\\"}","{\\"instId\\":1,\\"viewNo\\":0,\\"ppSeqNo\\":1,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"txnRootHash\\":null,\\"digest\\":\\"ecaad1f67e5ea4e31550422968a6afe13989bce1235dfd542c1487e6209fedac\\",\\"viewNo\\":0,\\"ppTime\\":1508776783,\\"stateRootHash\\":null,\\"ppSeqNo\\":4,\\"op\\":\\"PREPARE\\"}"]}'
2017-10-23 16:39:43,439 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"op":"BATCH","signature":null,"messages":["{\\"instId\\":1,\\"txnRootHash\\":null,\\"digest\\":\\"a0486a08c5c70381395ab1b2c037173865d93144b55f0acf8018472e0852c94f\\",\\"viewNo\\":0,\\"ppTime\\":1508776783,\\"stateRootHash\\":null,\\"ppSeqNo\\":3,\\"op\\":\\"PREPARE\\"}","{\\"instId\\":1,\\"viewNo\\":0,\\"ppSeqNo\\":1,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"txnRootHash\\":null,\\"digest\\":\\"ecaad1f67e5ea4e31550422968a6afe13989bce1235dfd542c1487e6209fedac\\",\\"viewNo\\":0,\\"ppTime\\":1508776783,\\"stateRootHash\\":null,\\"ppSeqNo\\":4,\\"op\\":\\"PREPARE\\"}"]}' to Node2
2017-10-23 16:39:43,440 | DEBUG    | batched.py           ( 100) | flushOutBoxes | Node4 batching 3 msgs to Node1 into one transmission
2017-10-23 16:39:43,440 | TRACE    | batched.py           ( 101) | flushOutBoxes |     messages: deque([b'{"instId":1,"txnRootHash":null,"digest":"a0486a08c5c70381395ab1b2c037173865d93144b55f0acf8018472e0852c94f","viewNo":0,"ppTime":1508776783,"stateRootHash":null,"ppSeqNo":3,"op":"PREPARE"}', b'{"instId":1,"viewNo":0,"ppSeqNo":1,"op":"COMMIT"}', b'{"instId":1,"txnRootHash":null,"digest":"ecaad1f67e5ea4e31550422968a6afe13989bce1235dfd542c1487e6209fedac","viewNo":0,"ppTime":1508776783,"stateRootHash":null,"ppSeqNo":4,"op":"PREPARE"}'])
2017-10-23 16:39:43,440 | TRACE    | batched.py           ( 110) | flushOutBoxes | Node4 sending payload to Node1: b'{"op":"BATCH","signature":null,"messages":["{\\"instId\\":1,\\"txnRootHash\\":null,\\"digest\\":\\"a0486a08c5c70381395ab1b2c037173865d93144b55f0acf8018472e0852c94f\\",\\"viewNo\\":0,\\"ppTime\\":1508776783,\\"stateRootHash\\":null,\\"ppSeqNo\\":3,\\"op\\":\\"PREPARE\\"}","{\\"instId\\":1,\\"viewNo\\":0,\\"ppSeqNo\\":1,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"txnRootHash\\":null,\\"digest\\":\\"ecaad1f67e5ea4e31550422968a6afe13989bce1235dfd542c1487e6209fedac\\",\\"viewNo\\":0,\\"ppTime\\":1508776783,\\"stateRootHash\\":null,\\"ppSeqNo\\":4,\\"op\\":\\"PREPARE\\"}"]}'
2017-10-23 16:39:43,440 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"op":"BATCH","signature":null,"messages":["{\\"instId\\":1,\\"txnRootHash\\":null,\\"digest\\":\\"a0486a08c5c70381395ab1b2c037173865d93144b55f0acf8018472e0852c94f\\",\\"viewNo\\":0,\\"ppTime\\":1508776783,\\"stateRootHash\\":null,\\"ppSeqNo\\":3,\\"op\\":\\"PREPARE\\"}","{\\"instId\\":1,\\"viewNo\\":0,\\"ppSeqNo\\":1,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"txnRootHash\\":null,\\"digest\\":\\"ecaad1f67e5ea4e31550422968a6afe13989bce1235dfd542c1487e6209fedac\\",\\"viewNo\\":0,\\"ppTime\\":1508776783,\\"stateRootHash\\":null,\\"ppSeqNo\\":4,\\"op\\":\\"PREPARE\\"}"]}' to Node1
2017-10-23 16:39:43,440 | DEBUG    | batched.py           ( 100) | flushOutBoxes | Node4 batching 3 msgs to Node3 into one transmission
2017-10-23 16:39:43,440 | TRACE    | batched.py           ( 101) | flushOutBoxes |     messages: deque([b'{"instId":1,"txnRootHash":null,"digest":"a0486a08c5c70381395ab1b2c037173865d93144b55f0acf8018472e0852c94f","viewNo":0,"ppTime":1508776783,"stateRootHash":null,"ppSeqNo":3,"op":"PREPARE"}', b'{"instId":1,"viewNo":0,"ppSeqNo":1,"op":"COMMIT"}', b'{"instId":1,"txnRootHash":null,"digest":"ecaad1f67e5ea4e31550422968a6afe13989bce1235dfd542c1487e6209fedac","viewNo":0,"ppTime":1508776783,"stateRootHash":null,"ppSeqNo":4,"op":"PREPARE"}'])
2017-10-23 16:39:43,441 | TRACE    | batched.py           ( 110) | flushOutBoxes | Node4 sending payload to Node3: b'{"op":"BATCH","signature":null,"messages":["{\\"instId\\":1,\\"txnRootHash\\":null,\\"digest\\":\\"a0486a08c5c70381395ab1b2c037173865d93144b55f0acf8018472e0852c94f\\",\\"viewNo\\":0,\\"ppTime\\":1508776783,\\"stateRootHash\\":null,\\"ppSeqNo\\":3,\\"op\\":\\"PREPARE\\"}","{\\"instId\\":1,\\"viewNo\\":0,\\"ppSeqNo\\":1,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"txnRootHash\\":null,\\"digest\\":\\"ecaad1f67e5ea4e31550422968a6afe13989bce1235dfd542c1487e6209fedac\\",\\"viewNo\\":0,\\"ppTime\\":1508776783,\\"stateRootHash\\":null,\\"ppSeqNo\\":4,\\"op\\":\\"PREPARE\\"}"]}'
2017-10-23 16:39:43,441 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"op":"BATCH","signature":null,"messages":["{\\"instId\\":1,\\"txnRootHash\\":null,\\"digest\\":\\"a0486a08c5c70381395ab1b2c037173865d93144b55f0acf8018472e0852c94f\\",\\"viewNo\\":0,\\"ppTime\\":1508776783,\\"stateRootHash\\":null,\\"ppSeqNo\\":3,\\"op\\":\\"PREPARE\\"}","{\\"instId\\":1,\\"viewNo\\":0,\\"ppSeqNo\\":1,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"txnRootHash\\":null,\\"digest\\":\\"ecaad1f67e5ea4e31550422968a6afe13989bce1235dfd542c1487e6209fedac\\",\\"viewNo\\":0,\\"ppTime\\":1508776783,\\"stateRootHash\\":null,\\"ppSeqNo\\":4,\\"op\\":\\"PREPARE\\"}"]}' to Node3
2017-10-23 16:39:43,441 | TRACE    | zstack.py            ( 479) | _receiveFromListener | Node4 got 5 messages through listener
2017-10-23 16:39:43,442 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 1}
2017-10-23 16:39:43,442 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'instId': 1, 'op': 'COMMIT', 'ppSeqNo': 1, 'viewNo': 0}, 'Node2')
2017-10-23 16:39:43,455 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 1}
2017-10-23 16:39:43,455 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 1}
2017-10-23 16:39:43,456 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'instId': 1, 'viewNo': 0, 'ppSeqNo': 1, 'op': 'COMMIT'}, 'Node3')
2017-10-23 16:39:43,456 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 1}
2017-10-23 16:39:43,456 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node1: BATCH{'signature': None, 'messages': ['{"senderClient":"Node2","request":{"reqId":1508776783283567,"signature":"2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9","operation":{"data":{"version":"1.1.43","action":"in_progress"},"signature":"Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t","type":"110"},"identifier":"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb"},"op":"PROPAGATE"}', '{"senderClient":"Node3","request":{"reqId":1508776783291152,"signature":"2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ","operation":{"data":{"version":"1.1.43","action":"in_progress"},"signature":"3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye","type":"110"},"identifier":"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya"},"op":"PROPAGATE"}', '{"senderClient":"Node1","request":{"reqId":1508776783276850,"signature":"5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ","operation":{"data":{"version":"1.1.43","action":"in_progress"},"signature":"2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst","type":"110"},"identifier":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv"},"op":"PROPAGATE"}', '{"senderClient":"Node4","request":{"reqId":1508776783297473,"signature":"491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm","operation":{"data":{"version":"1.1.43","action":"in_progress"},"signature":"5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1","type":"110"},"identifier":"4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA"},"op":"PROPAGATE"}']}
2017-10-23 16:39:43,456 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"senderClient":"Node2","request":{"reqId":1508776783283567,"signature":"2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9","operation":{"data":{"version":"1.1.43","action":"in_progress"},"signature":"Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t","type":"110"},"identifier":"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb"},"op":"PROPAGATE"}', '{"senderClient":"Node3","request":{"reqId":1508776783291152,"signature":"2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ","operation":{"data":{"version":"1.1.43","action":"in_progress"},"signature":"3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye","type":"110"},"identifier":"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya"},"op":"PROPAGATE"}', '{"senderClient":"Node1","request":{"reqId":1508776783276850,"signature":"5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ","operation":{"data":{"version":"1.1.43","action":"in_progress"},"signature":"2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst","type":"110"},"identifier":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv"},"op":"PROPAGATE"}', '{"senderClient":"Node4","request":{"reqId":1508776783297473,"signature":"491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm","operation":{"data":{"version":"1.1.43","action":"in_progress"},"signature":"5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1","type":"110"},"identifier":"4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA"},"op":"PROPAGATE"}']}, 'Node1')
2017-10-23 16:39:43,456 | DEBUG    | node.py              (1272) | unpackNodeMsg | Node4 processing a batch BATCH{'signature': None, 'messages': ['{"senderClient":"Node2","request":{"reqId":1508776783283567,"signature":"2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9","operation":{"data":{"version":"1.1.43","action":"in_progress"},"signature":"Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t","type":"110"},"identifier":"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb"},"op":"PROPAGATE"}', '{"senderClient":"Node3","request":{"reqId":1508776783291152,"signature":"2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ","operation":{"data":{"version":"1.1.43","action":"in_progress"},"signature":"3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye","type":"110"},"identifier":"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya"},"op":"PROPAGATE"}', '{"senderClient":"Node1","request":{"reqId":1508776783276850,"signature":"5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ","operation":{"data":{"version":"1.1.43","action":"in_progress"},"signature":"2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst","type":"110"},"identifier":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv"},"op":"PROPAGATE"}', '{"senderClient":"Node4","request":{"reqId":1508776783297473,"signature":"491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm","operation":{"data":{"version":"1.1.43","action":"in_progress"},"signature":"5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1","type":"110"},"identifier":"4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA"},"op":"PROPAGATE"}']}
2017-10-23 16:39:43,457 | DEBUG    | node.py              (2236) | verifySignature | Node4 authenticated 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb signature on propagate  request 1508776783283567
2017-10-23 16:39:43,457 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node1: PROPAGATE{'request': {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': 'Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t', 'type': '110'}, 'reqId': 1508776783283567, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9'}, 'senderClient': 'Node2'}
2017-10-23 16:39:43,457 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'request': {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': 'Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t', 'type': '110'}, 'reqId': 1508776783283567, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9'}, 'op': 'PROPAGATE', 'senderClient': 'Node2'}, 'Node1')
2017-10-23 16:39:43,457 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox PROPAGATE{'request': {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': 'Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t', 'type': '110'}, 'reqId': 1508776783283567, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9'}, 'senderClient': 'Node2'}
2017-10-23 16:39:43,458 | DEBUG    | node.py              (2236) | verifySignature | Node4 authenticated DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya signature on propagate  request 1508776783291152
2017-10-23 16:39:43,458 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node1: PROPAGATE{'request': {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye', 'type': '110'}, 'reqId': 1508776783291152, 'identifier': 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 'signature': '2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ'}, 'senderClient': 'Node3'}
2017-10-23 16:39:43,458 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'request': {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye', 'type': '110'}, 'reqId': 1508776783291152, 'identifier': 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 'signature': '2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ'}, 'op': 'PROPAGATE', 'senderClient': 'Node3'}, 'Node1')
2017-10-23 16:39:43,458 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox PROPAGATE{'request': {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye', 'type': '110'}, 'reqId': 1508776783291152, 'identifier': 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 'signature': '2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ'}, 'senderClient': 'Node3'}
2017-10-23 16:39:43,461 | DEBUG    | node.py              (2236) | verifySignature | Node4 authenticated Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv signature on propagate  request 1508776783276850
2017-10-23 16:39:43,461 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node1: PROPAGATE{'request': {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst', 'type': '110'}, 'reqId': 1508776783276850, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ'}, 'senderClient': 'Node1'}
2017-10-23 16:39:43,461 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'request': {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst', 'type': '110'}, 'reqId': 1508776783276850, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ'}, 'op': 'PROPAGATE', 'senderClient': 'Node1'}, 'Node1')
2017-10-23 16:39:43,461 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox PROPAGATE{'request': {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst', 'type': '110'}, 'reqId': 1508776783276850, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ'}, 'senderClient': 'Node1'}
2017-10-23 16:39:43,465 | DEBUG    | node.py              (2236) | verifySignature | Node4 authenticated 4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA signature on propagate  request 1508776783297473
2017-10-23 16:39:43,465 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node1: PROPAGATE{'request': {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1', 'type': '110'}, 'reqId': 1508776783297473, 'identifier': '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 'signature': '491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm'}, 'senderClient': 'Node4'}
2017-10-23 16:39:43,465 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'request': {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1', 'type': '110'}, 'reqId': 1508776783297473, 'identifier': '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 'signature': '491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm'}, 'op': 'PROPAGATE', 'senderClient': 'Node4'}, 'Node1')
2017-10-23 16:39:43,465 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox PROPAGATE{'request': {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1', 'type': '110'}, 'reqId': 1508776783297473, 'identifier': '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 'signature': '491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm'}, 'senderClient': 'Node4'}
2017-10-23 16:39:43,465 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 2}
2017-10-23 16:39:43,466 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'instId': 1, 'op': 'COMMIT', 'ppSeqNo': 2, 'viewNo': 0}, 'Node2')
2017-10-23 16:39:43,466 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 2}
2017-10-23 16:39:43,466 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: BATCH{'signature': None, 'messages': ['{"viewNo":0,"stateRootHash":null,"instId":1,"digest":"ff05236fae1ab1810aecdf0ca8845a2d462e6ce45103c3175a158df60390b4a1","ppTime":1508776783,"ppSeqNo":2,"txnRootHash":null,"op":"PREPARE"}', '{"viewNo":0,"op":"COMMIT","instId":1,"ppSeqNo":2}', '{"viewNo":0,"stateRootHash":null,"instId":1,"digest":"a0486a08c5c70381395ab1b2c037173865d93144b55f0acf8018472e0852c94f","ppTime":1508776783,"ppSeqNo":3,"txnRootHash":null,"op":"PREPARE"}', '{"viewNo":0,"stateRootHash":null,"instId":1,"digest":"ecaad1f67e5ea4e31550422968a6afe13989bce1235dfd542c1487e6209fedac","ppTime":1508776783,"ppSeqNo":4,"txnRootHash":null,"op":"PREPARE"}']}
2017-10-23 16:39:43,470 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"viewNo":0,"stateRootHash":null,"instId":1,"digest":"ff05236fae1ab1810aecdf0ca8845a2d462e6ce45103c3175a158df60390b4a1","ppTime":1508776783,"ppSeqNo":2,"txnRootHash":null,"op":"PREPARE"}', '{"viewNo":0,"op":"COMMIT","instId":1,"ppSeqNo":2}', '{"viewNo":0,"stateRootHash":null,"instId":1,"digest":"a0486a08c5c70381395ab1b2c037173865d93144b55f0acf8018472e0852c94f","ppTime":1508776783,"ppSeqNo":3,"txnRootHash":null,"op":"PREPARE"}', '{"viewNo":0,"stateRootHash":null,"instId":1,"digest":"ecaad1f67e5ea4e31550422968a6afe13989bce1235dfd542c1487e6209fedac","ppTime":1508776783,"ppSeqNo":4,"txnRootHash":null,"op":"PREPARE"}']}, 'Node3')
2017-10-23 16:39:43,470 | DEBUG    | node.py              (1272) | unpackNodeMsg | Node4 processing a batch BATCH{'signature': None, 'messages': ['{"viewNo":0,"stateRootHash":null,"instId":1,"digest":"ff05236fae1ab1810aecdf0ca8845a2d462e6ce45103c3175a158df60390b4a1","ppTime":1508776783,"ppSeqNo":2,"txnRootHash":null,"op":"PREPARE"}', '{"viewNo":0,"op":"COMMIT","instId":1,"ppSeqNo":2}', '{"viewNo":0,"stateRootHash":null,"instId":1,"digest":"a0486a08c5c70381395ab1b2c037173865d93144b55f0acf8018472e0852c94f","ppTime":1508776783,"ppSeqNo":3,"txnRootHash":null,"op":"PREPARE"}', '{"viewNo":0,"stateRootHash":null,"instId":1,"digest":"ecaad1f67e5ea4e31550422968a6afe13989bce1235dfd542c1487e6209fedac","ppTime":1508776783,"ppSeqNo":4,"txnRootHash":null,"op":"PREPARE"}']}
2017-10-23 16:39:43,470 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: PREPARE{'instId': 1, 'txnRootHash': None, 'digest': 'ff05236fae1ab1810aecdf0ca8845a2d462e6ce45103c3175a158df60390b4a1', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ppSeqNo': 2}
2017-10-23 16:39:43,471 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'instId': 1, 'txnRootHash': None, 'digest': 'ff05236fae1ab1810aecdf0ca8845a2d462e6ce45103c3175a158df60390b4a1', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ppSeqNo': 2, 'op': 'PREPARE'}, 'Node3')
2017-10-23 16:39:43,471 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox PREPARE{'instId': 1, 'txnRootHash': None, 'digest': 'ff05236fae1ab1810aecdf0ca8845a2d462e6ce45103c3175a158df60390b4a1', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ppSeqNo': 2}
2017-10-23 16:39:43,471 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 2}
2017-10-23 16:39:43,471 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'instId': 1, 'viewNo': 0, 'ppSeqNo': 2, 'op': 'COMMIT'}, 'Node3')
2017-10-23 16:39:43,471 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 2}
2017-10-23 16:39:43,471 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: PREPARE{'instId': 1, 'txnRootHash': None, 'digest': 'a0486a08c5c70381395ab1b2c037173865d93144b55f0acf8018472e0852c94f', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ppSeqNo': 3}
2017-10-23 16:39:43,471 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'instId': 1, 'txnRootHash': None, 'digest': 'a0486a08c5c70381395ab1b2c037173865d93144b55f0acf8018472e0852c94f', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ppSeqNo': 3, 'op': 'PREPARE'}, 'Node3')
2017-10-23 16:39:43,474 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox PREPARE{'instId': 1, 'txnRootHash': None, 'digest': 'a0486a08c5c70381395ab1b2c037173865d93144b55f0acf8018472e0852c94f', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ppSeqNo': 3}
2017-10-23 16:39:43,476 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: PREPARE{'instId': 1, 'txnRootHash': None, 'digest': 'ecaad1f67e5ea4e31550422968a6afe13989bce1235dfd542c1487e6209fedac', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ppSeqNo': 4}
2017-10-23 16:39:43,478 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'instId': 1, 'txnRootHash': None, 'digest': 'ecaad1f67e5ea4e31550422968a6afe13989bce1235dfd542c1487e6209fedac', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ppSeqNo': 4, 'op': 'PREPARE'}, 'Node3')
2017-10-23 16:39:43,478 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox PREPARE{'instId': 1, 'txnRootHash': None, 'digest': 'ecaad1f67e5ea4e31550422968a6afe13989bce1235dfd542c1487e6209fedac', 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ppSeqNo': 4}
2017-10-23 16:39:43,479 | DEBUG    | node.py              (1757) | processPropagate | Node Node4 received propagated request: PROPAGATE{'request': {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': 'Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t', 'type': '110'}, 'reqId': 1508776783283567, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9'}, 'senderClient': 'Node2'}
2017-10-23 16:39:43,479 | TRACE    | propagator.py        ( 141) | propagate | Node4 already propagated SafeRequest: {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': 'Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t', 'type': '110'}, 'reqId': 1508776783283567, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9'}
2017-10-23 16:39:43,479 | DEBUG    | propagator.py        ( 246) | tryForwarding | Node4 not forwarding request SafeRequest: {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': 'Dks1Pr6BmhDsXb6MSBjY8LWB3T75Vq4FmxLBCQ2WKcCEbJynEWVCAsRVtUD4ySQ9YtfQ2pudUqdZrenKG1pDp4t', 'type': '110'}, 'reqId': 1508776783283567, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '2JqpzmCvtM75Ztqho1rT1uXgRiGRRekMBdNvnmRLaTQiCp49VUqkUPbAuEN72jrknYMMEdqw1WboEt8VqAZgBeM9'} to its replicas since already forwarded
2017-10-23 16:39:43,479 | DEBUG    | node.py              (1757) | processPropagate | Node Node4 received propagated request: PROPAGATE{'request': {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye', 'type': '110'}, 'reqId': 1508776783291152, 'identifier': 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 'signature': '2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ'}, 'senderClient': 'Node3'}
2017-10-23 16:39:43,479 | TRACE    | propagator.py        ( 141) | propagate | Node4 already propagated SafeRequest: {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye', 'type': '110'}, 'reqId': 1508776783291152, 'identifier': 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 'signature': '2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ'}
2017-10-23 16:39:43,480 | DEBUG    | propagator.py        ( 246) | tryForwarding | Node4 not forwarding request SafeRequest: {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '3c8jf6puNxFzNue4T8tieLz3pgrV9Fwy7YzMPCwzxvcvg7dcDg5G5etKBLRfRvPZwsskdAQoiDN6FE4p3bk78nye', 'type': '110'}, 'reqId': 1508776783291152, 'identifier': 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 'signature': '2RjpR7hZSyM22qXRJ4XJJYTPHLBEH1VLfyWpBbqrE1DdXdyeKj4ZgbyYHc15MQAc4xSLf5tAt7xw8PGrfkbzTyoZ'} to its replicas since already forwarded
2017-10-23 16:39:43,480 | DEBUG    | node.py              (1757) | processPropagate | Node Node4 received propagated request: PROPAGATE{'request': {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst', 'type': '110'}, 'reqId': 1508776783276850, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ'}, 'senderClient': 'Node1'}
2017-10-23 16:39:43,480 | TRACE    | propagator.py        ( 141) | propagate | Node4 already propagated SafeRequest: {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst', 'type': '110'}, 'reqId': 1508776783276850, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ'}
2017-10-23 16:39:43,480 | DEBUG    | propagator.py        ( 246) | tryForwarding | Node4 not forwarding request SafeRequest: {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '2zdZQsQbLqj36VwSsKSqyqHcSvbdBdhEEfLNBdLJw881HqghGouHmS9kH1fGi5sHHB78BiqCbc7aHecYeQnCfKst', 'type': '110'}, 'reqId': 1508776783276850, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '5xBsMjDLGCBDmpgmQYn6M1PC4qkJqrfhnJ7HUNPuKFL5PdDgjvPNPCDoEPPzRXZkPBj7mphYggsP7ssu1vehz5fQ'} to its replicas since already forwarded
2017-10-23 16:39:43,480 | DEBUG    | node.py              (1757) | processPropagate | Node Node4 received propagated request: PROPAGATE{'request': {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1', 'type': '110'}, 'reqId': 1508776783297473, 'identifier': '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 'signature': '491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm'}, 'senderClient': 'Node4'}
2017-10-23 16:39:43,480 | TRACE    | propagator.py        ( 141) | propagate | Node4 already propagated SafeRequest: {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1', 'type': '110'}, 'reqId': 1508776783297473, 'identifier': '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 'signature': '491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm'}
2017-10-23 16:39:43,480 | DEBUG    | propagator.py        ( 246) | tryForwarding | Node4 not forwarding request SafeRequest: {'operation': {'data': {'action': 'in_progress', 'version': '1.1.43'}, 'signature': '5ccXKVaDFvn8fXzegsJKTzUE5VCr7rvLPdaGnESRmiYLsvzkgHkUzAGnTTFSEs3HS4KoE6rqp8qYqSPyRrV5BYr1', 'type': '110'}, 'reqId': 1508776783297473, 'identifier': '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 'signature': '491jKNaBNsWxZfUsgQzK71eMKeGsq76UxEpT19KnGAVBxyuPk5HUdkeK4WyAgLwV1ysXC9aK77W9omosDBZRqSJm'} to its replicas since already forwarded
2017-10-23 16:39:43,481 | DEBUG    | replica.py           ( 894) | processCommit | Node4:1 received COMMIT(0, 1) from Node2:1
2017-10-23 16:39:43,481 | DEBUG    | replica.py           ( 929) | tryOrder | Node4:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 1} commits where f is 1
2017-10-23 16:39:43,481 | DEBUG    | replica.py           ( 905) | processCommit | Node4:1 processed incoming COMMIT(0, 1)
2017-10-23 16:39:43,482 | DEBUG    | replica.py           ( 894) | processCommit | Node4:1 received COMMIT(0, 1) from Node3:1
2017-10-23 16:39:43,482 | TRACE    | replica.py           ( 925) | tryOrder | Node4:1 returning request to node
2017-10-23 16:39:43,482 | DEBUG    | replica.py           (1444) | doOrder | Node4:1 ordering COMMIT (0, 1)
2017-10-23 16:39:43,482 | DEBUG    | replica.py           ( 323) | last_ordered_3pc | Node4:1 set last ordered as (0, 1)
2017-10-23 16:39:43,488 | DEBUG    | replica.py           (2193) | send | Node4:1 sending Ordered
2017-10-23 16:39:43,488 | TRACE    | replica.py           (2194) | send | Node4:1 sending ORDERED{'instId': 1, 'txnRootHash': None, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1508776783249274)], 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ledgerId': 2, 'ppSeqNo': 1}
2017-10-23 16:39:43,488 | INFO     | replica.py           (1483) | order_3pc_key | Node4:1 ordered batch request, view no 0, ppSeqNo 1, ledger 2, state root None, txn root None, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1508776783249274)], discarded []
2017-10-23 16:39:43,488 | DEBUG    | replica.py           (1556) | _newCheckpointState | Node4:1 adding new checkpoint state for (1, 100)
2017-10-23 16:39:43,488 | DEBUG    | replica.py           ( 905) | processCommit | Node4:1 processed incoming COMMIT(0, 1)
2017-10-23 16:39:43,488 | DEBUG    | replica.py           ( 894) | processCommit | Node4:1 received COMMIT(0, 2) from Node2:1
2017-10-23 16:39:43,488 | DEBUG    | replica.py           ( 929) | tryOrder | Node4:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 2} commits where f is 1
2017-10-23 16:39:43,488 | DEBUG    | replica.py           ( 905) | processCommit | Node4:1 processed incoming COMMIT(0, 2)
2017-10-23 16:39:43,489 | DEBUG    | replica.py           ( 953) | doCommit | Node4:1 Sending COMMIT(0, 2) at 958283.850915896
2017-10-23 16:39:43,489 | DEBUG    | replica.py           (2193) | send | Node4:1 sending Commit
2017-10-23 16:39:43,489 | TRACE    | replica.py           (2194) | send | Node4:1 sending COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 2}
2017-10-23 16:39:43,489 | DEBUG    | replica.py           ( 929) | tryOrder | Node4:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 2} commits where f is 1
2017-10-23 16:39:43,489 | DEBUG    | replica.py           ( 876) | processPrepare | Node4:1 processed incoming PREPARE (0, 2)
2017-10-23 16:39:43,489 | DEBUG    | replica.py           ( 894) | processCommit | Node4:1 received COMMIT(0, 2) from Node3:1
2017-10-23 16:39:43,489 | TRACE    | replica.py           ( 925) | tryOrder | Node4:1 returning request to node
2017-10-23 16:39:43,489 | DEBUG    | replica.py           (1444) | doOrder | Node4:1 ordering COMMIT (0, 2)
2017-10-23 16:39:43,489 | DEBUG    | replica.py           ( 323) | last_ordered_3pc | Node4:1 set last ordered as (0, 2)
2017-10-23 16:39:43,490 | DEBUG    | replica.py           (2193) | send | Node4:1 sending Ordered
2017-10-23 16:39:43,490 | TRACE    | replica.py           (2194) | send | Node4:1 sending ORDERED{'instId': 1, 'txnRootHash': None, 'reqIdr': [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1508776783276850)], 'viewNo': 0, 'ppTime': 1508776783, 'stateRootHash': None, 'ledgerId': 2, 'ppSeqNo': 2}
2017-10-23 16:39:43,490 | INFO     | replica.py           (1483) | order_3pc_key | Node4:1 ordered batch request, view no 0, ppSeqNo 2, ledger 2, state root None, txn root None, requests ordered [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1508776783276850)], discarded []
2017-10-23 16:39:43,490 | DEBUG    | replica.py           ( 905) | processCommit | Node4:1 processed incoming COMMIT(0, 2)
2017-10-23 16:39:43,490 | DEBUG    | replica.py           ( 953) | doCommit | Node4:1 Sending COMMIT(0, 3) at 958283.852610133
2017-10-23 16:39:43,490 | DEBUG    | replica.py           (2193) | send | Node4:1 sending Commit
2017-10-23 16:39:43,491 | TRACE    | replica.py           (2194) | send | Node4:1 sending COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 3}
2017-10-23 16:39:43,491 | DEBUG    | replica.py           ( 929) | tryOrder | Node4:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 3} commits where f is 1
2017-10-23 16:39:43,491 | DEBUG    | replica.py           ( 876) | processPrepare | Node4:1 processed incoming PREPARE (0, 3)
2017-10-23 16:39:43,491 | DEBUG    | replica.py           ( 953) | doCommit | Node4:1 Sending COMMIT(0, 4) at 958283.853171592
2017-10-23 16:39:43,491 | DEBUG    | replica.py           (2193) | send | Node4:1 sending Commit
2017-10-23 16:39:43,491 | TRACE    | replica.py           (2194) | send | Node4:1 sending COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 4}
2017-10-23 16:39:43,491 | DEBUG    | replica.py           ( 929) | tryOrder | Node4:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 4} commits where f is 1
2017-10-23 16:39:43,491 | DEBUG    | replica.py           ( 876) | processPrepare | Node4:1 processed incoming PREPARE (0, 4)
2017-10-23 16:39:43,491 | TRACE    | node.py              (1858) | processOrdered | Node4 got ordered requests from backup replica 1
2017-10-23 16:39:43,492 | DEBUG    | node.py              (2608) | send | Node4 sending message COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 2} to all recipients: ['Node2', 'Node1', 'Node3']
2017-10-23 16:39:43,492 | TRACE    | node.py              (1858) | processOrdered | Node4 got ordered requests from backup replica 1
2017-10-23 16:39:43,492 | DEBUG    | node.py              (2608) | send | Node4 sending message COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 3} to all recipients: ['Node2', 'Node1', 'Node3']
2017-10-23 16:39:43,492 | DEBUG    | node.py              (2608) | send | Node4 sending message COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 4} to all recipients: ['Node2', 'Node1', 'Node3']
2017-10-23 16:39:43,492 | TRACE    | zstack.py            ( 479) | _receiveFromListener | Node4 got 1 messages through listener
2017-10-23 16:39:43,493 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: BATCH{'signature': None, 'messages': ['{"viewNo":0,"op":"COMMIT","instId":1,"ppSeqNo":3}', '{"viewNo":0,"op":"COMMIT","instId":1,"ppSeqNo":4}']}
2017-10-23 16:39:43,493 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"viewNo":0,"op":"COMMIT","instId":1,"ppSeqNo":3}', '{"viewNo":0,"op":"COMMIT","instId":1,"ppSeqNo":4}']}, 'Node3')
2017-10-23 16:39:43,493 | DEBUG    | node.py              (1272) | unpackNodeMsg | Node4 processing a batch BATCH{'signature': None, 'messages': ['{"viewNo":0,"op":"COMMIT","instId":1,"ppSeqNo":3}', '{"viewNo":0,"op":"COMMIT","instId":1,"ppSeqNo":4}']}
2017-10-23 16:39:43,493 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 3}
2017-10-23 16:39:43,493 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'instId': 1, 'viewNo': 0, 'ppSeqNo': 3, 'op': 'COMMIT'}, 'Node3')
2017-10-23 16:39:43,493 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 3}
2017-10-23 16:39:43,493 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node3: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 4}
2017-10-23 16:39:43,494 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'instId': 1, 'viewNo': 0, 'ppSeqNo': 4, 'op': 'COMMIT'}, 'Node3')
2017-10-23 16:39:43,495 | DEBUG    | node.py              (1286) | postToNodeInBox | Node4 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 4}
2017-10-23 16:39:43,495 | DEBUG    | batched.py           ( 100) | flushOutBoxes | Node4 batching 3 msgs to Node2 into one transmission
2017-10-23 16:39:43,495 | TRACE    | batched.py           ( 101) | flushOutBoxes |     messages: deque([b'{"instId":1,"viewNo":0,"ppSeqNo":2,"op":"COMMIT"}', b'{"instId":1,"viewNo":0,"ppSeqNo":3,"op":"COMMIT"}', b'{"instId":1,"viewNo":0,"ppSeqNo":4,"op":"COMMIT"}'])
2017-10-23 16:39:43,496 | TRACE    | batched.py           ( 110) | flushOutBoxes | Node4 sending payload to Node2: b'{"op":"BATCH","signature":null,"messages":["{\\"instId\\":1,\\"viewNo\\":0,\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":0,\\"ppSeqNo\\":3,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":0,\\"ppSeqNo\\":4,\\"op\\":\\"COMMIT\\"}"]}'
2017-10-23 16:39:43,496 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"op":"BATCH","signature":null,"messages":["{\\"instId\\":1,\\"viewNo\\":0,\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":0,\\"ppSeqNo\\":3,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":0,\\"ppSeqNo\\":4,\\"op\\":\\"COMMIT\\"}"]}' to Node2
2017-10-23 16:39:43,496 | DEBUG    | batched.py           ( 100) | flushOutBoxes | Node4 batching 3 msgs to Node1 into one transmission
2017-10-23 16:39:43,496 | TRACE    | batched.py           ( 101) | flushOutBoxes |     messages: deque([b'{"instId":1,"viewNo":0,"ppSeqNo":2,"op":"COMMIT"}', b'{"instId":1,"viewNo":0,"ppSeqNo":3,"op":"COMMIT"}', b'{"instId":1,"viewNo":0,"ppSeqNo":4,"op":"COMMIT"}'])
2017-10-23 16:39:43,496 | TRACE    | batched.py           ( 110) | flushOutBoxes | Node4 sending payload to Node1: b'{"op":"BATCH","signature":null,"messages":["{\\"instId\\":1,\\"viewNo\\":0,\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":0,\\"ppSeqNo\\":3,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":0,\\"ppSeqNo\\":4,\\"op\\":\\"COMMIT\\"}"]}'
2017-10-23 16:39:43,496 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"op":"BATCH","signature":null,"messages":["{\\"instId\\":1,\\"viewNo\\":0,\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":0,\\"ppSeqNo\\":3,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":0,\\"ppSeqNo\\":4,\\"op\\":\\"COMMIT\\"}"]}' to Node1
2017-10-23 16:39:43,496 | DEBUG    | batched.py           ( 100) | flushOutBoxes | Node4 batching 3 msgs to Node3 into one transmission
2017-10-23 16:39:43,497 | TRACE    | batched.py           ( 101) | flushOutBoxes |     messages: deque([b'{"instId":1,"viewNo":0,"ppSeqNo":2,"op":"COMMIT"}', b'{"instId":1,"viewNo":0,"ppSeqNo":3,"op":"COMMIT"}', b'{"instId":1,"viewNo":0,"ppSeqNo":4,"op":"COMMIT"}'])
2017-10-23 16:39:43,497 | TRACE    | batched.py           ( 110) | flushOutBoxes | Node4 sending payload to Node3: b'{"op":"BATCH","signature":null,"messages":["{\\"instId\\":1,\\"viewNo\\":0,\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":0,\\"ppSeqNo\\":3,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":0,\\"ppSeqNo\\":4,\\"op\\":\\"COMMIT\\"}"]}'
2017-10-23 16:39:43,497 | DEBUG    | zstack.py            ( 723) | transmit | Node4 transmitting message b'{"op":"BATCH","signature":null,"messages":["{\\"instId\\":1,\\"viewNo\\":0,\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":0,\\"ppSeqNo\\":3,\\"op\\":\\"COMMIT\\"}","{\\"instId\\":1,\\"viewNo\\":0,\\"ppSeqNo\\":4,\\"op\\":\\"COMMIT\\"}"]}' to Node3
2017-10-23 16:39:43,497 | DEBUG    | replica.py           ( 894) | processCommit | Node4:1 received COMMIT(0, 3) from Node3:1
2017-10-23 16:39:43,502 | DEBUG    | replica.py           ( 929) | tryOrder | Node4:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 3} commits where f is 1
2017-10-23 16:39:43,502 | DEBUG    | replica.py           ( 905) | processCommit | Node4:1 processed incoming COMMIT(0, 3)
2017-10-23 16:39:43,502 | DEBUG    | replica.py           ( 894) | processCommit | Node4:1 received COMMIT(0, 4) from Node3:1
2017-10-23 16:39:43,503 | DEBUG    | replica.py           ( 929) | tryOrder | Node4:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 4} commits where f is 1
2017-10-23 16:39:43,503 | DEBUG    | replica.py           ( 905) | processCommit | Node4:1 processed incoming COMMIT(0, 4)
2017-10-23 16:39:43,503 | TRACE    | zstack.py            ( 479) | _receiveFromListener | Node4 got 1 messages through listener
2017-10-23 16:39:43,503 | DEBUG    | node.py              (1257) | validateNodeMsg | Node4 received node message from Node2: BATCH{'signature': None, 'messages': ['{"ppSeqNo":3,"op":"COMMIT","viewNo":0,"instId":1}', '{"ppSeqNo":4,"op":"COMMIT","viewNo":0,"instId":1}']}
2017-10-23 16:39:43,503 | DEBUG    | node.py              (1219) | handleOneNodeMsg | Node4 msg validated ({'op': 'BATCH', 'signature': None, 'messages': ['{"ppSeqNo":3,"op":"COMMIT","viewNo":0,"instId":1}', '{"ppSeqNo":4,"op":"COMMIT","viewNo":0,"instId":1}