2017-10-11 16:59:20,033 | INFO | log.py ( 79) | setupRaet | Setting RAET log level 2 2017-10-11 16:59:20,046 | DEBUG | start_sovrin_node ( 39) | | You can find logs in /home/sovrin/.sovrin/Node2.log 2017-10-11 16:59:20,047 | DEBUG | start_sovrin_node ( 42) | | Sovrin related env vars: [] 2017-10-11 16:59:21,142 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: git 2017-10-11 16:59:21,161 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: hg 2017-10-11 16:59:21,203 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: svn 2017-10-11 16:59:21,204 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: bzr 2017-10-11 16:59:21,722 | DEBUG | selector_events.py ( 53) | __init__ | Using selector: EpollSelector 2017-10-11 16:59:21,722 | DEBUG | looper.py ( 123) | __init__ | Setting handler for SIGINT 2017-10-11 16:59:21,739 | DEBUG | ledger.py ( 206) | start | Starting ledger... 2017-10-11 16:59:21,740 | DEBUG | file_store.py ( 190) | appendNewLineIfReq | new line check for file: /home/sovrin/.sovrin/data/nodes/Node2/transactions_live/1 2017-10-11 16:59:21,740 | DEBUG | ledger.py ( 78) | recoverTree | Recovering tree from transaction log 2017-10-11 16:59:21,757 | DEBUG | ledger.py ( 93) | recoverTree | Recovered tree in 0.016689850017428398 seconds 2017-10-11 16:59:21,773 | DEBUG | idr_cache.py ( 25) | __init__ | Initializing identity cache Node2 2017-10-11 16:59:21,782 | INFO | node.py (2408) | initStateFromLedger | Node2 found state to be empty, recreating from ledger 2017-10-11 16:59:21,800 | DEBUG | ledger.py ( 206) | start | Starting ledger... 2017-10-11 16:59:21,801 | DEBUG | file_store.py ( 190) | appendNewLineIfReq | new line check for file: /home/sovrin/.sovrin/data/nodes/Node2/pool_transactions_live/1 2017-10-11 16:59:21,801 | DEBUG | ledger.py ( 78) | recoverTree | Recovering tree from transaction log 2017-10-11 16:59:21,814 | DEBUG | ledger.py ( 93) | recoverTree | Recovered tree in 0.012987431604415178 seconds 2017-10-11 16:59:21,815 | INFO | node.py (2408) | initStateFromLedger | Node2 found state to be empty, recreating from ledger 2017-10-11 16:59:21,866 | INFO | plugin_loader.py ( 116) | _load | plugin FirebaseStatsConsumer successfully loaded from module plugin_firebase_stats_consumer 2017-10-11 16:59:21,866 | DEBUG | plugin_loader.py ( 95) | _load | skipping plugin plugin_firebase_stats_consumer[class: typing.Dict<~KT, ~VT>] because it does not have a 'pluginType' attribute 2017-10-11 16:59:21,867 | DEBUG | plugin_loader.py ( 95) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-10-11 16:59:21,867 | DEBUG | plugin_loader.py ( 95) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-10-11 16:59:21,867 | DEBUG | plugin_loader.py ( 95) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-10-11 16:59:21,867 | DEBUG | plugin_loader.py ( 95) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-10-11 16:59:21,868 | DEBUG | has_action_queue.py ( 77) | startRepeating | checkPerformance will be repeating every 60 seconds 2017-10-11 16:59:21,868 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 1 to run in 60 seconds 2017-10-11 16:59:21,868 | INFO | replica.py ( 300) | h | Node2:0 set watermarks as 0 300 2017-10-11 16:59:21,869 | DISPLAY | node.py (1034) | addReplica | Node2 added replica Node2:0 to instance 0 (master) 2017-10-11 16:59:21,869 | INFO | replica.py ( 300) | h | Node2:1 set watermarks as 0 300 2017-10-11 16:59:21,869 | DISPLAY | node.py (1034) | addReplica | Node2 added replica Node2:1 to instance 1 (backup) 2017-10-11 16:59:21,869 | DEBUG | has_action_queue.py ( 77) | startRepeating | checkPerformance will be repeating every 10 seconds 2017-10-11 16:59:21,870 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 1 to run in 10 seconds 2017-10-11 16:59:21,870 | DEBUG | has_action_queue.py ( 77) | startRepeating | checkNodeRequestSpike will be repeating every 60 seconds 2017-10-11 16:59:21,870 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 2 to run in 60 seconds 2017-10-11 16:59:21,870 | DEBUG | plugin_helper.py ( 24) | loadPlugins | Plugin loading started to load plugins from basedir: /home/sovrin/.sovrin 2017-10-11 16:59:21,870 | DEBUG | plugin_helper.py ( 33) | loadPlugins | Plugin directory created at: /home/sovrin/.sovrin/plugins 2017-10-11 16:59:21,870 | DEBUG | plugin_helper.py ( 67) | loadPlugins | Total plugins loaded from basedir /home/sovrin/.sovrin are : 0 2017-10-11 16:59:21,871 | DEBUG | node.py ( 325) | __init__ | total plugins loaded in node: 0 2017-10-11 16:59:21,882 | DEBUG | ledger.py ( 206) | start | Starting ledger... 2017-10-11 16:59:21,882 | DEBUG | file_store.py ( 190) | appendNewLineIfReq | new line check for file: /home/sovrin/.sovrin/data/nodes/Node2/config_transactions/1 2017-10-11 16:59:21,882 | DEBUG | ledger.py ( 78) | recoverTree | Recovering tree from transaction log 2017-10-11 16:59:21,883 | DEBUG | ledger.py ( 93) | recoverTree | Recovered tree in 0.0005650781095027924 seconds 2017-10-11 16:59:21,891 | INFO | node.py (2408) | initStateFromLedger | Node2 found state to be empty, recreating from ledger 2017-10-11 16:59:21,891 | DEBUG | motor.py ( 34) | set_status | Node2 changing status from stopped to starting 2017-10-11 16:59:21,891 | DEBUG | ledger.py ( 204) | start | Ledger already started. 2017-10-11 16:59:21,891 | INFO | zstack.py ( 312) | start | Node2 starting with restricted as True and reSetupAuth as True 2017-10-11 16:59:21,892 | DEBUG | authenticator.py ( 31) | start | Starting ZAP at inproc://zeromq.zap.1 2017-10-11 16:59:21,892 | DEBUG | base.py ( 72) | allow | Allowing 0.0.0.0 2017-10-11 16:59:21,892 | DEBUG | base.py ( 112) | configure_curve | Configure curve: *[/home/sovrin/.sovrin/Node2/public_keys] 2017-10-11 16:59:21,893 | DEBUG | zstack.py ( 339) | open | Node2 will bind its listener at 9703 2017-10-11 16:59:21,893 | INFO | stacks.py ( 76) | start | Node2 listening for other nodes at 0.0.0.0:9703 2017-10-11 16:59:21,893 | INFO | zstack.py ( 312) | start | Node2C starting with restricted as False and reSetupAuth as True 2017-10-11 16:59:21,893 | DEBUG | authenticator.py ( 31) | start | Starting ZAP at inproc://zeromq.zap.2 2017-10-11 16:59:21,893 | DEBUG | base.py ( 72) | allow | Allowing 0.0.0.0 2017-10-11 16:59:21,894 | DEBUG | base.py ( 112) | configure_curve | Configure curve: *[*] 2017-10-11 16:59:21,894 | DEBUG | zstack.py ( 339) | open | Node2C will bind its listener at 9704 2017-10-11 16:59:21,894 | INFO | node.py ( 594) | start | Node2 first time running... 2017-10-11 16:59:21,898 | DEBUG | kit_zstack.py ( 96) | connectToMissing | Node2 found the following missing connections: Node1, Node4, Node3 2017-10-11 16:59:21,900 | TRACE | remote.py ( 84) | connect | connecting socket 58 30729648 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 16:59:21,900 | INFO | zstack.py ( 580) | connect | Node2 looking for Node1 at 10.0.0.2:9701 2017-10-11 16:59:21,901 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 will be sending in batch 2017-10-11 16:59:21,902 | TRACE | remote.py ( 84) | connect | connecting socket 60 30833024 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 16:59:21,902 | INFO | zstack.py ( 580) | connect | Node2 looking for Node4 at 10.0.0.5:9707 2017-10-11 16:59:21,902 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 will be sending in batch 2017-10-11 16:59:21,902 | TRACE | remote.py ( 84) | connect | connecting socket 61 30870448 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 16:59:21,903 | INFO | zstack.py ( 580) | connect | Node2 looking for Node3 at 10.0.0.4:9705 2017-10-11 16:59:21,903 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 will be sending in batch 2017-10-11 16:59:21,903 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 16:59:21,917 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'pi' to Node1 2017-10-11 16:59:21,917 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'pi' to Node1 2017-10-11 16:59:21,918 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'pi' to Node4 2017-10-11 16:59:21,918 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'pi' to Node4 2017-10-11 16:59:21,918 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'pi' to Node3 2017-10-11 16:59:21,918 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'pi' to Node3 2017-10-11 16:59:22,010 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 2 messages through listener 2017-10-11 16:59:22,011 | DEBUG | zstack.py ( 652) | handlePingPong | Node2 got ping from Node1 2017-10-11 16:59:22,011 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 will be sending in batch 2017-10-11 16:59:22,011 | DEBUG | zstack.py ( 658) | handlePingPong | Node2 got pong from Node1 2017-10-11 16:59:22,012 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'po' to Node1 2017-10-11 16:59:22,027 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'po' to Node1 2017-10-11 16:59:22,027 | DEBUG | keep_in_touch.py ( 68) | conns | Node2's connections changed from set() to {'Node1'} 2017-10-11 16:59:22,027 | INFO | keep_in_touch.py ( 96) | _connsChanged | Node2 now connected to Node1 2017-10-11 16:59:22,028 | DEBUG | node.py (2593) | send | Node2 sending message LEDGER_STATUS{'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'txnSeqNo': 4, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 0} to 1 recipients: ['Node1'] 2017-10-11 16:59:22,030 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"txnSeqNo":4,"viewNo":null,"ppSeqNo":null,"ledgerId":0,"op":"LEDGER_STATUS","merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"}' to Node1 2017-10-11 16:59:22,031 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"txnSeqNo":4,"viewNo":null,"ppSeqNo":null,"ledgerId":0,"op":"LEDGER_STATUS","merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"}' to Node1 2017-10-11 16:59:22,057 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 16:59:22,058 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node1: LEDGER_STATUS{'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'txnSeqNo': 4, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 0} 2017-10-11 16:59:22,058 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'txnSeqNo': 4, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 0, 'op': 'LEDGER_STATUS', 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'}, 'Node1') 2017-10-11 16:59:22,058 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox LEDGER_STATUS{'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'txnSeqNo': 4, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 0} 2017-10-11 16:59:22,058 | DEBUG | ledger_manager.py ( 244) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'txnSeqNo': 4, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 0} from Node1 2017-10-11 16:59:22,059 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 16:59:22,059 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 16:59:22,682 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 16:59:22,684 | DEBUG | zstack.py ( 652) | handlePingPong | Node2 got ping from Node3 2017-10-11 16:59:22,685 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 will be sending in batch 2017-10-11 16:59:22,686 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'po' to Node3 2017-10-11 16:59:22,687 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'po' to Node3 2017-10-11 16:59:22,706 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 16:59:22,707 | DEBUG | zstack.py ( 658) | handlePingPong | Node2 got pong from Node3 2017-10-11 16:59:22,709 | DEBUG | keep_in_touch.py ( 68) | conns | Node2's connections changed from {'Node1'} to {'Node1', 'Node3'} 2017-10-11 16:59:22,709 | INFO | keep_in_touch.py ( 96) | _connsChanged | Node2 now connected to Node3 2017-10-11 16:59:22,709 | DEBUG | motor.py ( 34) | set_status | Node2 changing status from starting to started_hungry 2017-10-11 16:59:22,709 | DEBUG | node.py ( 918) | checkInstances | Node2 choosing to start election on the basis of count 3 and nodes {'Node1', 'Node3'} 2017-10-11 16:59:22,709 | DEBUG | primary_selector.py ( 74) | get_msgs_for_lagged_nodes | Node2 has no ViewChangeDone message to send for view 0 2017-10-11 16:59:22,710 | DEBUG | node.py ( 879) | send_current_state_to_lagging_node | Node2 sending current state CURRENT_STATE{'primary': [], 'viewNo': 0} to lagged node Node3 2017-10-11 16:59:22,710 | DEBUG | node.py (2593) | send | Node2 sending message CURRENT_STATE{'primary': [], 'viewNo': 0} to 1 recipients: ['Node3'] 2017-10-11 16:59:22,715 | DEBUG | node.py (2593) | send | Node2 sending message LEDGER_STATUS{'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'txnSeqNo': 4, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 0} to 1 recipients: ['Node3'] 2017-10-11 16:59:22,717 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 16:59:22,719 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: LEDGER_STATUS{'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'txnSeqNo': 4, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 0} 2017-10-11 16:59:22,719 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'txnSeqNo': 4, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 0, 'op': 'LEDGER_STATUS', 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'}, 'Node3') 2017-10-11 16:59:22,719 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox LEDGER_STATUS{'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'txnSeqNo': 4, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 0} 2017-10-11 16:59:22,719 | DEBUG | ledger_manager.py ( 244) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'txnSeqNo': 4, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 0} from Node3 2017-10-11 16:59:22,720 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 16:59:22,720 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 16:59:22,720 | DEBUG | ledger_manager.py ( 309) | processLedgerStatus | Node2 found out from {'Node1', 'Node3'} that its ledger of type 0 is latest 2017-10-11 16:59:22,720 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found from ledger status LEDGER_STATUS{'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'txnSeqNo': 4, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 0} that it does not need catchup 2017-10-11 16:59:22,720 | DEBUG | node.py (1537) | preLedgerCatchUp | Node2 going to process any ordered requests before starting catchup. 2017-10-11 16:59:22,720 | DEBUG | replica.py (2130) | _remove_ordered_from_queue | Node2:0 going to remove 0 Ordered messages from outbox 2017-10-11 16:59:22,720 | DEBUG | node.py (1918) | force_process_ordered | Node2 processed 0 Ordered batches for instance 0 before starting catch up 2017-10-11 16:59:22,720 | DEBUG | replica.py (2130) | _remove_ordered_from_queue | Node2:1 going to remove 0 Ordered messages from outbox 2017-10-11 16:59:22,721 | DEBUG | node.py (1918) | force_process_ordered | Node2 processed 0 Ordered batches for instance 1 before starting catch up 2017-10-11 16:59:22,721 | DEBUG | node.py (2451) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-10-11 16:59:22,721 | DEBUG | monitor.py ( 183) | reset | Monitor being reset 2017-10-11 16:59:22,721 | DEBUG | node.py (1547) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 0 2017-10-11 16:59:22,721 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 16:59:22,721 | DEBUG | node.py ( 918) | checkInstances | Node2 choosing to start election on the basis of count 3 and nodes {'Node1', 'Node3'} 2017-10-11 16:59:22,721 | DEBUG | node.py (2593) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node1'] 2017-10-11 16:59:22,722 | DEBUG | node.py ( 832) | _ask_for_ledger_status | Node2 asking Node1 for ledger status of ledger 2 2017-10-11 16:59:22,722 | DEBUG | node.py (2593) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to all recipients: ['Node1', 'Node4', 'Node3'] 2017-10-11 16:59:22,722 | DEBUG | node.py ( 832) | _ask_for_ledger_status | Node2 asking Node2 for ledger status of ledger 2 2017-10-11 16:59:22,722 | DEBUG | node.py (2593) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-10-11 16:59:22,722 | DEBUG | node.py ( 832) | _ask_for_ledger_status | Node2 asking Node3 for ledger status of ledger 2 2017-10-11 16:59:22,722 | DEBUG | node.py (2593) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-10-11 16:59:22,724 | DEBUG | node.py ( 832) | _ask_for_ledger_status | Node2 asking Node4 for ledger status of ledger 2 2017-10-11 16:59:22,725 | DEBUG | ledger_manager.py (1006) | processStashedLedgerStatuses | Node2 going to process 0 stashed ledger statuses for ledger 2 2017-10-11 16:59:22,725 | INFO | ledger_manager.py ( 831) | catchupCompleted | Node2 completed catching up ledger 0, caught up 0 in total 2017-10-11 16:59:22,726 | DEBUG | batched.py ( 89) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-10-11 16:59:22,726 | TRACE | batched.py ( 90) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}']) 2017-10-11 16:59:22,726 | TRACE | batched.py ( 98) | flushOutBoxes | Node2 sending payload to Node1: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH","signature":null}' 2017-10-11 16:59:22,727 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH","signature":null}' to Node1 2017-10-11 16:59:22,727 | DEBUG | batched.py ( 89) | flushOutBoxes | Node2 batching 2 msgs to Node4 into one transmission 2017-10-11 16:59:22,727 | TRACE | batched.py ( 90) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}']) 2017-10-11 16:59:22,727 | TRACE | batched.py ( 98) | flushOutBoxes | Node2 sending payload to Node4: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH","signature":null}' 2017-10-11 16:59:22,727 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH","signature":null}' to Node4 2017-10-11 16:59:22,728 | WARNING | zstack.py ( 704) | transmit | Remote Node4 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 16:59:22,728 | DEBUG | batched.py ( 89) | flushOutBoxes | Node2 batching 4 msgs to Node3 into one transmission 2017-10-11 16:59:22,728 | TRACE | batched.py ( 90) | flushOutBoxes | messages: deque([b'{"primary":[],"op":"CURRENT_STATE","viewNo":0}', b'{"txnSeqNo":4,"viewNo":null,"ppSeqNo":null,"ledgerId":0,"op":"LEDGER_STATUS","merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}']) 2017-10-11 16:59:22,728 | TRACE | batched.py ( 98) | flushOutBoxes | Node2 sending payload to Node3: b'{"messages":["{\\"primary\\":[],\\"op\\":\\"CURRENT_STATE\\",\\"viewNo\\":0}","{\\"txnSeqNo\\":4,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH","signature":null}' 2017-10-11 16:59:22,728 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"messages":["{\\"primary\\":[],\\"op\\":\\"CURRENT_STATE\\",\\"viewNo\\":0}","{\\"txnSeqNo\\":4,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA\\"}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH","signature":null}' to Node3 2017-10-11 16:59:22,759 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 16:59:22,760 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node1: BATCH{'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE"}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-10-11 16:59:22,760 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE"}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE"}'], 'op': 'BATCH', 'signature': None}, 'Node1') 2017-10-11 16:59:22,760 | DEBUG | node.py (1328) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE"}', '{"msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-10-11 16:59:22,760 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2}, 'params': {'ledgerId': 2}} 2017-10-11 16:59:22,760 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}}, 'Node1') 2017-10-11 16:59:22,761 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2}, 'params': {'ledgerId': 2}} 2017-10-11 16:59:22,761 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2}, 'params': {'ledgerId': 2}} 2017-10-11 16:59:22,761 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}}, 'Node1') 2017-10-11 16:59:22,761 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2}, 'params': {'ledgerId': 2}} 2017-10-11 16:59:22,762 | DEBUG | ledger_manager.py ( 244) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2} from Node1 2017-10-11 16:59:22,762 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-10-11 16:59:22,762 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-10-11 16:59:22,762 | DEBUG | ledger_manager.py ( 244) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2} from Node1 2017-10-11 16:59:22,763 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-10-11 16:59:22,763 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-10-11 16:59:22,780 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 16:59:22,781 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: BATCH{'messages': ['{"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}'], 'signature': None} 2017-10-11 16:59:22,781 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}'], 'op': 'BATCH', 'signature': None}, 'Node3') 2017-10-11 16:59:22,781 | DEBUG | node.py (1328) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}'], 'signature': None} 2017-10-11 16:59:22,781 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2}, 'params': {'ledgerId': 2}} 2017-10-11 16:59:22,781 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}}, 'Node3') 2017-10-11 16:59:22,782 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2}, 'params': {'ledgerId': 2}} 2017-10-11 16:59:22,782 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2}, 'params': {'ledgerId': 2}} 2017-10-11 16:59:22,782 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}}, 'Node3') 2017-10-11 16:59:22,782 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2}, 'params': {'ledgerId': 2}} 2017-10-11 16:59:22,783 | DEBUG | ledger_manager.py ( 244) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2} from Node3 2017-10-11 16:59:22,783 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-10-11 16:59:22,783 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-10-11 16:59:22,783 | DEBUG | ledger_manager.py ( 309) | processLedgerStatus | Node2 found out from {'Node1', 'Node3'} that its ledger of type 2 is latest 2017-10-11 16:59:22,783 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found from ledger status LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2} that it does not need catchup 2017-10-11 16:59:22,783 | DEBUG | node.py (1537) | preLedgerCatchUp | Node2 going to process any ordered requests before starting catchup. 2017-10-11 16:59:22,783 | DEBUG | replica.py (2130) | _remove_ordered_from_queue | Node2:0 going to remove 0 Ordered messages from outbox 2017-10-11 16:59:22,784 | DEBUG | node.py (1918) | force_process_ordered | Node2 processed 0 Ordered batches for instance 0 before starting catch up 2017-10-11 16:59:22,784 | DEBUG | replica.py (2130) | _remove_ordered_from_queue | Node2:1 going to remove 0 Ordered messages from outbox 2017-10-11 16:59:22,784 | DEBUG | node.py (1918) | force_process_ordered | Node2 processed 0 Ordered batches for instance 1 before starting catch up 2017-10-11 16:59:22,784 | DEBUG | node.py (2451) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-10-11 16:59:22,784 | DEBUG | monitor.py ( 183) | reset | Monitor being reset 2017-10-11 16:59:22,784 | DEBUG | node.py (1547) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 2 2017-10-11 16:59:22,785 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-10-11 16:59:22,785 | INFO | pool_config.py ( 33) | processLedger | processing config ledger for any POOL_CONFIGs 2017-10-11 16:59:22,785 | INFO | upgrader.py ( 145) | processLedger | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb processing config ledger for any upgrades 2017-10-11 16:59:22,785 | DEBUG | node.py (2593) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node1'] 2017-10-11 16:59:22,786 | DEBUG | node.py ( 832) | _ask_for_ledger_status | Node2 asking Node1 for ledger status of ledger 1 2017-10-11 16:59:22,786 | DEBUG | node.py (2593) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to all recipients: ['Node1', 'Node4', 'Node3'] 2017-10-11 16:59:22,786 | DEBUG | node.py ( 832) | _ask_for_ledger_status | Node2 asking Node2 for ledger status of ledger 1 2017-10-11 16:59:22,786 | DEBUG | node.py (2593) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-10-11 16:59:22,786 | DEBUG | node.py ( 832) | _ask_for_ledger_status | Node2 asking Node3 for ledger status of ledger 1 2017-10-11 16:59:22,786 | DEBUG | node.py (2593) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-10-11 16:59:22,786 | DEBUG | node.py ( 832) | _ask_for_ledger_status | Node2 asking Node4 for ledger status of ledger 1 2017-10-11 16:59:22,787 | DEBUG | ledger_manager.py (1006) | processStashedLedgerStatuses | Node2 going to process 0 stashed ledger statuses for ledger 1 2017-10-11 16:59:22,787 | INFO | ledger_manager.py ( 831) | catchupCompleted | Node2 completed catching up ledger 2, caught up 0 in total 2017-10-11 16:59:22,787 | DEBUG | ledger_manager.py ( 244) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2} from Node3 2017-10-11 16:59:22,787 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-10-11 16:59:22,787 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-10-11 16:59:22,789 | DEBUG | batched.py ( 89) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-10-11 16:59:22,789 | TRACE | batched.py ( 90) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}']) 2017-10-11 16:59:22,789 | TRACE | batched.py ( 98) | flushOutBoxes | Node2 sending payload to Node1: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH","signature":null}' 2017-10-11 16:59:22,789 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH","signature":null}' to Node1 2017-10-11 16:59:22,790 | DEBUG | batched.py ( 89) | flushOutBoxes | Node2 batching 2 msgs to Node4 into one transmission 2017-10-11 16:59:22,790 | TRACE | batched.py ( 90) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}']) 2017-10-11 16:59:22,790 | TRACE | batched.py ( 98) | flushOutBoxes | Node2 sending payload to Node4: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH","signature":null}' 2017-10-11 16:59:22,790 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH","signature":null}' to Node4 2017-10-11 16:59:22,790 | WARNING | zstack.py ( 704) | transmit | Remote Node4 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 16:59:22,790 | DEBUG | batched.py ( 89) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-10-11 16:59:22,790 | TRACE | batched.py ( 90) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}']) 2017-10-11 16:59:22,791 | TRACE | batched.py ( 98) | flushOutBoxes | Node2 sending payload to Node3: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH","signature":null}' 2017-10-11 16:59:22,791 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH","signature":null}' to Node3 2017-10-11 16:59:22,832 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 2 messages through listener 2017-10-11 16:59:22,833 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: BATCH{'messages': ['{"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}'], 'signature': None} 2017-10-11 16:59:22,833 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}'], 'op': 'BATCH', 'signature': None}, 'Node3') 2017-10-11 16:59:22,833 | DEBUG | node.py (1328) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}'], 'signature': None} 2017-10-11 16:59:22,833 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 9, 'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1}, 'params': {'ledgerId': 1}} 2017-10-11 16:59:22,833 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 9, 'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}}, 'Node3') 2017-10-11 16:59:22,833 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 9, 'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1}, 'params': {'ledgerId': 1}} 2017-10-11 16:59:22,834 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 9, 'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1}, 'params': {'ledgerId': 1}} 2017-10-11 16:59:22,834 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 9, 'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}}, 'Node3') 2017-10-11 16:59:22,834 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 9, 'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1}, 'params': {'ledgerId': 1}} 2017-10-11 16:59:22,834 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node1: BATCH{'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}'], 'signature': None} 2017-10-11 16:59:22,834 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}'], 'op': 'BATCH', 'signature': None}, 'Node1') 2017-10-11 16:59:22,834 | DEBUG | node.py (1328) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}'], 'signature': None} 2017-10-11 16:59:22,835 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 16:59:22,835 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}}, 'Node1') 2017-10-11 16:59:22,835 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 16:59:22,835 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 16:59:22,835 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}}, 'Node1') 2017-10-11 16:59:22,836 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 16:59:22,836 | DEBUG | ledger_manager.py ( 244) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'txnSeqNo': 9, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1} from Node3 2017-10-11 16:59:22,836 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 1 of size 9 with 9 2017-10-11 16:59:22,836 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 1 of size 9 with 9 2017-10-11 16:59:22,837 | DEBUG | ledger_manager.py ( 244) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'txnSeqNo': 9, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1} from Node3 2017-10-11 16:59:22,837 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 1 of size 9 with 9 2017-10-11 16:59:22,837 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 1 of size 9 with 9 2017-10-11 16:59:22,838 | DEBUG | node.py (2593) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2}, 'params': {'ledgerId': 2}} to 1 recipients: ['Node1'] 2017-10-11 16:59:22,838 | DEBUG | node.py (2593) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2}, 'params': {'ledgerId': 2}} to 1 recipients: ['Node1'] 2017-10-11 16:59:22,840 | DEBUG | batched.py ( 89) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-10-11 16:59:22,840 | TRACE | batched.py ( 90) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', b'{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}']) 2017-10-11 16:59:22,840 | TRACE | batched.py ( 98) | flushOutBoxes | Node2 sending payload to Node1: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH","signature":null}' 2017-10-11 16:59:22,840 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH","signature":null}' to Node1 2017-10-11 16:59:22,858 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 16:59:22,859 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node1: BATCH{'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE"}', '{"msg":{"ledgerId":1,"merkleRoot":"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-10-11 16:59:22,859 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE"}', '{"msg":{"ledgerId":1,"merkleRoot":"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE"}'], 'op': 'BATCH', 'signature': None}, 'Node1') 2017-10-11 16:59:22,859 | DEBUG | node.py (1328) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"msg":{"ledgerId":1,"merkleRoot":"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE"}', '{"msg":{"ledgerId":1,"merkleRoot":"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-10-11 16:59:22,859 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 9, 'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1}, 'params': {'ledgerId': 1}} 2017-10-11 16:59:22,859 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 9, 'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}}, 'Node1') 2017-10-11 16:59:22,860 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 9, 'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1}, 'params': {'ledgerId': 1}} 2017-10-11 16:59:22,860 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 9, 'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1}, 'params': {'ledgerId': 1}} 2017-10-11 16:59:22,860 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 9, 'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}}, 'Node1') 2017-10-11 16:59:22,860 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 9, 'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1}, 'params': {'ledgerId': 1}} 2017-10-11 16:59:22,861 | DEBUG | ledger_manager.py ( 244) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'txnSeqNo': 9, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1} from Node1 2017-10-11 16:59:22,861 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 1 of size 9 with 9 2017-10-11 16:59:22,861 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 1 of size 9 with 9 2017-10-11 16:59:22,861 | DEBUG | ledger_manager.py ( 309) | processLedgerStatus | Node2 found out from {'Node1', 'Node3'} that its ledger of type 1 is latest 2017-10-11 16:59:22,861 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found from ledger status LEDGER_STATUS{'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'txnSeqNo': 9, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1} that it does not need catchup 2017-10-11 16:59:22,861 | DEBUG | node.py (1537) | preLedgerCatchUp | Node2 going to process any ordered requests before starting catchup. 2017-10-11 16:59:22,861 | DEBUG | replica.py (2130) | _remove_ordered_from_queue | Node2:0 going to remove 0 Ordered messages from outbox 2017-10-11 16:59:22,861 | DEBUG | node.py (1918) | force_process_ordered | Node2 processed 0 Ordered batches for instance 0 before starting catch up 2017-10-11 16:59:22,862 | DEBUG | replica.py (2130) | _remove_ordered_from_queue | Node2:1 going to remove 0 Ordered messages from outbox 2017-10-11 16:59:22,862 | DEBUG | node.py (1918) | force_process_ordered | Node2 processed 0 Ordered batches for instance 1 before starting catch up 2017-10-11 16:59:22,862 | DEBUG | node.py (2451) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-10-11 16:59:22,862 | DEBUG | monitor.py ( 183) | reset | Monitor being reset 2017-10-11 16:59:22,862 | DEBUG | node.py (1547) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 1 2017-10-11 16:59:22,862 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 1 of size 9 with 9 2017-10-11 16:59:22,862 | INFO | ledger_manager.py ( 831) | catchupCompleted | Node2 completed catching up ledger 1, caught up 0 in total 2017-10-11 16:59:22,863 | DEBUG | node.py (1653) | num_txns_caught_up_in_last_catchup | Node2 caught up to 0 txns in the last catchup 2017-10-11 16:59:22,863 | DEBUG | node.py (2451) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-10-11 16:59:22,863 | DEBUG | monitor.py ( 183) | reset | Monitor being reset 2017-10-11 16:59:22,863 | DEBUG | primary_selector.py ( 186) | _hasViewChangeQuorum | Node2 needs 2 ViewChangeDone messages 2017-10-11 16:59:22,863 | DEBUG | node.py (1625) | caught_up_for_current_view | Node2 does not have view change quorum for view 0 2017-10-11 16:59:22,863 | DEBUG | node.py (1608) | is_catchup_needed | Node2 is not caught up for the current view 0 2017-10-11 16:59:22,863 | DEBUG | node.py (1653) | num_txns_caught_up_in_last_catchup | Node2 caught up to 0 txns in the last catchup 2017-10-11 16:59:22,863 | DEBUG | node.py (1611) | is_catchup_needed | Node2 ordered till last prepared certificate 2017-10-11 16:59:22,863 | INFO | node.py (1593) | allLedgersCaughtUp | Node2 does not need any more catchups 2017-10-11 16:59:22,865 | DEBUG | primary_decider.py ( 131) | send | Node2's elector sending VIEW_CHANGE_DONE{'ledgerInfo': [(0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'), (1, 9, '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'name': 'Node1', 'viewNo': 0} 2017-10-11 16:59:22,865 | DEBUG | primary_selector.py ( 186) | _hasViewChangeQuorum | Node2 needs 1 ViewChangeDone messages 2017-10-11 16:59:22,865 | DEBUG | primary_selector.py ( 258) | _startSelection | Node2 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-10-11 16:59:22,866 | DEBUG | ledger_manager.py ( 244) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'txnSeqNo': 9, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1} from Node1 2017-10-11 16:59:22,866 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 1 of size 9 with 9 2017-10-11 16:59:22,868 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 1 of size 9 with 9 2017-10-11 16:59:22,869 | DEBUG | node.py (2593) | send | Node2 sending message VIEW_CHANGE_DONE{'ledgerInfo': [(0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'), (1, 9, '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')], 'name': 'Node1', 'viewNo': 0} to all recipients: ['Node1', 'Node4', 'Node3'] 2017-10-11 16:59:22,878 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,9,"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","op":"VIEW_CHANGE_DONE","viewNo":0}' to Node1 2017-10-11 16:59:22,879 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,9,"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","op":"VIEW_CHANGE_DONE","viewNo":0}' to Node1 2017-10-11 16:59:22,879 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,9,"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","op":"VIEW_CHANGE_DONE","viewNo":0}' to Node4 2017-10-11 16:59:22,879 | WARNING | zstack.py ( 704) | transmit | Remote Node4 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 16:59:22,879 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,9,"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","op":"VIEW_CHANGE_DONE","viewNo":0}' to Node4 2017-10-11 16:59:22,879 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,9,"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","op":"VIEW_CHANGE_DONE","viewNo":0}' to Node3 2017-10-11 16:59:22,883 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,9,"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","op":"VIEW_CHANGE_DONE","viewNo":0}' to Node3 2017-10-11 16:59:22,906 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 2 messages through listener 2017-10-11 16:59:22,907 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node1: BATCH{'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}'], 'signature': None} 2017-10-11 16:59:22,907 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}'], 'op': 'BATCH', 'signature': None}, 'Node1') 2017-10-11 16:59:22,907 | DEBUG | node.py (1328) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}'], 'signature': None} 2017-10-11 16:59:22,908 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 16:59:22,908 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}}, 'Node1') 2017-10-11 16:59:22,908 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 16:59:22,908 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 16:59:22,908 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}}, 'Node1') 2017-10-11 16:59:22,908 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 16:59:22,908 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: BATCH{'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}'], 'signature': None} 2017-10-11 16:59:22,909 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}'], 'op': 'BATCH', 'signature': None}, 'Node3') 2017-10-11 16:59:22,909 | DEBUG | node.py (1328) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":2},"msg_type":"LEDGER_STATUS"}'], 'signature': None} 2017-10-11 16:59:22,909 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 16:59:22,909 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}}, 'Node3') 2017-10-11 16:59:22,909 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 16:59:22,909 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 16:59:22,909 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}}, 'Node3') 2017-10-11 16:59:22,910 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 16:59:22,910 | DEBUG | node.py (2593) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'txnSeqNo': 9, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1}, 'params': {'ledgerId': 1}} to 1 recipients: ['Node1'] 2017-10-11 16:59:22,911 | DEBUG | node.py (2593) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'txnSeqNo': 9, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1}, 'params': {'ledgerId': 1}} to 1 recipients: ['Node1'] 2017-10-11 16:59:22,912 | DEBUG | node.py (2593) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2}, 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-10-11 16:59:22,912 | DEBUG | node.py (2593) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2}, 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-10-11 16:59:22,914 | DEBUG | batched.py ( 89) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-10-11 16:59:22,914 | TRACE | batched.py ( 90) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', b'{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}']) 2017-10-11 16:59:22,914 | TRACE | batched.py ( 98) | flushOutBoxes | Node2 sending payload to Node1: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH","signature":null}' 2017-10-11 16:59:22,915 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH","signature":null}' to Node1 2017-10-11 16:59:22,915 | DEBUG | batched.py ( 89) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-10-11 16:59:22,915 | TRACE | batched.py ( 90) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', b'{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}']) 2017-10-11 16:59:22,916 | TRACE | batched.py ( 98) | flushOutBoxes | Node2 sending payload to Node3: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH","signature":null}' 2017-10-11 16:59:22,916 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}"],"op":"BATCH","signature":null}' to Node3 2017-10-11 16:59:22,948 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 16:59:22,948 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: BATCH{'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}'], 'signature': None} 2017-10-11 16:59:22,948 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}'], 'op': 'BATCH', 'signature': None}, 'Node3') 2017-10-11 16:59:22,949 | DEBUG | node.py (1328) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}', '{"op":"MESSAGE_REQUEST","params":{"ledgerId":1},"msg_type":"LEDGER_STATUS"}'], 'signature': None} 2017-10-11 16:59:22,949 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 16:59:22,949 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}}, 'Node3') 2017-10-11 16:59:22,949 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 16:59:22,950 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 16:59:22,950 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}}, 'Node3') 2017-10-11 16:59:22,950 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 16:59:22,951 | DEBUG | node.py (2593) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'txnSeqNo': 9, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1}, 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-10-11 16:59:22,951 | DEBUG | node.py (2593) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'txnSeqNo': 9, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1}, 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-10-11 16:59:22,953 | DEBUG | batched.py ( 89) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-10-11 16:59:22,953 | TRACE | batched.py ( 90) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', b'{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}']) 2017-10-11 16:59:22,953 | TRACE | batched.py ( 98) | flushOutBoxes | Node2 sending payload to Node3: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH","signature":null}' 2017-10-11 16:59:22,953 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH","signature":null}' to Node3 2017-10-11 16:59:22,982 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 2 messages through listener 2017-10-11 16:59:22,983 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 9, '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-10-11 16:59:22,983 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 9, '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'op': 'VIEW_CHANGE_DONE', 'viewNo': 0}, 'Node3') 2017-10-11 16:59:22,983 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 9, '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-10-11 16:59:22,983 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node1: VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 9, '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-10-11 16:59:22,984 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 9, '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'op': 'VIEW_CHANGE_DONE', 'viewNo': 0}, 'Node1') 2017-10-11 16:59:22,984 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 9, '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-10-11 16:59:22,984 | DEBUG | node.py (1261) | sendToElector | Node2 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 9, '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0}, 'Node3') 2017-10-11 16:59:22,985 | DEBUG | node.py (1261) | sendToElector | Node2 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 9, '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0}, 'Node1') 2017-10-11 16:59:22,986 | DEBUG | primary_selector.py ( 103) | _processViewChangeDoneMessage | Node2's primary selector started processing of ViewChangeDone msg from Node3 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 9, '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-10-11 16:59:22,987 | INFO | primary_selector.py ( 192) | _hasViewChangeQuorum | Node2 got view change quorum (2 >= 2) 2017-10-11 16:59:22,988 | DEBUG | primary_selector.py ( 203) | has_view_change_from_primary | Node2 has not received ViewChangeDone from the next primary Node1 2017-10-11 16:59:22,988 | DEBUG | primary_selector.py ( 258) | _startSelection | Node2 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-10-11 16:59:22,988 | DEBUG | primary_selector.py ( 103) | _processViewChangeDoneMessage | Node2's primary selector started processing of ViewChangeDone msg from Node1 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 9, '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-10-11 16:59:22,988 | INFO | primary_selector.py ( 192) | _hasViewChangeQuorum | Node2 got view change quorum (3 >= 2) 2017-10-11 16:59:22,990 | DEBUG | primary_selector.py ( 209) | has_view_change_from_primary | Node2 received ViewChangeDone from primary Node1 2017-10-11 16:59:22,990 | DEBUG | primary_selector.py ( 231) | has_sufficient_same_view_change_done_messages | Node2 found acceptable primary Node1 and ledger info ((0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'), (1, 9, '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')) 2017-10-11 16:59:22,991 | DEBUG | primary_selector.py ( 272) | _startSelection | Node2 starting selection 2017-10-11 16:59:22,992 | DISPLAY | primary_selector.py ( 284) | _startSelection | Node2:0 selected primary Node1:0 for instance 0 (view 0) 2017-10-11 16:59:22,992 | INFO | node.py ( 482) | start_participating | Node2 started participating 2017-10-11 16:59:22,992 | INFO | replica.py ( 393) | primaryName | Node2:0 setting primaryName for view no 0 to: Node1:0 2017-10-11 16:59:22,992 | DEBUG | replica.py (1625) | _gc | Node2:0 cleaning up till (0, 0) 2017-10-11 16:59:22,992 | DEBUG | replica.py (1640) | _gc | Node2:0 found 0 3-phase keys to clean 2017-10-11 16:59:22,993 | DEBUG | replica.py (1642) | _gc | Node2:0 found 0 request keys to clean 2017-10-11 16:59:22,993 | INFO | replica.py ( 300) | h | Node2:0 set watermarks as 0 300 2017-10-11 16:59:22,993 | DISPLAY | primary_selector.py ( 307) | _startSelection | Node2:0 declares view change 0 as completed for instance 0, new primary is Node1:0, ledger info is [(0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'), (1, 9, '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-10-11 16:59:22,994 | DISPLAY | primary_selector.py ( 284) | _startSelection | Node2:1 selected primary Node2:1 for instance 1 (view 0) 2017-10-11 16:59:22,994 | INFO | replica.py ( 393) | primaryName | Node2:1 setting primaryName for view no 0 to: Node2:1 2017-10-11 16:59:22,994 | DEBUG | replica.py (1625) | _gc | Node2:1 cleaning up till (0, 0) 2017-10-11 16:59:22,994 | DEBUG | replica.py (1640) | _gc | Node2:1 found 0 3-phase keys to clean 2017-10-11 16:59:22,994 | DEBUG | replica.py (1642) | _gc | Node2:1 found 0 request keys to clean 2017-10-11 16:59:22,994 | INFO | replica.py ( 300) | h | Node2:1 set watermarks as 0 300 2017-10-11 16:59:22,995 | DEBUG | replica.py ( 494) | _setup_for_non_master | Node2:1 Setting last ordered for non-master as (0, 0) 2017-10-11 16:59:22,995 | DEBUG | replica.py ( 310) | last_ordered_3pc | Node2:1 set last ordered as (0, 0) 2017-10-11 16:59:22,995 | DISPLAY | primary_selector.py ( 307) | _startSelection | Node2:1 declares view change 0 as completed for instance 1, new primary is Node2:1, ledger info is [(0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'), (1, 9, '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-10-11 16:59:23,266 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 16:59:23,267 | DEBUG | zstack.py ( 652) | handlePingPong | Node2 got ping from Node4 2017-10-11 16:59:23,267 | DEBUG | zstack.py ( 643) | sendPingPong | Node2 will be sending in batch 2017-10-11 16:59:23,269 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'po' to Node4 2017-10-11 16:59:23,270 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'po' to Node4 2017-10-11 16:59:23,376 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 16:59:23,377 | DEBUG | zstack.py ( 658) | handlePingPong | Node2 got pong from Node4 2017-10-11 16:59:23,377 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node4: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp","ppSeqNo":null,"txnSeqNo":9,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp","ppSeqNo":null,"txnSeqNo":9,"viewNo":null}}'], 'signature': None} 2017-10-11 16:59:23,378 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp","ppSeqNo":null,"txnSeqNo":9,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp","ppSeqNo":null,"txnSeqNo":9,"viewNo":null}}'], 'op': 'BATCH', 'signature': None}, 'Node4') 2017-10-11 16:59:23,378 | DEBUG | node.py (1328) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp","ppSeqNo":null,"txnSeqNo":9,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp","ppSeqNo":null,"txnSeqNo":9,"viewNo":null}}'], 'signature': None} 2017-10-11 16:59:23,379 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2}, 'params': {'ledgerId': 2}} 2017-10-11 16:59:23,380 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}}, 'Node4') 2017-10-11 16:59:23,380 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2}, 'params': {'ledgerId': 2}} 2017-10-11 16:59:23,380 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2}, 'params': {'ledgerId': 2}} 2017-10-11 16:59:23,381 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2}, 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}}, 'Node4') 2017-10-11 16:59:23,381 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 0, 'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2}, 'params': {'ledgerId': 2}} 2017-10-11 16:59:23,382 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 9, 'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1}, 'params': {'ledgerId': 1}} 2017-10-11 16:59:23,382 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 9, 'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}}, 'Node4') 2017-10-11 16:59:23,383 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 9, 'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1}, 'params': {'ledgerId': 1}} 2017-10-11 16:59:23,383 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 9, 'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1}, 'params': {'ledgerId': 1}} 2017-10-11 16:59:23,383 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 9, 'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1}, 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}}, 'Node4') 2017-10-11 16:59:23,384 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': {'txnSeqNo': 9, 'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1}, 'params': {'ledgerId': 1}} 2017-10-11 16:59:23,384 | DEBUG | ledger_manager.py ( 244) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2} from Node4 2017-10-11 16:59:23,385 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-10-11 16:59:23,385 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-10-11 16:59:23,385 | DEBUG | ledger_manager.py ( 309) | processLedgerStatus | Node2 found out from {'Node4', 'Node3'} that its ledger of type 2 is latest 2017-10-11 16:59:23,386 | DEBUG | ledger_manager.py ( 244) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2} from Node4 2017-10-11 16:59:23,387 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-10-11 16:59:23,387 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 2 of size 0 with 0 2017-10-11 16:59:23,387 | DEBUG | ledger_manager.py ( 309) | processLedgerStatus | Node2 found out from {'Node4', 'Node3'} that its ledger of type 2 is latest 2017-10-11 16:59:23,388 | DEBUG | ledger_manager.py ( 244) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'txnSeqNo': 9, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1} from Node4 2017-10-11 16:59:23,388 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 1 of size 9 with 9 2017-10-11 16:59:23,388 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 1 of size 9 with 9 2017-10-11 16:59:23,388 | DEBUG | ledger_manager.py ( 309) | processLedgerStatus | Node2 found out from {'Node1', 'Node4'} that its ledger of type 1 is latest 2017-10-11 16:59:23,389 | DEBUG | ledger_manager.py ( 244) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'txnSeqNo': 9, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1} from Node4 2017-10-11 16:59:23,389 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 1 of size 9 with 9 2017-10-11 16:59:23,389 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 1 of size 9 with 9 2017-10-11 16:59:23,389 | DEBUG | ledger_manager.py ( 309) | processLedgerStatus | Node2 found out from {'Node1', 'Node4'} that its ledger of type 1 is latest 2017-10-11 16:59:23,392 | DEBUG | keep_in_touch.py ( 68) | conns | Node2's connections changed from {'Node1', 'Node3'} to {'Node1', 'Node4', 'Node3'} 2017-10-11 16:59:23,392 | INFO | keep_in_touch.py ( 96) | _connsChanged | Node2 now connected to Node4 2017-10-11 16:59:23,392 | DEBUG | motor.py ( 34) | set_status | Node2 changing status from started_hungry to started 2017-10-11 16:59:23,392 | DEBUG | node.py ( 918) | checkInstances | Node2 choosing to start election on the basis of count 4 and nodes {'Node1', 'Node4', 'Node3'} 2017-10-11 16:59:23,393 | DEBUG | node.py ( 879) | send_current_state_to_lagging_node | Node2 sending current state CURRENT_STATE{'primary': [VIEW_CHANGE_DONE{'ledgerInfo': ((0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'), (1, 9, '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')), 'name': 'Node1', 'viewNo': 0}], 'viewNo': 0} to lagged node Node4 2017-10-11 16:59:23,393 | DEBUG | node.py (2593) | send | Node2 sending message CURRENT_STATE{'primary': [VIEW_CHANGE_DONE{'ledgerInfo': ((0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'), (1, 9, '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')), 'name': 'Node1', 'viewNo': 0}], 'viewNo': 0} to 1 recipients: ['Node4'] 2017-10-11 16:59:23,393 | DEBUG | node.py (2593) | send | Node2 sending message LEDGER_STATUS{'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'txnSeqNo': 4, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 0} to 1 recipients: ['Node4'] 2017-10-11 16:59:23,394 | DEBUG | node.py (2593) | send | Node2 sending message LEDGER_STATUS{'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'txnSeqNo': 9, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1} to 1 recipients: ['Node4'] 2017-10-11 16:59:23,394 | DEBUG | node.py (2593) | send | Node2 sending message LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2} to 1 recipients: ['Node4'] 2017-10-11 16:59:23,396 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 16:59:23,396 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node4: BATCH{'messages': ['{"primary":[],"op":"CURRENT_STATE","viewNo":0}', '{"merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA","ledgerId":0,"viewNo":null,"txnSeqNo":4,"op":"LEDGER_STATUS","ppSeqNo":null}'], 'signature': None} 2017-10-11 16:59:23,397 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"primary":[],"op":"CURRENT_STATE","viewNo":0}', '{"merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA","ledgerId":0,"viewNo":null,"txnSeqNo":4,"op":"LEDGER_STATUS","ppSeqNo":null}'], 'op': 'BATCH', 'signature': None}, 'Node4') 2017-10-11 16:59:23,397 | DEBUG | node.py (1328) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"primary":[],"op":"CURRENT_STATE","viewNo":0}', '{"merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA","ledgerId":0,"viewNo":null,"txnSeqNo":4,"op":"LEDGER_STATUS","ppSeqNo":null}'], 'signature': None} 2017-10-11 16:59:23,397 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node4: CURRENT_STATE{'primary': [], 'viewNo': 0} 2017-10-11 16:59:23,397 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'primary': [], 'op': 'CURRENT_STATE', 'viewNo': 0}, 'Node4') 2017-10-11 16:59:23,398 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox CURRENT_STATE{'primary': [], 'viewNo': 0} 2017-10-11 16:59:23,398 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node4: LEDGER_STATUS{'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'txnSeqNo': 4, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 0} 2017-10-11 16:59:23,398 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'txnSeqNo': 4, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 0, 'op': 'LEDGER_STATUS', 'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'}, 'Node4') 2017-10-11 16:59:23,398 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox LEDGER_STATUS{'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'txnSeqNo': 4, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 0} 2017-10-11 16:59:23,399 | DEBUG | node.py ( 884) | process_current_state_message | Node2 processing current state CURRENT_STATE{'primary': [], 'viewNo': 0} from Node4 2017-10-11 16:59:23,399 | DEBUG | ledger_manager.py ( 244) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'txnSeqNo': 4, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 0} from Node4 2017-10-11 16:59:23,400 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 16:59:23,400 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 16:59:23,402 | DEBUG | batched.py ( 89) | flushOutBoxes | Node2 batching 4 msgs to Node4 into one transmission 2017-10-11 16:59:23,402 | TRACE | batched.py ( 90) | flushOutBoxes | messages: deque([b'{"primary":[{"ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,9,"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"name":"Node1","viewNo":0}],"op":"CURRENT_STATE","viewNo":0}', b'{"txnSeqNo":4,"viewNo":null,"ppSeqNo":null,"ledgerId":0,"op":"LEDGER_STATUS","merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"}', b'{"txnSeqNo":9,"viewNo":null,"ppSeqNo":null,"ledgerId":1,"op":"LEDGER_STATUS","merkleRoot":"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp"}', b'{"txnSeqNo":0,"viewNo":null,"ppSeqNo":null,"ledgerId":2,"op":"LEDGER_STATUS","merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"}']) 2017-10-11 16:59:23,402 | TRACE | batched.py ( 98) | flushOutBoxes | Node2 sending payload to Node4: b'{"messages":["{\\"primary\\":[{\\"ledgerInfo\\":[[0,4,\\"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA\\"],[1,9,\\"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"name\\":\\"Node1\\",\\"viewNo\\":0}],\\"op\\":\\"CURRENT_STATE\\",\\"viewNo\\":0}","{\\"txnSeqNo\\":4,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA\\"}","{\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp\\"}","{\\"txnSeqNo\\":0,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":2,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"}"],"op":"BATCH","signature":null}' 2017-10-11 16:59:23,403 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"messages":["{\\"primary\\":[{\\"ledgerInfo\\":[[0,4,\\"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA\\"],[1,9,\\"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp\\"],[2,0,\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"]],\\"name\\":\\"Node1\\",\\"viewNo\\":0}],\\"op\\":\\"CURRENT_STATE\\",\\"viewNo\\":0}","{\\"txnSeqNo\\":4,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":0,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA\\"}","{\\"txnSeqNo\\":9,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":1,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp\\"}","{\\"txnSeqNo\\":0,\\"viewNo\\":null,\\"ppSeqNo\\":null,\\"ledgerId\\":2,\\"op\\":\\"LEDGER_STATUS\\",\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\"}"],"op":"BATCH","signature":null}' to Node4 2017-10-11 16:59:23,436 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 16:59:23,437 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node4: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,9,"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","viewNo":0,"name":"Node1"}'], 'signature': None} 2017-10-11 16:59:23,437 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,9,"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","viewNo":0,"name":"Node1"}'], 'op': 'BATCH', 'signature': None}, 'Node4') 2017-10-11 16:59:23,437 | DEBUG | node.py (1328) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"ledgerInfo":[[0,4,"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"],[1,9,"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp"],[2,0,"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn"]],"op":"VIEW_CHANGE_DONE","viewNo":0,"name":"Node1"}'], 'signature': None} 2017-10-11 16:59:23,438 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 16:59:23,438 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}}, 'Node4') 2017-10-11 16:59:23,438 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 16:59:23,439 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 16:59:23,439 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}}, 'Node4') 2017-10-11 16:59:23,439 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 16:59:23,439 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 16:59:23,439 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}}, 'Node4') 2017-10-11 16:59:23,439 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 16:59:23,440 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 16:59:23,440 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}}, 'Node4') 2017-10-11 16:59:23,440 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 16:59:23,440 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node4: VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 9, '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-10-11 16:59:23,447 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 9, '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'op': 'VIEW_CHANGE_DONE', 'viewNo': 0}, 'Node4') 2017-10-11 16:59:23,450 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 9, '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-10-11 16:59:23,452 | DEBUG | node.py (2593) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2}, 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-10-11 16:59:23,453 | DEBUG | node.py (2593) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'merkleRoot': 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn', 'txnSeqNo': 0, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 2}, 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-10-11 16:59:23,453 | DEBUG | node.py (2593) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'txnSeqNo': 9, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1}, 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-10-11 16:59:23,459 | DEBUG | node.py (2593) | send | Node2 sending message MESSAGE_RESPONSE{'msg_type': 'LEDGER_STATUS', 'msg': LEDGER_STATUS{'merkleRoot': '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp', 'txnSeqNo': 9, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 1}, 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-10-11 16:59:23,460 | DEBUG | node.py (1261) | sendToElector | Node2 sending message to elector: (VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 9, '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0}, 'Node4') 2017-10-11 16:59:23,461 | DEBUG | primary_selector.py ( 103) | _processViewChangeDoneMessage | Node2's primary selector started processing of ViewChangeDone msg from Node4 : VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 9, '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} 2017-10-11 16:59:23,461 | DEBUG | message_processor.py ( 28) | discard | Node2 discarding message VIEW_CHANGE_DONE{'ledgerInfo': [[0, 4, '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA'], [1, 9, '9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp'], [2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn']], 'name': 'Node1', 'viewNo': 0} because it already decided primary which is Node1:0 2017-10-11 16:59:23,462 | DEBUG | batched.py ( 89) | flushOutBoxes | Node2 batching 4 msgs to Node4 into one transmission 2017-10-11 16:59:23,462 | TRACE | batched.py ( 90) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', b'{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn","ppSeqNo":null,"txnSeqNo":0,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', b'{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', b'{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp","ppSeqNo":null,"txnSeqNo":9,"viewNo":null},"op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}']) 2017-10-11 16:59:23,462 | TRACE | batched.py ( 98) | flushOutBoxes | Node2 sending payload to Node4: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH","signature":null}' 2017-10-11 16:59:23,462 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":0,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":9,\\"viewNo\\":null},\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}"],"op":"BATCH","signature":null}' to Node4 2017-10-11 16:59:31,884 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkPerformance with id 1 2017-10-11 16:59:31,885 | TRACE | node.py (2005) | checkPerformance | Node2 checking its performance 2017-10-11 16:59:31,885 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 16:59:31,885 | DEBUG | monitor.py ( 290) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-10-11 16:59:31,885 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 16:59:31,885 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 16:59:31,885 | DEBUG | node.py (2022) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 16:59:31,885 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 3 to run in 10 seconds 2017-10-11 16:59:36,908 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 16:59:36,909 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 16:59:36,909 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 16:59:36,910 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 16:59:41,898 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkPerformance with id 3 2017-10-11 16:59:41,898 | TRACE | node.py (2005) | checkPerformance | Node2 checking its performance 2017-10-11 16:59:41,898 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 16:59:41,898 | DEBUG | monitor.py ( 290) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-10-11 16:59:41,898 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 16:59:41,898 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 16:59:41,899 | DEBUG | node.py (2022) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 16:59:41,899 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 4 to run in 10 seconds 2017-10-11 16:59:51,909 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkPerformance with id 4 2017-10-11 16:59:51,909 | TRACE | node.py (2005) | checkPerformance | Node2 checking its performance 2017-10-11 16:59:51,909 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 16:59:51,909 | DEBUG | monitor.py ( 290) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-10-11 16:59:51,910 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 16:59:51,910 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 16:59:51,910 | DEBUG | node.py (2022) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 16:59:51,910 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 5 to run in 10 seconds 2017-10-11 16:59:51,911 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 16:59:51,911 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 16:59:51,911 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 16:59:51,912 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:00:01,923 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkPerformance with id 5 2017-10-11 17:00:01,923 | TRACE | node.py (2005) | checkPerformance | Node2 checking its performance 2017-10-11 17:00:01,923 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:00:01,923 | DEBUG | monitor.py ( 290) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-10-11 17:00:01,923 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:00:01,924 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:00:01,924 | DEBUG | node.py (2022) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:00:01,924 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 6 to run in 10 seconds 2017-10-11 17:00:06,920 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:00:06,920 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:00:06,920 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:00:06,921 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:00:11,933 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkPerformance with id 6 2017-10-11 17:00:11,933 | TRACE | node.py (2005) | checkPerformance | Node2 checking its performance 2017-10-11 17:00:11,934 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:00:11,934 | DEBUG | monitor.py ( 290) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-10-11 17:00:11,934 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:00:11,934 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:00:11,935 | DEBUG | node.py (2022) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:00:11,935 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 7 to run in 10 seconds 2017-10-11 17:00:21,873 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkNodeRequestSpike with id 2 2017-10-11 17:00:21,873 | DEBUG | node.py (2026) | checkNodeRequestSpike | Node2 checking its request amount 2017-10-11 17:00:21,873 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:00:21,873 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 8 to run in 60 seconds 2017-10-11 17:00:21,873 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkPerformance with id 1 2017-10-11 17:00:21,873 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:00:21,874 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 2 to run in 60 seconds 2017-10-11 17:00:21,923 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:00:21,923 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:00:21,923 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:00:21,923 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:00:21,937 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkPerformance with id 7 2017-10-11 17:00:21,937 | TRACE | node.py (2005) | checkPerformance | Node2 checking its performance 2017-10-11 17:00:21,937 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:00:21,938 | DEBUG | monitor.py ( 290) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-10-11 17:00:21,938 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:00:21,938 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:00:21,938 | DEBUG | node.py (2022) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:00:21,938 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 9 to run in 10 seconds 2017-10-11 17:00:31,942 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkPerformance with id 9 2017-10-11 17:00:31,942 | TRACE | node.py (2005) | checkPerformance | Node2 checking its performance 2017-10-11 17:00:31,943 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:00:31,943 | DEBUG | monitor.py ( 290) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-10-11 17:00:31,943 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:00:31,943 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:00:31,943 | DEBUG | node.py (2022) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:00:31,943 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 10 to run in 10 seconds 2017-10-11 17:00:36,926 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:00:36,930 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:00:36,930 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:00:36,931 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:00:38,028 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2C got 1 messages through listener 2017-10-11 17:00:38,029 | DEBUG | zstack.py ( 652) | handlePingPong | Node2C got ping from b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' 2017-10-11 17:00:38,029 | TRACE | zstack.py ( 726) | transmitThroughListener | Node2C transmitting b'po' to b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' through listener socket 2017-10-11 17:00:38,030 | DEBUG | zstack.py ( 635) | sendPingPong | Node2C ponged b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' 2017-10-11 17:00:38,077 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2C got 1 messages through listener 2017-10-11 17:00:38,077 | TRACE | node.py (1444) | validateClientMsg | Node2C received CLIENT message: LEDGER_STATUS{'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'txnSeqNo': 4, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 0} 2017-10-11 17:00:38,078 | DISPLAY | node.py (1490) | processClientInBox | Node2C processing b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' request LEDGER_STATUS{'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'txnSeqNo': 4, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 0} 2017-10-11 17:00:38,078 | DEBUG | ledger_manager.py ( 244) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'txnSeqNo': 4, 'viewNo': None, 'ppSeqNo': None, 'ledgerId': 0} from b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' 2017-10-11 17:00:38,079 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 17:00:38,079 | TRACE | zstack.py ( 726) | transmitThroughListener | Node2C transmitting b'{"txnSeqNo":4,"viewNo":null,"ppSeqNo":null,"ledgerId":0,"op":"LEDGER_STATUS","merkleRoot":"6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA"}' to b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' through listener socket 2017-10-11 17:00:38,080 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 17:00:38,080 | DEBUG | ledger_manager.py ( 965) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 17:00:41,944 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkPerformance with id 10 2017-10-11 17:00:41,944 | TRACE | node.py (2005) | checkPerformance | Node2 checking its performance 2017-10-11 17:00:41,944 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:00:41,944 | DEBUG | monitor.py ( 290) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-10-11 17:00:41,944 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:00:41,945 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:00:41,945 | DEBUG | node.py (2022) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:00:41,945 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 11 to run in 10 seconds 2017-10-11 17:00:51,929 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:00:51,929 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:00:51,929 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:00:51,929 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:00:51,957 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkPerformance with id 11 2017-10-11 17:00:51,957 | TRACE | node.py (2005) | checkPerformance | Node2 checking its performance 2017-10-11 17:00:51,957 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:00:51,957 | DEBUG | monitor.py ( 290) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-10-11 17:00:51,957 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:00:51,958 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:00:51,958 | DEBUG | node.py (2022) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:00:51,958 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 12 to run in 10 seconds 2017-10-11 17:00:52,114 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2C got 1 messages through listener 2017-10-11 17:00:52,115 | TRACE | signing.py ( 98) | serializeMsg | serialized msg {'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741252094258, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT1'}} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:dest:542MVr22zcHbVyGzaXmbT1|type:1|reqId:1507741252094258 2017-10-11 17:00:52,116 | TRACE | signing.py ( 98) | serializeMsg | serialized msg {'signature': '4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741252094258, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT1'}} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:dest:542MVr22zcHbVyGzaXmbT1|type:1|reqId:1507741252094258 2017-10-11 17:00:52,118 | INFO | node.py (2250) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1507741252094258 2017-10-11 17:00:52,118 | TRACE | node.py (1444) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'signature': '4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741252094258, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT1'}} 2017-10-11 17:00:52,118 | DISPLAY | node.py (1490) | processClientInBox | Node2C processing b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' request SafeRequest: {'signature': '4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741252094258, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT1'}} 2017-10-11 17:00:52,119 | DEBUG | node.py (1751) | processRequest | Node2 received client request: SafeRequest: {'signature': '4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741252094258, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT1'}} from b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' 2017-10-11 17:00:52,120 | TRACE | propagator.py ( 159) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'signature': '4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741252094258, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT1'}} 2017-10-11 17:00:52,120 | INFO | propagator.py ( 145) | propagate | Node2 propagating V4SGRU86Z58d6TV7PBUe6f request 1507741252094258 from client b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' 2017-10-11 17:00:52,120 | DEBUG | node.py (2593) | send | Node2 sending message PROPAGATE{'request': {'signature': '4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741252094258, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT1'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'} to all recipients: ['Node1', 'Node4', 'Node3'] 2017-10-11 17:00:52,121 | DEBUG | propagator.py ( 241) | tryForwarding | Node2 not forwarding request SafeRequest: {'signature': '4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741252094258, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT1'}} to its replicas since not finalised 2017-10-11 17:00:52,121 | TRACE | zstack.py ( 726) | transmitThroughListener | Node2C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1507741252094258}' to b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' through listener socket 2017-10-11 17:00:52,122 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"request":{"signature":"4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1507741252094258,"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT1"}},"op":"PROPAGATE","senderClient":"fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%"}' to Node1 2017-10-11 17:00:52,123 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"request":{"signature":"4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1507741252094258,"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT1"}},"op":"PROPAGATE","senderClient":"fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%"}' to Node1 2017-10-11 17:00:52,123 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"request":{"signature":"4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1507741252094258,"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT1"}},"op":"PROPAGATE","senderClient":"fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%"}' to Node4 2017-10-11 17:00:52,123 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"request":{"signature":"4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1507741252094258,"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT1"}},"op":"PROPAGATE","senderClient":"fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%"}' to Node4 2017-10-11 17:00:52,123 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"request":{"signature":"4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1507741252094258,"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT1"}},"op":"PROPAGATE","senderClient":"fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%"}' to Node3 2017-10-11 17:00:52,124 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"request":{"signature":"4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1507741252094258,"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT1"}},"op":"PROPAGATE","senderClient":"fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%"}' to Node3 2017-10-11 17:00:52,145 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 2 messages through listener 2017-10-11 17:00:52,146 | TRACE | signing.py ( 98) | serializeMsg | serialized msg {'reqId': 1507741252094258, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh', 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT1'}} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:dest:542MVr22zcHbVyGzaXmbT1|type:1|reqId:1507741252094258 2017-10-11 17:00:52,147 | INFO | node.py (2250) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1507741252094258 2017-10-11 17:00:52,150 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node4: PROPAGATE{'request': {'reqId': 1507741252094258, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh', 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT1'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'} 2017-10-11 17:00:52,151 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'request': {'reqId': 1507741252094258, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh', 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT1'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'}, 'Node4') 2017-10-11 17:00:52,151 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'request': {'reqId': 1507741252094258, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh', 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT1'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'} 2017-10-11 17:00:52,153 | TRACE | signing.py ( 98) | serializeMsg | serialized msg {'reqId': 1507741252094258, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh', 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT1'}} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:dest:542MVr22zcHbVyGzaXmbT1|type:1|reqId:1507741252094258 2017-10-11 17:00:52,160 | INFO | node.py (2250) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1507741252094258 2017-10-11 17:00:52,160 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node1: PROPAGATE{'request': {'reqId': 1507741252094258, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh', 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT1'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'} 2017-10-11 17:00:52,160 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'request': {'reqId': 1507741252094258, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh', 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT1'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'}, 'Node1') 2017-10-11 17:00:52,160 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'request': {'reqId': 1507741252094258, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh', 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT1'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'} 2017-10-11 17:00:52,161 | DEBUG | node.py (1799) | processPropagate | Node Node2 received propagated request: PROPAGATE{'request': {'reqId': 1507741252094258, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh', 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT1'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'} 2017-10-11 17:00:52,167 | TRACE | signing.py ( 98) | serializeMsg | serialized msg {'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741252094258, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT1'}} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:dest:542MVr22zcHbVyGzaXmbT1|type:1|reqId:1507741252094258 2017-10-11 17:00:52,167 | TRACE | propagator.py ( 138) | propagate | Node2 already propagated SafeRequest: {'signature': '4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741252094258, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT1'}} 2017-10-11 17:00:52,168 | DEBUG | propagator.py ( 208) | forward | Node2 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1507741252094258) to 2 replicas 2017-10-11 17:00:52,168 | DEBUG | node.py (1799) | processPropagate | Node Node2 received propagated request: PROPAGATE{'request': {'reqId': 1507741252094258, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh', 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT1'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'} 2017-10-11 17:00:52,168 | TRACE | signing.py ( 98) | serializeMsg | serialized msg {'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741252094258, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT1'}} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:dest:542MVr22zcHbVyGzaXmbT1|type:1|reqId:1507741252094258 2017-10-11 17:00:52,168 | TRACE | propagator.py ( 138) | propagate | Node2 already propagated SafeRequest: {'signature': '4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741252094258, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT1'}} 2017-10-11 17:00:52,169 | DEBUG | propagator.py ( 241) | tryForwarding | Node2 not forwarding request SafeRequest: {'signature': '4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741252094258, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT1'}} to its replicas since already forwarded 2017-10-11 17:00:52,189 | INFO | replica.py ( 642) | create3PCBatch | Node2:1 creating batch 1 for ledger 1 with state root None 2017-10-11 17:00:52,191 | DISPLAY | replica.py ( 673) | create3PCBatch | Node2:1 created a PRE-PREPARE with 1 requests for ledger 1 2017-10-11 17:00:52,191 | INFO | replica.py (2064) | send | Node2:1 sending PrePrepare 2017-10-11 17:00:52,191 | TRACE | replica.py (2065) | send | Node2:1 sending PREPREPARE{'instId': 1, 'stateRootHash': None, 'digest': 'cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59', 'viewNo': 0, 'discarded': 1, 'ppSeqNo': 1, 'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741252094258)], 'ppTime': 1507741252, 'txnRootHash': None} 2017-10-11 17:00:52,192 | DEBUG | replica.py ( 594) | trackBatches | Node2:1 tracking batch for PREPREPARE{'instId': 1, 'stateRootHash': None, 'digest': 'cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59', 'viewNo': 0, 'discarded': 1, 'ppSeqNo': 1, 'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741252094258)], 'ppTime': 1507741252, 'txnRootHash': None} with state root None 2017-10-11 17:00:52,193 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:00:52,198 | TRACE | signing.py ( 98) | serializeMsg | serialized msg {'reqId': 1507741252094258, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh', 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT1'}} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:dest:542MVr22zcHbVyGzaXmbT1|type:1|reqId:1507741252094258 2017-10-11 17:00:52,199 | INFO | node.py (2250) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1507741252094258 2017-10-11 17:00:52,200 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: PROPAGATE{'request': {'reqId': 1507741252094258, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh', 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT1'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'} 2017-10-11 17:00:52,201 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'request': {'reqId': 1507741252094258, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh', 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT1'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'}, 'Node3') 2017-10-11 17:00:52,201 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'request': {'reqId': 1507741252094258, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh', 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT1'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'} 2017-10-11 17:00:52,202 | DEBUG | node.py (1799) | processPropagate | Node Node2 received propagated request: PROPAGATE{'request': {'reqId': 1507741252094258, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh', 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT1'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'} 2017-10-11 17:00:52,203 | TRACE | signing.py ( 98) | serializeMsg | serialized msg {'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741252094258, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT1'}} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:dest:542MVr22zcHbVyGzaXmbT1|type:1|reqId:1507741252094258 2017-10-11 17:00:52,203 | TRACE | propagator.py ( 138) | propagate | Node2 already propagated SafeRequest: {'signature': '4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741252094258, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT1'}} 2017-10-11 17:00:52,203 | DEBUG | propagator.py ( 241) | tryForwarding | Node2 not forwarding request SafeRequest: {'signature': '4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741252094258, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT1'}} to its replicas since already forwarded 2017-10-11 17:00:52,207 | DEBUG | node.py (2593) | send | Node2 sending message PREPREPARE{'instId': 1, 'stateRootHash': None, 'digest': 'cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59', 'viewNo': 0, 'discarded': 1, 'ppSeqNo': 1, 'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741252094258)], 'ppTime': 1507741252, 'txnRootHash': None} to all recipients: ['Node1', 'Node4', 'Node3'] 2017-10-11 17:00:52,208 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:00:52,210 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node1: PREPREPARE{'instId': 0, 'stateRootHash': 'HysNuYRsJBXaczDjpZerFF2eevtjVVJDqVDQU2HJZwsZ', 'digest': 'cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59', 'viewNo': 0, 'discarded': 1, 'ppSeqNo': 1, 'ledgerId': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1507741252094258]], 'ppTime': 1507741252, 'txnRootHash': '2zbyjmFFrtYpWE6aVggU7hoc4uz2B1K5BgNnExpnRRy3'} 2017-10-11 17:00:52,211 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'discarded': 1, 'stateRootHash': 'HysNuYRsJBXaczDjpZerFF2eevtjVVJDqVDQU2HJZwsZ', 'viewNo': 0, 'digest': 'cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59', 'ppSeqNo': 1, 'ledgerId': 1, 'op': 'PREPREPARE', 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1507741252094258]], 'ppTime': 1507741252, 'txnRootHash': '2zbyjmFFrtYpWE6aVggU7hoc4uz2B1K5BgNnExpnRRy3'}, 'Node1') 2017-10-11 17:00:52,211 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox PREPREPARE{'instId': 0, 'stateRootHash': 'HysNuYRsJBXaczDjpZerFF2eevtjVVJDqVDQU2HJZwsZ', 'digest': 'cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59', 'viewNo': 0, 'discarded': 1, 'ppSeqNo': 1, 'ledgerId': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1507741252094258]], 'ppTime': 1507741252, 'txnRootHash': '2zbyjmFFrtYpWE6aVggU7hoc4uz2B1K5BgNnExpnRRy3'} 2017-10-11 17:00:52,213 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":1,"viewNo":0,"ppSeqNo":1,"discarded":1,"ledgerId":1,"op":"PREPREPARE","ppTime":1507741252,"digest":"cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741252094258]],"stateRootHash":null,"txnRootHash":null}' to Node1 2017-10-11 17:00:52,213 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":1,"viewNo":0,"ppSeqNo":1,"discarded":1,"ledgerId":1,"op":"PREPREPARE","ppTime":1507741252,"digest":"cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741252094258]],"stateRootHash":null,"txnRootHash":null}' to Node1 2017-10-11 17:00:52,214 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":1,"viewNo":0,"ppSeqNo":1,"discarded":1,"ledgerId":1,"op":"PREPREPARE","ppTime":1507741252,"digest":"cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741252094258]],"stateRootHash":null,"txnRootHash":null}' to Node4 2017-10-11 17:00:52,214 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":1,"viewNo":0,"ppSeqNo":1,"discarded":1,"ledgerId":1,"op":"PREPREPARE","ppTime":1507741252,"digest":"cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741252094258]],"stateRootHash":null,"txnRootHash":null}' to Node4 2017-10-11 17:00:52,214 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":1,"viewNo":0,"ppSeqNo":1,"discarded":1,"ledgerId":1,"op":"PREPREPARE","ppTime":1507741252,"digest":"cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741252094258]],"stateRootHash":null,"txnRootHash":null}' to Node3 2017-10-11 17:00:52,214 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":1,"viewNo":0,"ppSeqNo":1,"discarded":1,"ledgerId":1,"op":"PREPREPARE","ppTime":1507741252,"digest":"cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741252094258]],"stateRootHash":null,"txnRootHash":null}' to Node3 2017-10-11 17:00:52,215 | DEBUG | replica.py ( 790) | processPrePrepare | Node2:0 received PRE-PREPARE(0, 1) from Node1:0 at 2270337.673086218 2017-10-11 17:00:52,216 | DEBUG | replica.py (1002) | validate_pre_prepare | Node2:0 state root before processing PREPREPARE{'instId': 0, 'stateRootHash': 'HysNuYRsJBXaczDjpZerFF2eevtjVVJDqVDQU2HJZwsZ', 'digest': 'cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59', 'viewNo': 0, 'discarded': 1, 'ppSeqNo': 1, 'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741252094258)], 'ppTime': 1507741252, 'txnRootHash': '2zbyjmFFrtYpWE6aVggU7hoc4uz2B1K5BgNnExpnRRy3'} is b'\xe6\x82\x0f\x99\xf6@c\xaf\x02\x05\xec\xd1\xb3m\x1f\xbb\x0c\xcd\xa2\xedv\x1ef\xa6\xa2\xec\x9f\xdfI\xa0Cw', 9Db2gWhHtb3sKcbdwCymaX3Uo82kf4wKUbTqmBVmFxAp 2017-10-11 17:00:52,217 | DEBUG | replica.py ( 903) | doPrepare | Node2:0 Sending PREPARE(0, 1) at 2270337.675261945 2017-10-11 17:00:52,218 | INFO | replica.py (2064) | send | Node2:0 sending Prepare 2017-10-11 17:00:52,218 | TRACE | replica.py (2065) | send | Node2:0 sending PREPARE{'instId': 0, 'stateRootHash': 'HysNuYRsJBXaczDjpZerFF2eevtjVVJDqVDQU2HJZwsZ', 'viewNo': 0, 'ppTime': 1507741252, 'ppSeqNo': 1, 'digest': 'cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59', 'txnRootHash': '2zbyjmFFrtYpWE6aVggU7hoc4uz2B1K5BgNnExpnRRy3'} 2017-10-11 17:00:52,218 | DEBUG | replica.py ( 886) | tryCommit | Node2:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 0, 'stateRootHash': 'HysNuYRsJBXaczDjpZerFF2eevtjVVJDqVDQU2HJZwsZ', 'viewNo': 0, 'ppTime': 1507741252, 'ppSeqNo': 1, 'digest': 'cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59', 'txnRootHash': '2zbyjmFFrtYpWE6aVggU7hoc4uz2B1K5BgNnExpnRRy3'} 2017-10-11 17:00:52,218 | DEBUG | replica.py ( 594) | trackBatches | Node2:0 tracking batch for PREPREPARE{'instId': 0, 'stateRootHash': 'HysNuYRsJBXaczDjpZerFF2eevtjVVJDqVDQU2HJZwsZ', 'digest': 'cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59', 'viewNo': 0, 'discarded': 1, 'ppSeqNo': 1, 'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741252094258)], 'ppTime': 1507741252, 'txnRootHash': '2zbyjmFFrtYpWE6aVggU7hoc4uz2B1K5BgNnExpnRRy3'} with state root b'\xe6\x82\x0f\x99\xf6@c\xaf\x02\x05\xec\xd1\xb3m\x1f\xbb\x0c\xcd\xa2\xedv\x1ef\xa6\xa2\xec\x9f\xdfI\xa0Cw' 2017-10-11 17:00:52,218 | DEBUG | replica.py ( 809) | processPrePrepare | Node2:0 processed incoming PRE-PREPARE(0, 1) 2017-10-11 17:00:52,222 | DEBUG | node.py (2593) | send | Node2 sending message PREPARE{'instId': 0, 'stateRootHash': 'HysNuYRsJBXaczDjpZerFF2eevtjVVJDqVDQU2HJZwsZ', 'viewNo': 0, 'ppTime': 1507741252, 'ppSeqNo': 1, 'digest': 'cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59', 'txnRootHash': '2zbyjmFFrtYpWE6aVggU7hoc4uz2B1K5BgNnExpnRRy3'} to all recipients: ['Node1', 'Node4', 'Node3'] 2017-10-11 17:00:52,227 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":0,"viewNo":0,"ppSeqNo":1,"op":"PREPARE","ppTime":1507741252,"digest":"cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59","stateRootHash":"HysNuYRsJBXaczDjpZerFF2eevtjVVJDqVDQU2HJZwsZ","txnRootHash":"2zbyjmFFrtYpWE6aVggU7hoc4uz2B1K5BgNnExpnRRy3"}' to Node1 2017-10-11 17:00:52,228 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":0,"viewNo":0,"ppSeqNo":1,"op":"PREPARE","ppTime":1507741252,"digest":"cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59","stateRootHash":"HysNuYRsJBXaczDjpZerFF2eevtjVVJDqVDQU2HJZwsZ","txnRootHash":"2zbyjmFFrtYpWE6aVggU7hoc4uz2B1K5BgNnExpnRRy3"}' to Node1 2017-10-11 17:00:52,228 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":0,"viewNo":0,"ppSeqNo":1,"op":"PREPARE","ppTime":1507741252,"digest":"cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59","stateRootHash":"HysNuYRsJBXaczDjpZerFF2eevtjVVJDqVDQU2HJZwsZ","txnRootHash":"2zbyjmFFrtYpWE6aVggU7hoc4uz2B1K5BgNnExpnRRy3"}' to Node4 2017-10-11 17:00:52,228 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":0,"viewNo":0,"ppSeqNo":1,"op":"PREPARE","ppTime":1507741252,"digest":"cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59","stateRootHash":"HysNuYRsJBXaczDjpZerFF2eevtjVVJDqVDQU2HJZwsZ","txnRootHash":"2zbyjmFFrtYpWE6aVggU7hoc4uz2B1K5BgNnExpnRRy3"}' to Node4 2017-10-11 17:00:52,228 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":0,"viewNo":0,"ppSeqNo":1,"op":"PREPARE","ppTime":1507741252,"digest":"cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59","stateRootHash":"HysNuYRsJBXaczDjpZerFF2eevtjVVJDqVDQU2HJZwsZ","txnRootHash":"2zbyjmFFrtYpWE6aVggU7hoc4uz2B1K5BgNnExpnRRy3"}' to Node3 2017-10-11 17:00:52,228 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":0,"viewNo":0,"ppSeqNo":1,"op":"PREPARE","ppTime":1507741252,"digest":"cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59","stateRootHash":"HysNuYRsJBXaczDjpZerFF2eevtjVVJDqVDQU2HJZwsZ","txnRootHash":"2zbyjmFFrtYpWE6aVggU7hoc4uz2B1K5BgNnExpnRRy3"}' to Node3 2017-10-11 17:00:52,230 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:00:52,232 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'instId': 0, 'stateRootHash': 'HysNuYRsJBXaczDjpZerFF2eevtjVVJDqVDQU2HJZwsZ', 'viewNo': 0, 'ppTime': 1507741252, 'ppSeqNo': 1, 'digest': 'cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59', 'txnRootHash': '2zbyjmFFrtYpWE6aVggU7hoc4uz2B1K5BgNnExpnRRy3'} 2017-10-11 17:00:52,232 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'digest': 'cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59', 'viewNo': 0, 'txnRootHash': '2zbyjmFFrtYpWE6aVggU7hoc4uz2B1K5BgNnExpnRRy3', 'ppSeqNo': 1, 'ppTime': 1507741252, 'op': 'PREPARE', 'stateRootHash': 'HysNuYRsJBXaczDjpZerFF2eevtjVVJDqVDQU2HJZwsZ'}, 'Node4') 2017-10-11 17:00:52,232 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'instId': 0, 'stateRootHash': 'HysNuYRsJBXaczDjpZerFF2eevtjVVJDqVDQU2HJZwsZ', 'viewNo': 0, 'ppTime': 1507741252, 'ppSeqNo': 1, 'digest': 'cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59', 'txnRootHash': '2zbyjmFFrtYpWE6aVggU7hoc4uz2B1K5BgNnExpnRRy3'} 2017-10-11 17:00:52,235 | DEBUG | replica.py ( 834) | processPrepare | Node2:0 received PREPARE(0, 1) from Node4:0 2017-10-11 17:00:52,235 | DEBUG | replica.py ( 922) | doCommit | Node2:0 Sending COMMIT(0, 1) at 2270337.693343349 2017-10-11 17:00:52,236 | INFO | replica.py (2064) | send | Node2:0 sending Commit 2017-10-11 17:00:52,236 | TRACE | replica.py (2065) | send | Node2:0 sending COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 1} 2017-10-11 17:00:52,236 | DEBUG | replica.py ( 898) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 1} commits where f is 1 2017-10-11 17:00:52,236 | DEBUG | replica.py ( 845) | processPrepare | Node2:0 processed incoming PREPARE (0, 1) 2017-10-11 17:00:52,237 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 2 messages through listener 2017-10-11 17:00:52,240 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node1: PREPARE{'instId': 1, 'stateRootHash': None, 'viewNo': 0, 'ppTime': 1507741252, 'ppSeqNo': 1, 'digest': 'cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59', 'txnRootHash': None} 2017-10-11 17:00:52,241 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'stateRootHash': None, 'viewNo': 0, 'digest': 'cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59', 'ppSeqNo': 1, 'op': 'PREPARE', 'ppTime': 1507741252, 'txnRootHash': None}, 'Node1') 2017-10-11 17:00:52,241 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'instId': 1, 'stateRootHash': None, 'viewNo': 0, 'ppTime': 1507741252, 'ppSeqNo': 1, 'digest': 'cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59', 'txnRootHash': None} 2017-10-11 17:00:52,241 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'instId': 1, 'stateRootHash': None, 'viewNo': 0, 'ppTime': 1507741252, 'ppSeqNo': 1, 'digest': 'cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59', 'txnRootHash': None} 2017-10-11 17:00:52,242 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'digest': 'cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59', 'viewNo': 0, 'txnRootHash': None, 'ppSeqNo': 1, 'ppTime': 1507741252, 'op': 'PREPARE', 'stateRootHash': None}, 'Node4') 2017-10-11 17:00:52,243 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'instId': 1, 'stateRootHash': None, 'viewNo': 0, 'ppTime': 1507741252, 'ppSeqNo': 1, 'digest': 'cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59', 'txnRootHash': None} 2017-10-11 17:00:52,247 | DEBUG | node.py (2593) | send | Node2 sending message COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 1} to all recipients: ['Node1', 'Node4', 'Node3'] 2017-10-11 17:00:52,248 | DEBUG | replica.py ( 834) | processPrepare | Node2:1 received PREPARE(0, 1) from Node1:1 2017-10-11 17:00:52,249 | DEBUG | replica.py ( 886) | tryCommit | Node2:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 1, 'stateRootHash': None, 'viewNo': 0, 'ppTime': 1507741252, 'ppSeqNo': 1, 'digest': 'cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59', 'txnRootHash': None} 2017-10-11 17:00:52,249 | DEBUG | replica.py ( 845) | processPrepare | Node2:1 processed incoming PREPARE (0, 1) 2017-10-11 17:00:52,249 | DEBUG | replica.py ( 834) | processPrepare | Node2:1 received PREPARE(0, 1) from Node4:1 2017-10-11 17:00:52,250 | DEBUG | replica.py ( 922) | doCommit | Node2:1 Sending COMMIT(0, 1) at 2270337.707442711 2017-10-11 17:00:52,250 | INFO | replica.py (2064) | send | Node2:1 sending Commit 2017-10-11 17:00:52,250 | TRACE | replica.py (2065) | send | Node2:1 sending COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 1} 2017-10-11 17:00:52,251 | DEBUG | replica.py ( 898) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 1} commits where f is 1 2017-10-11 17:00:52,251 | DEBUG | replica.py ( 845) | processPrepare | Node2:1 processed incoming PREPARE (0, 1) 2017-10-11 17:00:52,254 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:00:52,256 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'instId': 0, 'stateRootHash': 'HysNuYRsJBXaczDjpZerFF2eevtjVVJDqVDQU2HJZwsZ', 'viewNo': 0, 'ppTime': 1507741252, 'ppSeqNo': 1, 'digest': 'cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59', 'txnRootHash': '2zbyjmFFrtYpWE6aVggU7hoc4uz2B1K5BgNnExpnRRy3'} 2017-10-11 17:00:52,257 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'digest': 'cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59', 'viewNo': 0, 'ppTime': 1507741252, 'ppSeqNo': 1, 'op': 'PREPARE', 'stateRootHash': 'HysNuYRsJBXaczDjpZerFF2eevtjVVJDqVDQU2HJZwsZ', 'txnRootHash': '2zbyjmFFrtYpWE6aVggU7hoc4uz2B1K5BgNnExpnRRy3'}, 'Node3') 2017-10-11 17:00:52,257 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'instId': 0, 'stateRootHash': 'HysNuYRsJBXaczDjpZerFF2eevtjVVJDqVDQU2HJZwsZ', 'viewNo': 0, 'ppTime': 1507741252, 'ppSeqNo': 1, 'digest': 'cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59', 'txnRootHash': '2zbyjmFFrtYpWE6aVggU7hoc4uz2B1K5BgNnExpnRRy3'} 2017-10-11 17:00:52,261 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node1 2017-10-11 17:00:52,261 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node1 2017-10-11 17:00:52,262 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node4 2017-10-11 17:00:52,262 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node4 2017-10-11 17:00:52,263 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node3 2017-10-11 17:00:52,264 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node3 2017-10-11 17:00:52,266 | DEBUG | node.py (2593) | send | Node2 sending message COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 1} to all recipients: ['Node1', 'Node4', 'Node3'] 2017-10-11 17:00:52,267 | DEBUG | replica.py ( 834) | processPrepare | Node2:0 received PREPARE(0, 1) from Node3:0 2017-10-11 17:00:52,267 | DEBUG | replica.py ( 886) | tryCommit | Node2:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'instId': 0, 'stateRootHash': 'HysNuYRsJBXaczDjpZerFF2eevtjVVJDqVDQU2HJZwsZ', 'viewNo': 0, 'ppTime': 1507741252, 'ppSeqNo': 1, 'digest': 'cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59', 'txnRootHash': '2zbyjmFFrtYpWE6aVggU7hoc4uz2B1K5BgNnExpnRRy3'} 2017-10-11 17:00:52,268 | DEBUG | replica.py ( 845) | processPrepare | Node2:0 processed incoming PREPARE (0, 1) 2017-10-11 17:00:52,270 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 5 messages through listener 2017-10-11 17:00:52,271 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 1} 2017-10-11 17:00:52,271 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 1}, 'Node1') 2017-10-11 17:00:52,272 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 1} 2017-10-11 17:00:52,275 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 1} 2017-10-11 17:00:52,275 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 1}, 'Node4') 2017-10-11 17:00:52,276 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 1} 2017-10-11 17:00:52,276 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: BATCH{'messages': ['{"op":"COMMIT","ppSeqNo":1,"viewNo":0,"instId":0}', '{"ppSeqNo":1,"op":"PREPARE","ppTime":1507741252,"stateRootHash":null,"viewNo":0,"digest":"cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59","txnRootHash":null,"instId":1}'], 'signature': None} 2017-10-11 17:00:52,276 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"op":"COMMIT","ppSeqNo":1,"viewNo":0,"instId":0}', '{"ppSeqNo":1,"op":"PREPARE","ppTime":1507741252,"stateRootHash":null,"viewNo":0,"digest":"cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59","txnRootHash":null,"instId":1}'], 'op': 'BATCH', 'signature': None}, 'Node3') 2017-10-11 17:00:52,276 | DEBUG | node.py (1328) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"op":"COMMIT","ppSeqNo":1,"viewNo":0,"instId":0}', '{"ppSeqNo":1,"op":"PREPARE","ppTime":1507741252,"stateRootHash":null,"viewNo":0,"digest":"cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59","txnRootHash":null,"instId":1}'], 'signature': None} 2017-10-11 17:00:52,277 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 1} 2017-10-11 17:00:52,277 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 1}, 'Node3') 2017-10-11 17:00:52,277 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 1} 2017-10-11 17:00:52,278 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'instId': 1, 'stateRootHash': None, 'viewNo': 0, 'ppTime': 1507741252, 'ppSeqNo': 1, 'digest': 'cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59', 'txnRootHash': None} 2017-10-11 17:00:52,278 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'digest': 'cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59', 'viewNo': 0, 'ppTime': 1507741252, 'ppSeqNo': 1, 'op': 'PREPARE', 'stateRootHash': None, 'txnRootHash': None}, 'Node3') 2017-10-11 17:00:52,279 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'instId': 1, 'stateRootHash': None, 'viewNo': 0, 'ppTime': 1507741252, 'ppSeqNo': 1, 'digest': 'cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59', 'txnRootHash': None} 2017-10-11 17:00:52,281 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 1} 2017-10-11 17:00:52,282 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 1}, 'Node1') 2017-10-11 17:00:52,282 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 1} 2017-10-11 17:00:52,283 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 1} 2017-10-11 17:00:52,283 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 1}, 'Node4') 2017-10-11 17:00:52,284 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 1} 2017-10-11 17:00:52,287 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node1 2017-10-11 17:00:52,288 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node1 2017-10-11 17:00:52,288 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node4 2017-10-11 17:00:52,290 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node4 2017-10-11 17:00:52,291 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node3 2017-10-11 17:00:52,291 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":1}' to Node3 2017-10-11 17:00:52,293 | DEBUG | replica.py ( 863) | processCommit | Node2:0 received COMMIT(0, 1) from Node1:0 2017-10-11 17:00:52,294 | DEBUG | replica.py ( 898) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 1} commits where f is 1 2017-10-11 17:00:52,294 | DEBUG | replica.py ( 874) | processCommit | Node2:0 processed incoming COMMIT(0, 1) 2017-10-11 17:00:52,295 | DEBUG | replica.py ( 863) | processCommit | Node2:0 received COMMIT(0, 1) from Node4:0 2017-10-11 17:00:52,296 | TRACE | replica.py ( 894) | tryOrder | Node2:0 returning request to node 2017-10-11 17:00:52,296 | INFO | replica.py (1395) | doOrder | Node2:0 ordering COMMIT(0, 1) 2017-10-11 17:00:52,296 | DEBUG | replica.py ( 310) | last_ordered_3pc | Node2:0 set last ordered as (0, 1) 2017-10-11 17:00:52,298 | INFO | replica.py (2064) | send | Node2:0 sending Ordered 2017-10-11 17:00:52,299 | TRACE | replica.py (2065) | send | Node2:0 sending ORDERED{'instId': 0, 'viewNo': 0, 'stateRootHash': 'HysNuYRsJBXaczDjpZerFF2eevtjVVJDqVDQU2HJZwsZ', 'ppSeqNo': 1, 'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741252094258)], 'ppTime': 1507741252, 'txnRootHash': '2zbyjmFFrtYpWE6aVggU7hoc4uz2B1K5BgNnExpnRRy3'} 2017-10-11 17:00:52,299 | DEBUG | replica.py (1426) | order_3pc_key | Node2:0 ordered request (0, 1) 2017-10-11 17:00:52,299 | DEBUG | replica.py (1494) | _newCheckpointState | Node2:0 adding new checkpoint state for (1, 100) 2017-10-11 17:00:52,300 | DEBUG | replica.py ( 874) | processCommit | Node2:0 processed incoming COMMIT(0, 1) 2017-10-11 17:00:52,300 | TRACE | message_processor.py ( 28) | discard | Node2:0 discarding message COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 1} because already ordered 3 phase message 2017-10-11 17:00:52,301 | DEBUG | replica.py ( 834) | processPrepare | Node2:1 received PREPARE(0, 1) from Node3:1 2017-10-11 17:00:52,301 | DEBUG | replica.py ( 886) | tryCommit | Node2:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'instId': 1, 'stateRootHash': None, 'viewNo': 0, 'ppTime': 1507741252, 'ppSeqNo': 1, 'digest': 'cff18401c6bd55c81642da455db0a1d5c557f47fdd2da9646a4f5163b9170a59', 'txnRootHash': None} 2017-10-11 17:00:52,301 | DEBUG | replica.py ( 845) | processPrepare | Node2:1 processed incoming PREPARE (0, 1) 2017-10-11 17:00:52,302 | DEBUG | replica.py ( 863) | processCommit | Node2:1 received COMMIT(0, 1) from Node1:1 2017-10-11 17:00:52,302 | DEBUG | replica.py ( 898) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 1} commits where f is 1 2017-10-11 17:00:52,302 | DEBUG | replica.py ( 874) | processCommit | Node2:1 processed incoming COMMIT(0, 1) 2017-10-11 17:00:52,303 | DEBUG | replica.py ( 863) | processCommit | Node2:1 received COMMIT(0, 1) from Node4:1 2017-10-11 17:00:52,303 | TRACE | replica.py ( 894) | tryOrder | Node2:1 returning request to node 2017-10-11 17:00:52,306 | INFO | replica.py (1395) | doOrder | Node2:1 ordering COMMIT(0, 1) 2017-10-11 17:00:52,306 | DEBUG | replica.py ( 310) | last_ordered_3pc | Node2:1 set last ordered as (0, 1) 2017-10-11 17:00:52,307 | INFO | replica.py (2064) | send | Node2:1 sending Ordered 2017-10-11 17:00:52,307 | TRACE | replica.py (2065) | send | Node2:1 sending ORDERED{'instId': 1, 'viewNo': 0, 'stateRootHash': None, 'ppSeqNo': 1, 'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741252094258)], 'ppTime': 1507741252, 'txnRootHash': None} 2017-10-11 17:00:52,308 | DEBUG | replica.py (1426) | order_3pc_key | Node2:1 ordered request (0, 1) 2017-10-11 17:00:52,309 | DEBUG | replica.py (1494) | _newCheckpointState | Node2:1 adding new checkpoint state for (1, 100) 2017-10-11 17:00:52,309 | DEBUG | replica.py ( 874) | processCommit | Node2:1 processed incoming COMMIT(0, 1) 2017-10-11 17:00:52,310 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:00:52,312 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 1} 2017-10-11 17:00:52,312 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 1}, 'Node3') 2017-10-11 17:00:52,312 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 1} 2017-10-11 17:00:52,316 | DEBUG | node.py (1881) | processOrdered | Node2 executing Ordered batch 0 1 of 1 requests 2017-10-11 17:00:52,319 | DEBUG | ledger.py ( 68) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-10-11 17:00:52,320 | DEBUG | node.py (2374) | sendReplyToClient | Node2 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1507741252094258) to client 2017-10-11 17:00:52,320 | TRACE | zstack.py ( 726) | transmitThroughListener | Node2C transmitting b'{"op":"REPLY","result":{"type":"1","signature":"4RSVWEnDKexCuWpxLdbhmXbfQprab1QFbNSF2J9eWivyHa1jjuemxp5i6iKToEe7dZagtgAsifzeVZYg4Jf6C1Vh","dest":"542MVr22zcHbVyGzaXmbT1","auditPath":["GbwFpoGxJMdHfwZJmT2WpE3rtWD2cW2Cevq7bC75nkyb","HYRnUmRJ2hQMtr1smmXw3sqWyQyKTzCeTh5D9EweiD8z"],"seqNo":10,"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1507741252094258,"txnTime":1507741252,"rootHash":"2zbyjmFFrtYpWE6aVggU7hoc4uz2B1K5BgNnExpnRRy3"}}' to b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' through listener socket 2017-10-11 17:00:52,322 | DEBUG | node.py (2295) | executeBatch | Node2 storing 3PC key (0, 1) for ledger 1 range (10, 10) 2017-10-11 17:00:52,322 | TRACE | node.py (1892) | processOrdered | Node2 got ordered requests from backup replica 1 2017-10-11 17:00:52,323 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:00:52,323 | TRACE | message_processor.py ( 28) | discard | Node2:1 discarding message COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 1} because already ordered 3 phase message 2017-10-11 17:00:52,329 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.186 seconds 2017-10-11 17:00:56,182 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2C got 1 messages through listener 2017-10-11 17:00:56,182 | TRACE | signing.py ( 98) | serializeMsg | serialized msg {'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741256159186, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT2'}} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:dest:542MVr22zcHbVyGzaXmbT2|type:1|reqId:1507741256159186 2017-10-11 17:00:56,183 | TRACE | signing.py ( 98) | serializeMsg | serialized msg {'signature': '2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741256159186, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT2'}} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:dest:542MVr22zcHbVyGzaXmbT2|type:1|reqId:1507741256159186 2017-10-11 17:00:56,183 | INFO | node.py (2250) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1507741256159186 2017-10-11 17:00:56,184 | TRACE | node.py (1444) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'signature': '2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741256159186, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT2'}} 2017-10-11 17:00:56,184 | DISPLAY | node.py (1490) | processClientInBox | Node2C processing b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' request SafeRequest: {'signature': '2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741256159186, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT2'}} 2017-10-11 17:00:56,186 | DEBUG | node.py (1751) | processRequest | Node2 received client request: SafeRequest: {'signature': '2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741256159186, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT2'}} from b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' 2017-10-11 17:00:56,186 | TRACE | propagator.py ( 159) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'signature': '2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741256159186, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT2'}} 2017-10-11 17:00:56,187 | INFO | propagator.py ( 145) | propagate | Node2 propagating V4SGRU86Z58d6TV7PBUe6f request 1507741256159186 from client b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' 2017-10-11 17:00:56,187 | DEBUG | node.py (2593) | send | Node2 sending message PROPAGATE{'request': {'signature': '2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741256159186, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT2'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'} to all recipients: ['Node1', 'Node4', 'Node3'] 2017-10-11 17:00:56,187 | DEBUG | propagator.py ( 241) | tryForwarding | Node2 not forwarding request SafeRequest: {'signature': '2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741256159186, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT2'}} to its replicas since not finalised 2017-10-11 17:00:56,187 | TRACE | zstack.py ( 726) | transmitThroughListener | Node2C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1507741256159186}' to b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' through listener socket 2017-10-11 17:00:56,189 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"request":{"signature":"2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1507741256159186,"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT2"}},"op":"PROPAGATE","senderClient":"fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%"}' to Node1 2017-10-11 17:00:56,203 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"request":{"signature":"2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1507741256159186,"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT2"}},"op":"PROPAGATE","senderClient":"fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%"}' to Node1 2017-10-11 17:00:56,205 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"request":{"signature":"2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1507741256159186,"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT2"}},"op":"PROPAGATE","senderClient":"fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%"}' to Node4 2017-10-11 17:00:56,205 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"request":{"signature":"2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1507741256159186,"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT2"}},"op":"PROPAGATE","senderClient":"fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%"}' to Node4 2017-10-11 17:00:56,205 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"request":{"signature":"2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1507741256159186,"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT2"}},"op":"PROPAGATE","senderClient":"fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%"}' to Node3 2017-10-11 17:00:56,205 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"request":{"signature":"2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1507741256159186,"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT2"}},"op":"PROPAGATE","senderClient":"fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%"}' to Node3 2017-10-11 17:00:56,223 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:00:56,224 | TRACE | signing.py ( 98) | serializeMsg | serialized msg {'reqId': 1507741256159186, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3', 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT2'}} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:dest:542MVr22zcHbVyGzaXmbT2|type:1|reqId:1507741256159186 2017-10-11 17:00:56,225 | INFO | node.py (2250) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1507741256159186 2017-10-11 17:00:56,225 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node4: PROPAGATE{'request': {'reqId': 1507741256159186, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3', 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT2'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'} 2017-10-11 17:00:56,225 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'request': {'reqId': 1507741256159186, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3', 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT2'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'}, 'Node4') 2017-10-11 17:00:56,230 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'request': {'reqId': 1507741256159186, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3', 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT2'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'} 2017-10-11 17:00:56,231 | DEBUG | node.py (1799) | processPropagate | Node Node2 received propagated request: PROPAGATE{'request': {'reqId': 1507741256159186, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3', 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT2'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'} 2017-10-11 17:00:56,231 | TRACE | signing.py ( 98) | serializeMsg | serialized msg {'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741256159186, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT2'}} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:dest:542MVr22zcHbVyGzaXmbT2|type:1|reqId:1507741256159186 2017-10-11 17:00:56,231 | TRACE | propagator.py ( 138) | propagate | Node2 already propagated SafeRequest: {'signature': '2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741256159186, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT2'}} 2017-10-11 17:00:56,232 | DEBUG | propagator.py ( 208) | forward | Node2 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1507741256159186) to 2 replicas 2017-10-11 17:00:56,248 | INFO | replica.py ( 642) | create3PCBatch | Node2:1 creating batch 2 for ledger 1 with state root None 2017-10-11 17:00:56,249 | DISPLAY | replica.py ( 673) | create3PCBatch | Node2:1 created a PRE-PREPARE with 1 requests for ledger 1 2017-10-11 17:00:56,249 | INFO | replica.py (2064) | send | Node2:1 sending PrePrepare 2017-10-11 17:00:56,249 | TRACE | replica.py (2065) | send | Node2:1 sending PREPREPARE{'instId': 1, 'stateRootHash': None, 'digest': '9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42', 'viewNo': 0, 'discarded': 1, 'ppSeqNo': 2, 'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741256159186)], 'ppTime': 1507741256, 'txnRootHash': None} 2017-10-11 17:00:56,249 | DEBUG | replica.py ( 594) | trackBatches | Node2:1 tracking batch for PREPREPARE{'instId': 1, 'stateRootHash': None, 'digest': '9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42', 'viewNo': 0, 'discarded': 1, 'ppSeqNo': 2, 'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741256159186)], 'ppTime': 1507741256, 'txnRootHash': None} with state root None 2017-10-11 17:00:56,250 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 2 messages through listener 2017-10-11 17:00:56,257 | TRACE | signing.py ( 98) | serializeMsg | serialized msg {'reqId': 1507741256159186, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3', 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT2'}} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:dest:542MVr22zcHbVyGzaXmbT2|type:1|reqId:1507741256159186 2017-10-11 17:00:56,258 | INFO | node.py (2250) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1507741256159186 2017-10-11 17:00:56,259 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: PROPAGATE{'request': {'reqId': 1507741256159186, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3', 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT2'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'} 2017-10-11 17:00:56,259 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'request': {'reqId': 1507741256159186, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3', 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT2'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'}, 'Node3') 2017-10-11 17:00:56,259 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'request': {'reqId': 1507741256159186, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3', 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT2'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'} 2017-10-11 17:00:56,260 | TRACE | signing.py ( 98) | serializeMsg | serialized msg {'reqId': 1507741256159186, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3', 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT2'}} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:dest:542MVr22zcHbVyGzaXmbT2|type:1|reqId:1507741256159186 2017-10-11 17:00:56,261 | INFO | node.py (2250) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1507741256159186 2017-10-11 17:00:56,266 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node1: PROPAGATE{'request': {'reqId': 1507741256159186, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3', 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT2'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'} 2017-10-11 17:00:56,266 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'request': {'reqId': 1507741256159186, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3', 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT2'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'}, 'Node1') 2017-10-11 17:00:56,267 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'request': {'reqId': 1507741256159186, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3', 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT2'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'} 2017-10-11 17:00:56,268 | DEBUG | node.py (1799) | processPropagate | Node Node2 received propagated request: PROPAGATE{'request': {'reqId': 1507741256159186, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3', 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT2'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'} 2017-10-11 17:00:56,268 | TRACE | signing.py ( 98) | serializeMsg | serialized msg {'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741256159186, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT2'}} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:dest:542MVr22zcHbVyGzaXmbT2|type:1|reqId:1507741256159186 2017-10-11 17:00:56,271 | TRACE | propagator.py ( 138) | propagate | Node2 already propagated SafeRequest: {'signature': '2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741256159186, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT2'}} 2017-10-11 17:00:56,272 | DEBUG | propagator.py ( 241) | tryForwarding | Node2 not forwarding request SafeRequest: {'signature': '2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741256159186, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT2'}} to its replicas since already forwarded 2017-10-11 17:00:56,273 | DEBUG | node.py (1799) | processPropagate | Node Node2 received propagated request: PROPAGATE{'request': {'reqId': 1507741256159186, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3', 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT2'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'} 2017-10-11 17:00:56,274 | TRACE | signing.py ( 98) | serializeMsg | serialized msg {'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741256159186, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT2'}} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:dest:542MVr22zcHbVyGzaXmbT2|type:1|reqId:1507741256159186 2017-10-11 17:00:56,274 | TRACE | propagator.py ( 138) | propagate | Node2 already propagated SafeRequest: {'signature': '2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741256159186, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT2'}} 2017-10-11 17:00:56,274 | DEBUG | propagator.py ( 241) | tryForwarding | Node2 not forwarding request SafeRequest: {'signature': '2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741256159186, 'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT2'}} to its replicas since already forwarded 2017-10-11 17:00:56,277 | DEBUG | node.py (2593) | send | Node2 sending message PREPREPARE{'instId': 1, 'stateRootHash': None, 'digest': '9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42', 'viewNo': 0, 'discarded': 1, 'ppSeqNo': 2, 'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741256159186)], 'ppTime': 1507741256, 'txnRootHash': None} to all recipients: ['Node1', 'Node4', 'Node3'] 2017-10-11 17:00:56,279 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:00:56,280 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node1: PREPREPARE{'instId': 0, 'stateRootHash': '8F8st3cEZDFKTucjctdXwyAddoLTXatnRtpS63YHxZYX', 'digest': '9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42', 'viewNo': 0, 'discarded': 1, 'ppSeqNo': 2, 'ledgerId': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1507741256159186]], 'ppTime': 1507741256, 'txnRootHash': 'CmvKha4N44FPRqPyp2VfEjjDq6FCSjfXPwBusRQz7sXw'} 2017-10-11 17:00:56,290 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'discarded': 1, 'stateRootHash': '8F8st3cEZDFKTucjctdXwyAddoLTXatnRtpS63YHxZYX', 'viewNo': 0, 'digest': '9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42', 'ppSeqNo': 2, 'ledgerId': 1, 'op': 'PREPREPARE', 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1507741256159186]], 'ppTime': 1507741256, 'txnRootHash': 'CmvKha4N44FPRqPyp2VfEjjDq6FCSjfXPwBusRQz7sXw'}, 'Node1') 2017-10-11 17:00:56,291 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox PREPREPARE{'instId': 0, 'stateRootHash': '8F8st3cEZDFKTucjctdXwyAddoLTXatnRtpS63YHxZYX', 'digest': '9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42', 'viewNo': 0, 'discarded': 1, 'ppSeqNo': 2, 'ledgerId': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1507741256159186]], 'ppTime': 1507741256, 'txnRootHash': 'CmvKha4N44FPRqPyp2VfEjjDq6FCSjfXPwBusRQz7sXw'} 2017-10-11 17:00:56,297 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":1,"viewNo":0,"ppSeqNo":2,"discarded":1,"ledgerId":1,"op":"PREPREPARE","ppTime":1507741256,"digest":"9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741256159186]],"stateRootHash":null,"txnRootHash":null}' to Node1 2017-10-11 17:00:56,298 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":1,"viewNo":0,"ppSeqNo":2,"discarded":1,"ledgerId":1,"op":"PREPREPARE","ppTime":1507741256,"digest":"9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741256159186]],"stateRootHash":null,"txnRootHash":null}' to Node1 2017-10-11 17:00:56,298 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":1,"viewNo":0,"ppSeqNo":2,"discarded":1,"ledgerId":1,"op":"PREPREPARE","ppTime":1507741256,"digest":"9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741256159186]],"stateRootHash":null,"txnRootHash":null}' to Node4 2017-10-11 17:00:56,299 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":1,"viewNo":0,"ppSeqNo":2,"discarded":1,"ledgerId":1,"op":"PREPREPARE","ppTime":1507741256,"digest":"9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741256159186]],"stateRootHash":null,"txnRootHash":null}' to Node4 2017-10-11 17:00:56,299 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":1,"viewNo":0,"ppSeqNo":2,"discarded":1,"ledgerId":1,"op":"PREPREPARE","ppTime":1507741256,"digest":"9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741256159186]],"stateRootHash":null,"txnRootHash":null}' to Node3 2017-10-11 17:00:56,300 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":1,"viewNo":0,"ppSeqNo":2,"discarded":1,"ledgerId":1,"op":"PREPREPARE","ppTime":1507741256,"digest":"9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741256159186]],"stateRootHash":null,"txnRootHash":null}' to Node3 2017-10-11 17:00:56,301 | DEBUG | replica.py ( 790) | processPrePrepare | Node2:0 received PRE-PREPARE(0, 2) from Node1:0 at 2270341.758838298 2017-10-11 17:00:56,302 | DEBUG | replica.py (1002) | validate_pre_prepare | Node2:0 state root before processing PREPREPARE{'instId': 0, 'stateRootHash': '8F8st3cEZDFKTucjctdXwyAddoLTXatnRtpS63YHxZYX', 'digest': '9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42', 'viewNo': 0, 'discarded': 1, 'ppSeqNo': 2, 'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741256159186)], 'ppTime': 1507741256, 'txnRootHash': 'CmvKha4N44FPRqPyp2VfEjjDq6FCSjfXPwBusRQz7sXw'} is b'\xfcLzv\x00\x9b#\xb5j\xc5\x1c\xde\x84a\xd6\xf5\xb0[\x9e\x9fB\x11\xc0\xb0\x13\xef\x84\x98@\xaa\x0e\xdc', 2zbyjmFFrtYpWE6aVggU7hoc4uz2B1K5BgNnExpnRRy3 2017-10-11 17:00:56,305 | DEBUG | replica.py ( 903) | doPrepare | Node2:0 Sending PREPARE(0, 2) at 2270341.762941835 2017-10-11 17:00:56,309 | INFO | replica.py (2064) | send | Node2:0 sending Prepare 2017-10-11 17:00:56,309 | TRACE | replica.py (2065) | send | Node2:0 sending PREPARE{'instId': 0, 'stateRootHash': '8F8st3cEZDFKTucjctdXwyAddoLTXatnRtpS63YHxZYX', 'viewNo': 0, 'ppTime': 1507741256, 'ppSeqNo': 2, 'digest': '9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42', 'txnRootHash': 'CmvKha4N44FPRqPyp2VfEjjDq6FCSjfXPwBusRQz7sXw'} 2017-10-11 17:00:56,313 | DEBUG | replica.py ( 886) | tryCommit | Node2:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 0, 'stateRootHash': '8F8st3cEZDFKTucjctdXwyAddoLTXatnRtpS63YHxZYX', 'viewNo': 0, 'ppTime': 1507741256, 'ppSeqNo': 2, 'digest': '9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42', 'txnRootHash': 'CmvKha4N44FPRqPyp2VfEjjDq6FCSjfXPwBusRQz7sXw'} 2017-10-11 17:00:56,313 | DEBUG | replica.py ( 594) | trackBatches | Node2:0 tracking batch for PREPREPARE{'instId': 0, 'stateRootHash': '8F8st3cEZDFKTucjctdXwyAddoLTXatnRtpS63YHxZYX', 'digest': '9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42', 'viewNo': 0, 'discarded': 1, 'ppSeqNo': 2, 'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741256159186)], 'ppTime': 1507741256, 'txnRootHash': 'CmvKha4N44FPRqPyp2VfEjjDq6FCSjfXPwBusRQz7sXw'} with state root b'\xfcLzv\x00\x9b#\xb5j\xc5\x1c\xde\x84a\xd6\xf5\xb0[\x9e\x9fB\x11\xc0\xb0\x13\xef\x84\x98@\xaa\x0e\xdc' 2017-10-11 17:00:56,313 | DEBUG | replica.py ( 809) | processPrePrepare | Node2:0 processed incoming PRE-PREPARE(0, 2) 2017-10-11 17:00:56,314 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 2 messages through listener 2017-10-11 17:00:56,320 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'instId': 0, 'stateRootHash': '8F8st3cEZDFKTucjctdXwyAddoLTXatnRtpS63YHxZYX', 'viewNo': 0, 'ppTime': 1507741256, 'ppSeqNo': 2, 'digest': '9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42', 'txnRootHash': 'CmvKha4N44FPRqPyp2VfEjjDq6FCSjfXPwBusRQz7sXw'} 2017-10-11 17:00:56,320 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'digest': '9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42', 'viewNo': 0, 'ppTime': 1507741256, 'ppSeqNo': 2, 'op': 'PREPARE', 'stateRootHash': '8F8st3cEZDFKTucjctdXwyAddoLTXatnRtpS63YHxZYX', 'txnRootHash': 'CmvKha4N44FPRqPyp2VfEjjDq6FCSjfXPwBusRQz7sXw'}, 'Node3') 2017-10-11 17:00:56,320 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'instId': 0, 'stateRootHash': '8F8st3cEZDFKTucjctdXwyAddoLTXatnRtpS63YHxZYX', 'viewNo': 0, 'ppTime': 1507741256, 'ppSeqNo': 2, 'digest': '9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42', 'txnRootHash': 'CmvKha4N44FPRqPyp2VfEjjDq6FCSjfXPwBusRQz7sXw'} 2017-10-11 17:00:56,321 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'instId': 0, 'stateRootHash': '8F8st3cEZDFKTucjctdXwyAddoLTXatnRtpS63YHxZYX', 'viewNo': 0, 'ppTime': 1507741256, 'ppSeqNo': 2, 'digest': '9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42', 'txnRootHash': 'CmvKha4N44FPRqPyp2VfEjjDq6FCSjfXPwBusRQz7sXw'} 2017-10-11 17:00:56,321 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'digest': '9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42', 'viewNo': 0, 'txnRootHash': 'CmvKha4N44FPRqPyp2VfEjjDq6FCSjfXPwBusRQz7sXw', 'ppSeqNo': 2, 'ppTime': 1507741256, 'op': 'PREPARE', 'stateRootHash': '8F8st3cEZDFKTucjctdXwyAddoLTXatnRtpS63YHxZYX'}, 'Node4') 2017-10-11 17:00:56,321 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'instId': 0, 'stateRootHash': '8F8st3cEZDFKTucjctdXwyAddoLTXatnRtpS63YHxZYX', 'viewNo': 0, 'ppTime': 1507741256, 'ppSeqNo': 2, 'digest': '9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42', 'txnRootHash': 'CmvKha4N44FPRqPyp2VfEjjDq6FCSjfXPwBusRQz7sXw'} 2017-10-11 17:00:56,324 | DEBUG | node.py (2593) | send | Node2 sending message PREPARE{'instId': 0, 'stateRootHash': '8F8st3cEZDFKTucjctdXwyAddoLTXatnRtpS63YHxZYX', 'viewNo': 0, 'ppTime': 1507741256, 'ppSeqNo': 2, 'digest': '9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42', 'txnRootHash': 'CmvKha4N44FPRqPyp2VfEjjDq6FCSjfXPwBusRQz7sXw'} to all recipients: ['Node1', 'Node4', 'Node3'] 2017-10-11 17:00:56,324 | DEBUG | replica.py ( 834) | processPrepare | Node2:0 received PREPARE(0, 2) from Node3:0 2017-10-11 17:00:56,324 | DEBUG | replica.py ( 922) | doCommit | Node2:0 Sending COMMIT(0, 2) at 2270341.782122178 2017-10-11 17:00:56,324 | INFO | replica.py (2064) | send | Node2:0 sending Commit 2017-10-11 17:00:56,325 | TRACE | replica.py (2065) | send | Node2:0 sending COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 2} 2017-10-11 17:00:56,325 | DEBUG | replica.py ( 898) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 2} commits where f is 1 2017-10-11 17:00:56,325 | DEBUG | replica.py ( 845) | processPrepare | Node2:0 processed incoming PREPARE (0, 2) 2017-10-11 17:00:56,325 | DEBUG | replica.py ( 834) | processPrepare | Node2:0 received PREPARE(0, 2) from Node4:0 2017-10-11 17:00:56,325 | DEBUG | replica.py ( 886) | tryCommit | Node2:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'instId': 0, 'stateRootHash': '8F8st3cEZDFKTucjctdXwyAddoLTXatnRtpS63YHxZYX', 'viewNo': 0, 'ppTime': 1507741256, 'ppSeqNo': 2, 'digest': '9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42', 'txnRootHash': 'CmvKha4N44FPRqPyp2VfEjjDq6FCSjfXPwBusRQz7sXw'} 2017-10-11 17:00:56,325 | DEBUG | replica.py ( 845) | processPrepare | Node2:0 processed incoming PREPARE (0, 2) 2017-10-11 17:00:56,326 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 2 messages through listener 2017-10-11 17:00:56,327 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: BATCH{'messages': ['{"op":"COMMIT","ppSeqNo":2,"viewNo":0,"instId":0}', '{"ppSeqNo":2,"op":"PREPARE","ppTime":1507741256,"stateRootHash":null,"viewNo":0,"digest":"9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42","txnRootHash":null,"instId":1}'], 'signature': None} 2017-10-11 17:00:56,327 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"op":"COMMIT","ppSeqNo":2,"viewNo":0,"instId":0}', '{"ppSeqNo":2,"op":"PREPARE","ppTime":1507741256,"stateRootHash":null,"viewNo":0,"digest":"9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42","txnRootHash":null,"instId":1}'], 'op': 'BATCH', 'signature': None}, 'Node3') 2017-10-11 17:00:56,327 | DEBUG | node.py (1328) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"op":"COMMIT","ppSeqNo":2,"viewNo":0,"instId":0}', '{"ppSeqNo":2,"op":"PREPARE","ppTime":1507741256,"stateRootHash":null,"viewNo":0,"digest":"9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42","txnRootHash":null,"instId":1}'], 'signature': None} 2017-10-11 17:00:56,327 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 2} 2017-10-11 17:00:56,327 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 2}, 'Node3') 2017-10-11 17:00:56,328 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 2} 2017-10-11 17:00:56,328 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'instId': 1, 'stateRootHash': None, 'viewNo': 0, 'ppTime': 1507741256, 'ppSeqNo': 2, 'digest': '9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42', 'txnRootHash': None} 2017-10-11 17:00:56,328 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'digest': '9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42', 'viewNo': 0, 'ppTime': 1507741256, 'ppSeqNo': 2, 'op': 'PREPARE', 'stateRootHash': None, 'txnRootHash': None}, 'Node3') 2017-10-11 17:00:56,328 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'instId': 1, 'stateRootHash': None, 'viewNo': 0, 'ppTime': 1507741256, 'ppSeqNo': 2, 'digest': '9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42', 'txnRootHash': None} 2017-10-11 17:00:56,328 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node4: BATCH{'messages': ['{"instId":0,"op":"COMMIT","ppSeqNo":2,"viewNo":0}', '{"viewNo":0,"instId":1,"op":"PREPARE","stateRootHash":null,"txnRootHash":null,"ppTime":1507741256,"digest":"9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42","ppSeqNo":2}'], 'signature': None} 2017-10-11 17:00:56,328 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"instId":0,"op":"COMMIT","ppSeqNo":2,"viewNo":0}', '{"viewNo":0,"instId":1,"op":"PREPARE","stateRootHash":null,"txnRootHash":null,"ppTime":1507741256,"digest":"9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42","ppSeqNo":2}'], 'op': 'BATCH', 'signature': None}, 'Node4') 2017-10-11 17:00:56,329 | DEBUG | node.py (1328) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"instId":0,"op":"COMMIT","ppSeqNo":2,"viewNo":0}', '{"viewNo":0,"instId":1,"op":"PREPARE","stateRootHash":null,"txnRootHash":null,"ppTime":1507741256,"digest":"9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42","ppSeqNo":2}'], 'signature': None} 2017-10-11 17:00:56,329 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 2} 2017-10-11 17:00:56,329 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 2}, 'Node4') 2017-10-11 17:00:56,329 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 2} 2017-10-11 17:00:56,329 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'instId': 1, 'stateRootHash': None, 'viewNo': 0, 'ppTime': 1507741256, 'ppSeqNo': 2, 'digest': '9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42', 'txnRootHash': None} 2017-10-11 17:00:56,329 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'digest': '9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42', 'viewNo': 0, 'txnRootHash': None, 'ppSeqNo': 2, 'ppTime': 1507741256, 'op': 'PREPARE', 'stateRootHash': None}, 'Node4') 2017-10-11 17:00:56,329 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'instId': 1, 'stateRootHash': None, 'viewNo': 0, 'ppTime': 1507741256, 'ppSeqNo': 2, 'digest': '9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42', 'txnRootHash': None} 2017-10-11 17:00:56,332 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":0,"viewNo":0,"ppSeqNo":2,"op":"PREPARE","ppTime":1507741256,"digest":"9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42","stateRootHash":"8F8st3cEZDFKTucjctdXwyAddoLTXatnRtpS63YHxZYX","txnRootHash":"CmvKha4N44FPRqPyp2VfEjjDq6FCSjfXPwBusRQz7sXw"}' to Node1 2017-10-11 17:00:56,332 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":0,"viewNo":0,"ppSeqNo":2,"op":"PREPARE","ppTime":1507741256,"digest":"9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42","stateRootHash":"8F8st3cEZDFKTucjctdXwyAddoLTXatnRtpS63YHxZYX","txnRootHash":"CmvKha4N44FPRqPyp2VfEjjDq6FCSjfXPwBusRQz7sXw"}' to Node1 2017-10-11 17:00:56,332 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":0,"viewNo":0,"ppSeqNo":2,"op":"PREPARE","ppTime":1507741256,"digest":"9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42","stateRootHash":"8F8st3cEZDFKTucjctdXwyAddoLTXatnRtpS63YHxZYX","txnRootHash":"CmvKha4N44FPRqPyp2VfEjjDq6FCSjfXPwBusRQz7sXw"}' to Node4 2017-10-11 17:00:56,332 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":0,"viewNo":0,"ppSeqNo":2,"op":"PREPARE","ppTime":1507741256,"digest":"9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42","stateRootHash":"8F8st3cEZDFKTucjctdXwyAddoLTXatnRtpS63YHxZYX","txnRootHash":"CmvKha4N44FPRqPyp2VfEjjDq6FCSjfXPwBusRQz7sXw"}' to Node4 2017-10-11 17:00:56,333 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":0,"viewNo":0,"ppSeqNo":2,"op":"PREPARE","ppTime":1507741256,"digest":"9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42","stateRootHash":"8F8st3cEZDFKTucjctdXwyAddoLTXatnRtpS63YHxZYX","txnRootHash":"CmvKha4N44FPRqPyp2VfEjjDq6FCSjfXPwBusRQz7sXw"}' to Node3 2017-10-11 17:00:56,333 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":0,"viewNo":0,"ppSeqNo":2,"op":"PREPARE","ppTime":1507741256,"digest":"9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42","stateRootHash":"8F8st3cEZDFKTucjctdXwyAddoLTXatnRtpS63YHxZYX","txnRootHash":"CmvKha4N44FPRqPyp2VfEjjDq6FCSjfXPwBusRQz7sXw"}' to Node3 2017-10-11 17:00:56,335 | DEBUG | node.py (2593) | send | Node2 sending message COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 2} to all recipients: ['Node1', 'Node4', 'Node3'] 2017-10-11 17:00:56,336 | DEBUG | replica.py ( 863) | processCommit | Node2:0 received COMMIT(0, 2) from Node3:0 2017-10-11 17:00:56,336 | DEBUG | replica.py ( 898) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 2} commits where f is 1 2017-10-11 17:00:56,336 | DEBUG | replica.py ( 874) | processCommit | Node2:0 processed incoming COMMIT(0, 2) 2017-10-11 17:00:56,336 | DEBUG | replica.py ( 863) | processCommit | Node2:0 received COMMIT(0, 2) from Node4:0 2017-10-11 17:00:56,336 | TRACE | replica.py ( 894) | tryOrder | Node2:0 returning request to node 2017-10-11 17:00:56,336 | INFO | replica.py (1395) | doOrder | Node2:0 ordering COMMIT(0, 2) 2017-10-11 17:00:56,337 | DEBUG | replica.py ( 310) | last_ordered_3pc | Node2:0 set last ordered as (0, 2) 2017-10-11 17:00:56,337 | INFO | replica.py (2064) | send | Node2:0 sending Ordered 2017-10-11 17:00:56,337 | TRACE | replica.py (2065) | send | Node2:0 sending ORDERED{'instId': 0, 'viewNo': 0, 'stateRootHash': '8F8st3cEZDFKTucjctdXwyAddoLTXatnRtpS63YHxZYX', 'ppSeqNo': 2, 'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741256159186)], 'ppTime': 1507741256, 'txnRootHash': 'CmvKha4N44FPRqPyp2VfEjjDq6FCSjfXPwBusRQz7sXw'} 2017-10-11 17:00:56,337 | DEBUG | replica.py (1426) | order_3pc_key | Node2:0 ordered request (0, 2) 2017-10-11 17:00:56,337 | DEBUG | replica.py ( 874) | processCommit | Node2:0 processed incoming COMMIT(0, 2) 2017-10-11 17:00:56,337 | DEBUG | replica.py ( 834) | processPrepare | Node2:1 received PREPARE(0, 2) from Node3:1 2017-10-11 17:00:56,338 | DEBUG | replica.py ( 886) | tryCommit | Node2:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 1, 'stateRootHash': None, 'viewNo': 0, 'ppTime': 1507741256, 'ppSeqNo': 2, 'digest': '9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42', 'txnRootHash': None} 2017-10-11 17:00:56,338 | DEBUG | replica.py ( 845) | processPrepare | Node2:1 processed incoming PREPARE (0, 2) 2017-10-11 17:00:56,338 | DEBUG | replica.py ( 834) | processPrepare | Node2:1 received PREPARE(0, 2) from Node4:1 2017-10-11 17:00:56,338 | DEBUG | replica.py ( 922) | doCommit | Node2:1 Sending COMMIT(0, 2) at 2270341.796297198 2017-10-11 17:00:56,339 | INFO | replica.py (2064) | send | Node2:1 sending Commit 2017-10-11 17:00:56,339 | TRACE | replica.py (2065) | send | Node2:1 sending COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 2} 2017-10-11 17:00:56,339 | DEBUG | replica.py ( 898) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 2} commits where f is 1 2017-10-11 17:00:56,339 | DEBUG | replica.py ( 845) | processPrepare | Node2:1 processed incoming PREPARE (0, 2) 2017-10-11 17:00:56,345 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node1 2017-10-11 17:00:56,345 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node1 2017-10-11 17:00:56,345 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node4 2017-10-11 17:00:56,346 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node4 2017-10-11 17:00:56,346 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node3 2017-10-11 17:00:56,346 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node3 2017-10-11 17:00:56,347 | DEBUG | node.py (1881) | processOrdered | Node2 executing Ordered batch 0 2 of 1 requests 2017-10-11 17:00:56,351 | DEBUG | ledger.py ( 68) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-10-11 17:00:56,352 | DEBUG | node.py (2374) | sendReplyToClient | Node2 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1507741256159186) to client 2017-10-11 17:00:56,352 | TRACE | zstack.py ( 726) | transmitThroughListener | Node2C transmitting b'{"op":"REPLY","result":{"type":"1","signature":"2ACvk8ddWAa5daTzD8prMaa4Eig62fv5WcDpth9QqCZe6PvR9UMnXXU9CTcWQyq9P3HcDV4naXfg7Syy9h22Q3R3","dest":"542MVr22zcHbVyGzaXmbT2","auditPath":["6BTmbK7RWRUL8AmtzKmreZKwvp2VDHorHRND22Jqr6pW","HYRnUmRJ2hQMtr1smmXw3sqWyQyKTzCeTh5D9EweiD8z"],"seqNo":11,"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1507741256159186,"txnTime":1507741256,"rootHash":"CmvKha4N44FPRqPyp2VfEjjDq6FCSjfXPwBusRQz7sXw"}}' to b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' through listener socket 2017-10-11 17:00:56,352 | DEBUG | node.py (2295) | executeBatch | Node2 storing 3PC key (0, 2) for ledger 1 range (11, 11) 2017-10-11 17:00:56,353 | DEBUG | node.py (2593) | send | Node2 sending message COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 2} to all recipients: ['Node1', 'Node4', 'Node3'] 2017-10-11 17:00:56,354 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 2 messages through listener 2017-10-11 17:00:56,355 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 2} 2017-10-11 17:00:56,355 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 2}, 'Node4') 2017-10-11 17:00:56,355 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 2} 2017-10-11 17:00:56,356 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node1: BATCH{'messages': ['{"viewNo":0,"op":"COMMIT","instId":0,"ppSeqNo":2}', '{"instId":1,"digest":"9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42","viewNo":0,"op":"PREPARE","ppTime":1507741256,"ppSeqNo":2,"stateRootHash":null,"txnRootHash":null}'], 'signature': None} 2017-10-11 17:00:56,356 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"viewNo":0,"op":"COMMIT","instId":0,"ppSeqNo":2}', '{"instId":1,"digest":"9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42","viewNo":0,"op":"PREPARE","ppTime":1507741256,"ppSeqNo":2,"stateRootHash":null,"txnRootHash":null}'], 'op': 'BATCH', 'signature': None}, 'Node1') 2017-10-11 17:00:56,356 | DEBUG | node.py (1328) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"viewNo":0,"op":"COMMIT","instId":0,"ppSeqNo":2}', '{"instId":1,"digest":"9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42","viewNo":0,"op":"PREPARE","ppTime":1507741256,"ppSeqNo":2,"stateRootHash":null,"txnRootHash":null}'], 'signature': None} 2017-10-11 17:00:56,357 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 2} 2017-10-11 17:00:56,357 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 2}, 'Node1') 2017-10-11 17:00:56,357 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 2} 2017-10-11 17:00:56,357 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node1: PREPARE{'instId': 1, 'stateRootHash': None, 'viewNo': 0, 'ppTime': 1507741256, 'ppSeqNo': 2, 'digest': '9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42', 'txnRootHash': None} 2017-10-11 17:00:56,357 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'stateRootHash': None, 'viewNo': 0, 'digest': '9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42', 'ppSeqNo': 2, 'op': 'PREPARE', 'ppTime': 1507741256, 'txnRootHash': None}, 'Node1') 2017-10-11 17:00:56,357 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'instId': 1, 'stateRootHash': None, 'viewNo': 0, 'ppTime': 1507741256, 'ppSeqNo': 2, 'digest': '9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42', 'txnRootHash': None} 2017-10-11 17:00:56,361 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node1 2017-10-11 17:00:56,362 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node1 2017-10-11 17:00:56,362 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node4 2017-10-11 17:00:56,363 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node4 2017-10-11 17:00:56,363 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node3 2017-10-11 17:00:56,364 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":2}' to Node3 2017-10-11 17:00:56,365 | TRACE | message_processor.py ( 28) | discard | Node2:0 discarding message COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 2} because already ordered 3 phase message 2017-10-11 17:00:56,365 | DEBUG | replica.py ( 863) | processCommit | Node2:1 received COMMIT(0, 2) from Node4:1 2017-10-11 17:00:56,366 | DEBUG | replica.py ( 898) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 2} commits where f is 1 2017-10-11 17:00:56,366 | DEBUG | replica.py ( 874) | processCommit | Node2:1 processed incoming COMMIT(0, 2) 2017-10-11 17:00:56,367 | DEBUG | replica.py ( 834) | processPrepare | Node2:1 received PREPARE(0, 2) from Node1:1 2017-10-11 17:00:56,367 | DEBUG | replica.py ( 886) | tryCommit | Node2:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'instId': 1, 'stateRootHash': None, 'viewNo': 0, 'ppTime': 1507741256, 'ppSeqNo': 2, 'digest': '9f801966da1529fe8b7a50b3fb23ec25a7e1e927dbb727a6d9580224f7bfad42', 'txnRootHash': None} 2017-10-11 17:00:56,368 | DEBUG | replica.py ( 845) | processPrepare | Node2:1 processed incoming PREPARE (0, 2) 2017-10-11 17:00:56,375 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 2 messages through listener 2017-10-11 17:00:56,376 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 2} 2017-10-11 17:00:56,376 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 2}, 'Node3') 2017-10-11 17:00:56,376 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 2} 2017-10-11 17:00:56,376 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 2} 2017-10-11 17:00:56,377 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 2}, 'Node1') 2017-10-11 17:00:56,377 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 2} 2017-10-11 17:00:56,383 | DEBUG | replica.py ( 863) | processCommit | Node2:1 received COMMIT(0, 2) from Node3:1 2017-10-11 17:00:56,383 | TRACE | replica.py ( 894) | tryOrder | Node2:1 returning request to node 2017-10-11 17:00:56,383 | INFO | replica.py (1395) | doOrder | Node2:1 ordering COMMIT(0, 2) 2017-10-11 17:00:56,383 | DEBUG | replica.py ( 310) | last_ordered_3pc | Node2:1 set last ordered as (0, 2) 2017-10-11 17:00:56,384 | INFO | replica.py (2064) | send | Node2:1 sending Ordered 2017-10-11 17:00:56,384 | TRACE | replica.py (2065) | send | Node2:1 sending ORDERED{'instId': 1, 'viewNo': 0, 'stateRootHash': None, 'ppSeqNo': 2, 'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741256159186)], 'ppTime': 1507741256, 'txnRootHash': None} 2017-10-11 17:00:56,384 | DEBUG | replica.py (1426) | order_3pc_key | Node2:1 ordered request (0, 2) 2017-10-11 17:00:56,384 | DEBUG | replica.py ( 874) | processCommit | Node2:1 processed incoming COMMIT(0, 2) 2017-10-11 17:00:56,384 | TRACE | message_processor.py ( 28) | discard | Node2:1 discarding message COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 2} because already ordered 3 phase message 2017-10-11 17:00:56,390 | TRACE | node.py (1892) | processOrdered | Node2 got ordered requests from backup replica 1 2017-10-11 17:00:56,391 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:00:56,403 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.182 seconds 2017-10-11 17:01:01,961 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkPerformance with id 12 2017-10-11 17:01:01,962 | TRACE | node.py (2005) | checkPerformance | Node2 checking its performance 2017-10-11 17:01:01,967 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:01:01,967 | TRACE | monitor.py ( 299) | isMasterThroughputTooLow | Node2 master throughput ratio 1.1394943028750164 is acceptable. 2017-10-11 17:01:01,968 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:01:01,968 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:01:01,968 | DEBUG | node.py (2022) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:01:01,969 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 13 to run in 10 seconds 2017-10-11 17:01:06,930 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:01:06,930 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:01:06,931 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:01:06,932 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:01:11,987 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkPerformance with id 13 2017-10-11 17:01:11,988 | TRACE | node.py (2005) | checkPerformance | Node2 checking its performance 2017-10-11 17:01:11,988 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:01:11,989 | TRACE | monitor.py ( 299) | isMasterThroughputTooLow | Node2 master throughput ratio 1.1394943028750164 is acceptable. 2017-10-11 17:01:11,989 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:01:11,990 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:01:11,991 | DEBUG | node.py (2022) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:01:11,991 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 14 to run in 10 seconds 2017-10-11 17:01:21,889 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkNodeRequestSpike with id 8 2017-10-11 17:01:21,890 | DEBUG | node.py (2026) | checkNodeRequestSpike | Node2 checking its request amount 2017-10-11 17:01:21,890 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:01:21,890 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 15 to run in 60 seconds 2017-10-11 17:01:21,891 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkPerformance with id 2 2017-10-11 17:01:21,891 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:01:21,891 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 3 to run in 60 seconds 2017-10-11 17:01:21,935 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:01:21,936 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:01:21,936 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:01:21,937 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:01:21,993 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkPerformance with id 14 2017-10-11 17:01:21,993 | TRACE | node.py (2005) | checkPerformance | Node2 checking its performance 2017-10-11 17:01:21,993 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:01:21,993 | TRACE | monitor.py ( 299) | isMasterThroughputTooLow | Node2 master throughput ratio 1.1394943028750164 is acceptable. 2017-10-11 17:01:21,993 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:01:21,993 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:01:21,994 | DEBUG | node.py (2022) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:01:21,994 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 16 to run in 10 seconds 2017-10-11 17:01:32,006 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkPerformance with id 16 2017-10-11 17:01:32,007 | TRACE | node.py (2005) | checkPerformance | Node2 checking its performance 2017-10-11 17:01:32,007 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:01:32,007 | TRACE | monitor.py ( 299) | isMasterThroughputTooLow | Node2 master throughput ratio 1.1394943028750164 is acceptable. 2017-10-11 17:01:32,007 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:01:32,007 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:01:32,007 | DEBUG | node.py (2022) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:01:32,007 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 17 to run in 10 seconds 2017-10-11 17:01:36,941 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:01:36,941 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:01:36,941 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:01:36,942 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:01:42,014 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkPerformance with id 17 2017-10-11 17:01:42,014 | TRACE | node.py (2005) | checkPerformance | Node2 checking its performance 2017-10-11 17:01:42,014 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:01:42,014 | TRACE | monitor.py ( 299) | isMasterThroughputTooLow | Node2 master throughput ratio 1.1394943028750164 is acceptable. 2017-10-11 17:01:42,014 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:01:42,014 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:01:42,015 | DEBUG | node.py (2022) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:01:42,015 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 18 to run in 10 seconds 2017-10-11 17:01:51,946 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:01:51,946 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:01:51,946 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:01:51,947 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:01:52,016 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkPerformance with id 18 2017-10-11 17:01:52,016 | TRACE | node.py (2005) | checkPerformance | Node2 checking its performance 2017-10-11 17:01:52,016 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:01:52,017 | TRACE | monitor.py ( 299) | isMasterThroughputTooLow | Node2 master throughput ratio 1.1394943028750164 is acceptable. 2017-10-11 17:01:52,017 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:01:52,017 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:01:52,017 | DEBUG | node.py (2022) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:01:52,018 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 19 to run in 10 seconds 2017-10-11 17:02:02,023 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkPerformance with id 19 2017-10-11 17:02:02,024 | TRACE | node.py (2005) | checkPerformance | Node2 checking its performance 2017-10-11 17:02:02,024 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:02:02,024 | TRACE | monitor.py ( 299) | isMasterThroughputTooLow | Node2 master throughput ratio 1.1394943028750164 is acceptable. 2017-10-11 17:02:02,024 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:02:02,025 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:02:02,025 | DEBUG | node.py (2022) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:02:02,025 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 20 to run in 10 seconds 2017-10-11 17:02:06,953 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:02:06,953 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:02:06,953 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:02:06,954 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:02:12,032 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkPerformance with id 20 2017-10-11 17:02:12,033 | TRACE | node.py (2005) | checkPerformance | Node2 checking its performance 2017-10-11 17:02:12,034 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:02:12,035 | TRACE | monitor.py ( 299) | isMasterThroughputTooLow | Node2 master throughput ratio 1.1394943028750164 is acceptable. 2017-10-11 17:02:12,035 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:02:12,035 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:02:12,036 | DEBUG | node.py (2022) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:02:12,036 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 21 to run in 10 seconds 2017-10-11 17:02:21,891 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkNodeRequestSpike with id 15 2017-10-11 17:02:21,891 | DEBUG | node.py (2026) | checkNodeRequestSpike | Node2 checking its request amount 2017-10-11 17:02:21,891 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:02:21,891 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 22 to run in 60 seconds 2017-10-11 17:02:21,891 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkPerformance with id 3 2017-10-11 17:02:21,892 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:02:21,892 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 4 to run in 60 seconds 2017-10-11 17:02:21,966 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:02:21,967 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:02:21,967 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:02:21,967 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:02:22,040 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkPerformance with id 21 2017-10-11 17:02:22,041 | TRACE | node.py (2005) | checkPerformance | Node2 checking its performance 2017-10-11 17:02:22,041 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:02:22,041 | TRACE | monitor.py ( 299) | isMasterThroughputTooLow | Node2 master throughput ratio 1.1394943028750164 is acceptable. 2017-10-11 17:02:22,041 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:02:22,041 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:02:22,041 | DEBUG | node.py (2022) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:02:22,041 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 23 to run in 10 seconds 2017-10-11 17:02:32,050 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkPerformance with id 23 2017-10-11 17:02:32,050 | TRACE | node.py (2005) | checkPerformance | Node2 checking its performance 2017-10-11 17:02:32,050 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:02:32,050 | TRACE | monitor.py ( 299) | isMasterThroughputTooLow | Node2 master throughput ratio 1.1394943028750164 is acceptable. 2017-10-11 17:02:32,051 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:02:32,051 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:02:32,051 | DEBUG | node.py (2022) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:02:32,051 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 24 to run in 10 seconds 2017-10-11 17:02:36,972 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:02:36,972 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:02:36,973 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:02:36,973 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:02:42,060 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkPerformance with id 24 2017-10-11 17:02:42,062 | TRACE | node.py (2005) | checkPerformance | Node2 checking its performance 2017-10-11 17:02:42,062 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:02:42,063 | TRACE | monitor.py ( 299) | isMasterThroughputTooLow | Node2 master throughput ratio 1.1394943028750164 is acceptable. 2017-10-11 17:02:42,064 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:02:42,064 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:02:42,065 | DEBUG | node.py (2022) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:02:42,065 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 25 to run in 10 seconds 2017-10-11 17:02:51,988 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:02:51,988 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:02:51,990 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:02:51,991 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:02:52,082 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkPerformance with id 25 2017-10-11 17:02:52,082 | TRACE | node.py (2005) | checkPerformance | Node2 checking its performance 2017-10-11 17:02:52,082 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:02:52,082 | TRACE | monitor.py ( 299) | isMasterThroughputTooLow | Node2 master throughput ratio 1.1394943028750164 is acceptable. 2017-10-11 17:02:52,082 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:02:52,083 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:02:52,083 | DEBUG | node.py (2022) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:02:52,083 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 26 to run in 10 seconds 2017-10-11 17:03:02,096 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkPerformance with id 26 2017-10-11 17:03:02,097 | TRACE | node.py (2005) | checkPerformance | Node2 checking its performance 2017-10-11 17:03:02,097 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:03:02,097 | TRACE | monitor.py ( 299) | isMasterThroughputTooLow | Node2 master throughput ratio 1.1394943028750164 is acceptable. 2017-10-11 17:03:02,097 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:03:02,098 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:03:02,098 | DEBUG | node.py (2022) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:03:02,098 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 27 to run in 10 seconds 2017-10-11 17:03:07,003 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:03:07,004 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:03:07,004 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:03:07,004 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:03:12,104 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkPerformance with id 27 2017-10-11 17:03:12,104 | TRACE | node.py (2005) | checkPerformance | Node2 checking its performance 2017-10-11 17:03:12,105 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:03:12,105 | TRACE | monitor.py ( 299) | isMasterThroughputTooLow | Node2 master throughput ratio 1.1394943028750164 is acceptable. 2017-10-11 17:03:12,105 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:03:12,106 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:03:12,106 | DEBUG | node.py (2022) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:03:12,106 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 28 to run in 10 seconds 2017-10-11 17:03:21,898 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkNodeRequestSpike with id 22 2017-10-11 17:03:21,898 | DEBUG | node.py (2026) | checkNodeRequestSpike | Node2 checking its request amount 2017-10-11 17:03:21,899 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:03:21,899 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 29 to run in 60 seconds 2017-10-11 17:03:21,899 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkPerformance with id 4 2017-10-11 17:03:21,899 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:03:21,899 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 5 to run in 60 seconds 2017-10-11 17:03:22,015 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:03:22,015 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:03:22,015 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:03:22,016 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:03:22,120 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkPerformance with id 28 2017-10-11 17:03:22,120 | TRACE | node.py (2005) | checkPerformance | Node2 checking its performance 2017-10-11 17:03:22,121 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:03:22,121 | TRACE | monitor.py ( 299) | isMasterThroughputTooLow | Node2 master throughput ratio 1.1394943028750164 is acceptable. 2017-10-11 17:03:22,121 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:03:22,122 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:03:22,122 | DEBUG | node.py (2022) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:03:22,122 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 30 to run in 10 seconds 2017-10-11 17:03:32,123 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkPerformance with id 30 2017-10-11 17:03:32,123 | TRACE | node.py (2005) | checkPerformance | Node2 checking its performance 2017-10-11 17:03:32,123 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:03:32,123 | TRACE | monitor.py ( 299) | isMasterThroughputTooLow | Node2 master throughput ratio 1.1394943028750164 is acceptable. 2017-10-11 17:03:32,123 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:03:32,124 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:03:32,124 | DEBUG | node.py (2022) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:03:32,124 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 31 to run in 10 seconds 2017-10-11 17:03:33,777 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2C got 1 messages through listener 2017-10-11 17:03:33,778 | TRACE | signing.py ( 98) | serializeMsg | serialized msg {'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741413733229, 'operation': {'type': '109', 'name': 'upgrade37', 'justification': None, 'timeout': 10, 'force': True, 'version': '1.1.37', 'schedule': {'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'action': 'start'}} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:action:start|force:True|justification:|name:upgrade37|schedule:4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA:2017-10-11T14:30:00.258870+00:00|8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb:2017-10-11T14:20:00.258870+00:00|DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya:2017-10-11T14:25:00.258870+00:00|Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv:2017-10-11T14:15:00.258870+00:00|sha256:ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f|timeout:10|type:109|version:1.1.37|reqId:1507741413733229 2017-10-11 17:03:33,780 | TRACE | signing.py ( 98) | serializeMsg | serialized msg {'signature': '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741413733229, 'operation': {'type': '109', 'name': 'upgrade37', 'justification': None, 'timeout': 10, 'force': True, 'version': '1.1.37', 'schedule': {'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'action': 'start'}} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:action:start|force:True|justification:|name:upgrade37|schedule:4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA:2017-10-11T14:30:00.258870+00:00|8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb:2017-10-11T14:20:00.258870+00:00|DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya:2017-10-11T14:25:00.258870+00:00|Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv:2017-10-11T14:15:00.258870+00:00|sha256:ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f|timeout:10|type:109|version:1.1.37|reqId:1507741413733229 2017-10-11 17:03:33,781 | INFO | node.py (2250) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1507741413733229 2017-10-11 17:03:33,781 | TRACE | node.py (1444) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'signature': '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741413733229, 'operation': {'type': '109', 'name': 'upgrade37', 'justification': None, 'timeout': 10, 'force': True, 'version': '1.1.37', 'schedule': {'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'action': 'start'}} 2017-10-11 17:03:33,782 | DISPLAY | node.py (1490) | processClientInBox | Node2C processing b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' request SafeRequest: {'signature': '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741413733229, 'operation': {'type': '109', 'name': 'upgrade37', 'justification': None, 'timeout': 10, 'force': True, 'version': '1.1.37', 'schedule': {'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'action': 'start'}} 2017-10-11 17:03:33,783 | INFO | upgrader.py ( 308) | _scheduleUpgrade | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb's upgrader processing upgrade for version 1.1.37 2017-10-11 17:03:33,795 | INFO | upgrader.py ( 369) | _callUpgradeAgent | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb's upgrader calling agent for upgrade 2017-10-11 17:03:33,796 | DEBUG | node.py (1751) | processRequest | Node2 received client request: SafeRequest: {'signature': '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741413733229, 'operation': {'type': '109', 'name': 'upgrade37', 'justification': None, 'timeout': 10, 'force': True, 'version': '1.1.37', 'schedule': {'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'action': 'start'}} from b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' 2017-10-11 17:03:33,796 | TRACE | propagator.py ( 159) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'signature': '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741413733229, 'operation': {'type': '109', 'name': 'upgrade37', 'justification': None, 'timeout': 10, 'force': True, 'version': '1.1.37', 'schedule': {'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'action': 'start'}} 2017-10-11 17:03:33,797 | INFO | propagator.py ( 145) | propagate | Node2 propagating V4SGRU86Z58d6TV7PBUe6f request 1507741413733229 from client b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' 2017-10-11 17:03:33,797 | DEBUG | node.py (2593) | send | Node2 sending message PROPAGATE{'request': {'signature': '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741413733229, 'operation': {'type': '109', 'name': 'upgrade37', 'justification': None, 'timeout': 10, 'force': True, 'version': '1.1.37', 'schedule': {'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'action': 'start'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'} to all recipients: ['Node1', 'Node4', 'Node3'] 2017-10-11 17:03:33,797 | DEBUG | propagator.py ( 241) | tryForwarding | Node2 not forwarding request SafeRequest: {'signature': '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741413733229, 'operation': {'type': '109', 'name': 'upgrade37', 'justification': None, 'timeout': 10, 'force': True, 'version': '1.1.37', 'schedule': {'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'action': 'start'}} to its replicas since not finalised 2017-10-11 17:03:33,797 | TRACE | zstack.py ( 726) | transmitThroughListener | Node2C transmitting b'{"op":"REQACK","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1507741413733229}' to b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' through listener socket 2017-10-11 17:03:33,798 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"request":{"signature":"4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1507741413733229,"operation":{"type":"109","name":"upgrade37","justification":null,"timeout":10,"force":true,"version":"1.1.37","schedule":{"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya":"2017-10-11T14:25:00.258870+00:00","8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb":"2017-10-11T14:20:00.258870+00:00","Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv":"2017-10-11T14:15:00.258870+00:00","4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA":"2017-10-11T14:30:00.258870+00:00"},"sha256":"ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f","action":"start"}},"op":"PROPAGATE","senderClient":"fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%"}' to Node1 2017-10-11 17:03:33,807 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"request":{"signature":"4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1507741413733229,"operation":{"type":"109","name":"upgrade37","justification":null,"timeout":10,"force":true,"version":"1.1.37","schedule":{"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya":"2017-10-11T14:25:00.258870+00:00","8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb":"2017-10-11T14:20:00.258870+00:00","Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv":"2017-10-11T14:15:00.258870+00:00","4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA":"2017-10-11T14:30:00.258870+00:00"},"sha256":"ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f","action":"start"}},"op":"PROPAGATE","senderClient":"fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%"}' to Node1 2017-10-11 17:03:33,807 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"request":{"signature":"4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1507741413733229,"operation":{"type":"109","name":"upgrade37","justification":null,"timeout":10,"force":true,"version":"1.1.37","schedule":{"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya":"2017-10-11T14:25:00.258870+00:00","8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb":"2017-10-11T14:20:00.258870+00:00","Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv":"2017-10-11T14:15:00.258870+00:00","4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA":"2017-10-11T14:30:00.258870+00:00"},"sha256":"ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f","action":"start"}},"op":"PROPAGATE","senderClient":"fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%"}' to Node4 2017-10-11 17:03:33,808 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"request":{"signature":"4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1507741413733229,"operation":{"type":"109","name":"upgrade37","justification":null,"timeout":10,"force":true,"version":"1.1.37","schedule":{"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya":"2017-10-11T14:25:00.258870+00:00","8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb":"2017-10-11T14:20:00.258870+00:00","Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv":"2017-10-11T14:15:00.258870+00:00","4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA":"2017-10-11T14:30:00.258870+00:00"},"sha256":"ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f","action":"start"}},"op":"PROPAGATE","senderClient":"fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%"}' to Node4 2017-10-11 17:03:33,808 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"request":{"signature":"4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1507741413733229,"operation":{"type":"109","name":"upgrade37","justification":null,"timeout":10,"force":true,"version":"1.1.37","schedule":{"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya":"2017-10-11T14:25:00.258870+00:00","8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb":"2017-10-11T14:20:00.258870+00:00","Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv":"2017-10-11T14:15:00.258870+00:00","4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA":"2017-10-11T14:30:00.258870+00:00"},"sha256":"ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f","action":"start"}},"op":"PROPAGATE","senderClient":"fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%"}' to Node3 2017-10-11 17:03:33,808 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"request":{"signature":"4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz","identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1507741413733229,"operation":{"type":"109","name":"upgrade37","justification":null,"timeout":10,"force":true,"version":"1.1.37","schedule":{"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya":"2017-10-11T14:25:00.258870+00:00","8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb":"2017-10-11T14:20:00.258870+00:00","Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv":"2017-10-11T14:15:00.258870+00:00","4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA":"2017-10-11T14:30:00.258870+00:00"},"sha256":"ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f","action":"start"}},"op":"PROPAGATE","senderClient":"fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%"}' to Node3 2017-10-11 17:03:33,810 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 2 messages through listener 2017-10-11 17:03:33,825 | TRACE | signing.py ( 98) | serializeMsg | serialized msg {'reqId': 1507741413733229, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz', 'operation': {'type': '109', 'name': 'upgrade37', 'action': 'start', 'timeout': 10, 'version': '1.1.37', 'force': True, 'justification': None, 'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'}} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:action:start|force:True|justification:|name:upgrade37|schedule:4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA:2017-10-11T14:30:00.258870+00:00|8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb:2017-10-11T14:20:00.258870+00:00|DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya:2017-10-11T14:25:00.258870+00:00|Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv:2017-10-11T14:15:00.258870+00:00|sha256:ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f|timeout:10|type:109|version:1.1.37|reqId:1507741413733229 2017-10-11 17:03:33,825 | INFO | node.py (2250) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1507741413733229 2017-10-11 17:03:33,825 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: PROPAGATE{'request': {'reqId': 1507741413733229, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz', 'operation': {'type': '109', 'name': 'upgrade37', 'action': 'start', 'timeout': 10, 'version': '1.1.37', 'force': True, 'justification': None, 'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'} 2017-10-11 17:03:33,825 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'request': {'reqId': 1507741413733229, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz', 'operation': {'type': '109', 'name': 'upgrade37', 'action': 'start', 'timeout': 10, 'version': '1.1.37', 'force': True, 'justification': None, 'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'}, 'Node3') 2017-10-11 17:03:33,826 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'request': {'reqId': 1507741413733229, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz', 'operation': {'type': '109', 'name': 'upgrade37', 'action': 'start', 'timeout': 10, 'version': '1.1.37', 'force': True, 'justification': None, 'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'} 2017-10-11 17:03:33,826 | TRACE | signing.py ( 98) | serializeMsg | serialized msg {'reqId': 1507741413733229, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz', 'operation': {'type': '109', 'name': 'upgrade37', 'justification': None, 'timeout': 10, 'force': True, 'version': '1.1.37', 'schedule': {'4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'action': 'start'}} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:action:start|force:True|justification:|name:upgrade37|schedule:4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA:2017-10-11T14:30:00.258870+00:00|8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb:2017-10-11T14:20:00.258870+00:00|DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya:2017-10-11T14:25:00.258870+00:00|Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv:2017-10-11T14:15:00.258870+00:00|sha256:ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f|timeout:10|type:109|version:1.1.37|reqId:1507741413733229 2017-10-11 17:03:33,843 | INFO | node.py (2250) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1507741413733229 2017-10-11 17:03:33,843 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node4: PROPAGATE{'request': {'reqId': 1507741413733229, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz', 'operation': {'type': '109', 'name': 'upgrade37', 'justification': None, 'timeout': 10, 'force': True, 'version': '1.1.37', 'schedule': {'4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'action': 'start'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'} 2017-10-11 17:03:33,844 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'request': {'reqId': 1507741413733229, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz', 'operation': {'type': '109', 'name': 'upgrade37', 'justification': None, 'timeout': 10, 'force': True, 'version': '1.1.37', 'schedule': {'4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'action': 'start'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'}, 'Node4') 2017-10-11 17:03:33,844 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'request': {'reqId': 1507741413733229, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz', 'operation': {'type': '109', 'name': 'upgrade37', 'justification': None, 'timeout': 10, 'force': True, 'version': '1.1.37', 'schedule': {'4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'action': 'start'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'} 2017-10-11 17:03:33,844 | DEBUG | node.py (1799) | processPropagate | Node Node2 received propagated request: PROPAGATE{'request': {'reqId': 1507741413733229, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz', 'operation': {'type': '109', 'name': 'upgrade37', 'action': 'start', 'timeout': 10, 'version': '1.1.37', 'force': True, 'justification': None, 'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'} 2017-10-11 17:03:33,845 | TRACE | signing.py ( 98) | serializeMsg | serialized msg {'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741413733229, 'operation': {'type': '109', 'name': 'upgrade37', 'action': 'start', 'timeout': 10, 'version': '1.1.37', 'force': True, 'justification': None, 'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'}} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:action:start|force:True|justification:|name:upgrade37|schedule:4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA:2017-10-11T14:30:00.258870+00:00|8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb:2017-10-11T14:20:00.258870+00:00|DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya:2017-10-11T14:25:00.258870+00:00|Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv:2017-10-11T14:15:00.258870+00:00|sha256:ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f|timeout:10|type:109|version:1.1.37|reqId:1507741413733229 2017-10-11 17:03:33,845 | TRACE | propagator.py ( 138) | propagate | Node2 already propagated SafeRequest: {'signature': '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741413733229, 'operation': {'type': '109', 'name': 'upgrade37', 'action': 'start', 'timeout': 10, 'version': '1.1.37', 'force': True, 'justification': None, 'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'}} 2017-10-11 17:03:33,845 | DEBUG | propagator.py ( 208) | forward | Node2 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1507741413733229) to 2 replicas 2017-10-11 17:03:33,846 | DEBUG | node.py (1799) | processPropagate | Node Node2 received propagated request: PROPAGATE{'request': {'reqId': 1507741413733229, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz', 'operation': {'type': '109', 'name': 'upgrade37', 'justification': None, 'timeout': 10, 'force': True, 'version': '1.1.37', 'schedule': {'4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'action': 'start'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'} 2017-10-11 17:03:33,846 | TRACE | signing.py ( 98) | serializeMsg | serialized msg {'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741413733229, 'operation': {'type': '109', 'name': 'upgrade37', 'justification': None, 'timeout': 10, 'force': True, 'version': '1.1.37', 'schedule': {'4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'action': 'start'}} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:action:start|force:True|justification:|name:upgrade37|schedule:4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA:2017-10-11T14:30:00.258870+00:00|8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb:2017-10-11T14:20:00.258870+00:00|DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya:2017-10-11T14:25:00.258870+00:00|Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv:2017-10-11T14:15:00.258870+00:00|sha256:ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f|timeout:10|type:109|version:1.1.37|reqId:1507741413733229 2017-10-11 17:03:33,846 | TRACE | propagator.py ( 138) | propagate | Node2 already propagated SafeRequest: {'signature': '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741413733229, 'operation': {'type': '109', 'name': 'upgrade37', 'justification': None, 'timeout': 10, 'force': True, 'version': '1.1.37', 'schedule': {'4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'action': 'start'}} 2017-10-11 17:03:33,846 | DEBUG | propagator.py ( 241) | tryForwarding | Node2 not forwarding request SafeRequest: {'signature': '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741413733229, 'operation': {'type': '109', 'name': 'upgrade37', 'justification': None, 'timeout': 10, 'force': True, 'version': '1.1.37', 'schedule': {'4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'action': 'start'}} to its replicas since already forwarded 2017-10-11 17:03:33,849 | INFO | replica.py ( 642) | create3PCBatch | Node2:1 creating batch 3 for ledger 2 with state root None 2017-10-11 17:03:33,850 | DISPLAY | replica.py ( 673) | create3PCBatch | Node2:1 created a PRE-PREPARE with 1 requests for ledger 2 2017-10-11 17:03:33,850 | INFO | replica.py (2064) | send | Node2:1 sending PrePrepare 2017-10-11 17:03:33,850 | TRACE | replica.py (2065) | send | Node2:1 sending PREPREPARE{'instId': 1, 'stateRootHash': None, 'digest': '6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff', 'viewNo': 0, 'discarded': 1, 'ppSeqNo': 3, 'ledgerId': 2, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741413733229)], 'ppTime': 1507741413, 'txnRootHash': None} 2017-10-11 17:03:33,856 | DEBUG | replica.py ( 594) | trackBatches | Node2:1 tracking batch for PREPREPARE{'instId': 1, 'stateRootHash': None, 'digest': '6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff', 'viewNo': 0, 'discarded': 1, 'ppSeqNo': 3, 'ledgerId': 2, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741413733229)], 'ppTime': 1507741413, 'txnRootHash': None} with state root None 2017-10-11 17:03:33,860 | DEBUG | node.py (2593) | send | Node2 sending message PREPREPARE{'instId': 1, 'stateRootHash': None, 'digest': '6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff', 'viewNo': 0, 'discarded': 1, 'ppSeqNo': 3, 'ledgerId': 2, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741413733229)], 'ppTime': 1507741413, 'txnRootHash': None} to all recipients: ['Node1', 'Node4', 'Node3'] 2017-10-11 17:03:33,867 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":1,"viewNo":0,"ppSeqNo":3,"discarded":1,"ledgerId":2,"op":"PREPREPARE","ppTime":1507741413,"digest":"6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741413733229]],"stateRootHash":null,"txnRootHash":null}' to Node1 2017-10-11 17:03:33,868 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":1,"viewNo":0,"ppSeqNo":3,"discarded":1,"ledgerId":2,"op":"PREPREPARE","ppTime":1507741413,"digest":"6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741413733229]],"stateRootHash":null,"txnRootHash":null}' to Node1 2017-10-11 17:03:33,868 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":1,"viewNo":0,"ppSeqNo":3,"discarded":1,"ledgerId":2,"op":"PREPREPARE","ppTime":1507741413,"digest":"6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741413733229]],"stateRootHash":null,"txnRootHash":null}' to Node4 2017-10-11 17:03:33,868 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":1,"viewNo":0,"ppSeqNo":3,"discarded":1,"ledgerId":2,"op":"PREPREPARE","ppTime":1507741413,"digest":"6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741413733229]],"stateRootHash":null,"txnRootHash":null}' to Node4 2017-10-11 17:03:33,869 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":1,"viewNo":0,"ppSeqNo":3,"discarded":1,"ledgerId":2,"op":"PREPREPARE","ppTime":1507741413,"digest":"6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741413733229]],"stateRootHash":null,"txnRootHash":null}' to Node3 2017-10-11 17:03:33,872 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":1,"viewNo":0,"ppSeqNo":3,"discarded":1,"ledgerId":2,"op":"PREPREPARE","ppTime":1507741413,"digest":"6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff","reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741413733229]],"stateRootHash":null,"txnRootHash":null}' to Node3 2017-10-11 17:03:33,874 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:03:33,883 | TRACE | signing.py ( 98) | serializeMsg | serialized msg {'reqId': 1507741413733229, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz', 'operation': {'type': '109', 'name': 'upgrade37', 'action': 'start', 'timeout': 10, 'justification': None, 'force': True, 'version': '1.1.37', 'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'}} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:action:start|force:True|justification:|name:upgrade37|schedule:4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA:2017-10-11T14:30:00.258870+00:00|8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb:2017-10-11T14:20:00.258870+00:00|DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya:2017-10-11T14:25:00.258870+00:00|Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv:2017-10-11T14:15:00.258870+00:00|sha256:ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f|timeout:10|type:109|version:1.1.37|reqId:1507741413733229 2017-10-11 17:03:33,885 | INFO | node.py (2250) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1507741413733229 2017-10-11 17:03:33,885 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node1: PROPAGATE{'request': {'reqId': 1507741413733229, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz', 'operation': {'type': '109', 'name': 'upgrade37', 'action': 'start', 'timeout': 10, 'justification': None, 'force': True, 'version': '1.1.37', 'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'} 2017-10-11 17:03:33,886 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'op': 'PROPAGATE', 'request': {'reqId': 1507741413733229, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz', 'operation': {'type': '109', 'name': 'upgrade37', 'action': 'start', 'timeout': 10, 'justification': None, 'force': True, 'version': '1.1.37', 'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'}, 'Node1') 2017-10-11 17:03:33,887 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'request': {'reqId': 1507741413733229, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz', 'operation': {'type': '109', 'name': 'upgrade37', 'action': 'start', 'timeout': 10, 'justification': None, 'force': True, 'version': '1.1.37', 'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'} 2017-10-11 17:03:33,890 | DEBUG | node.py (1799) | processPropagate | Node Node2 received propagated request: PROPAGATE{'request': {'reqId': 1507741413733229, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz', 'operation': {'type': '109', 'name': 'upgrade37', 'action': 'start', 'timeout': 10, 'justification': None, 'force': True, 'version': '1.1.37', 'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'}}, 'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%'} 2017-10-11 17:03:33,891 | TRACE | signing.py ( 98) | serializeMsg | serialized msg {'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741413733229, 'operation': {'type': '109', 'name': 'upgrade37', 'action': 'start', 'timeout': 10, 'justification': None, 'force': True, 'version': '1.1.37', 'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'}} into identifier:V4SGRU86Z58d6TV7PBUe6f|operation:action:start|force:True|justification:|name:upgrade37|schedule:4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA:2017-10-11T14:30:00.258870+00:00|8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb:2017-10-11T14:20:00.258870+00:00|DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya:2017-10-11T14:25:00.258870+00:00|Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv:2017-10-11T14:15:00.258870+00:00|sha256:ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f|timeout:10|type:109|version:1.1.37|reqId:1507741413733229 2017-10-11 17:03:33,892 | TRACE | propagator.py ( 138) | propagate | Node2 already propagated SafeRequest: {'signature': '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741413733229, 'operation': {'type': '109', 'name': 'upgrade37', 'action': 'start', 'timeout': 10, 'justification': None, 'force': True, 'version': '1.1.37', 'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'}} 2017-10-11 17:03:33,894 | DEBUG | propagator.py ( 241) | tryForwarding | Node2 not forwarding request SafeRequest: {'signature': '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'reqId': 1507741413733229, 'operation': {'type': '109', 'name': 'upgrade37', 'action': 'start', 'timeout': 10, 'justification': None, 'force': True, 'version': '1.1.37', 'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T14:15:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T14:20:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T14:25:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T14:30:00.258870+00:00'}, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'}} to its replicas since already forwarded 2017-10-11 17:03:33,898 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 2 messages through listener 2017-10-11 17:03:33,911 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node1: PREPREPARE{'instId': 0, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'digest': '6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff', 'viewNo': 0, 'discarded': 1, 'ppSeqNo': 3, 'ledgerId': 2, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1507741413733229]], 'ppTime': 1507741413, 'txnRootHash': '4KLT8Jvfc9dEy91Tk8nsiWbFrvmwhBNu6avtVG3jcami'} 2017-10-11 17:03:33,912 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'discarded': 1, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'viewNo': 0, 'digest': '6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff', 'ppSeqNo': 3, 'ledgerId': 2, 'op': 'PREPREPARE', 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1507741413733229]], 'ppTime': 1507741413, 'txnRootHash': '4KLT8Jvfc9dEy91Tk8nsiWbFrvmwhBNu6avtVG3jcami'}, 'Node1') 2017-10-11 17:03:33,912 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox PREPREPARE{'instId': 0, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'digest': '6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff', 'viewNo': 0, 'discarded': 1, 'ppSeqNo': 3, 'ledgerId': 2, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1507741413733229]], 'ppTime': 1507741413, 'txnRootHash': '4KLT8Jvfc9dEy91Tk8nsiWbFrvmwhBNu6avtVG3jcami'} 2017-10-11 17:03:33,912 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node1: PREPARE{'instId': 1, 'stateRootHash': None, 'viewNo': 0, 'ppTime': 1507741413, 'ppSeqNo': 3, 'digest': '6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff', 'txnRootHash': None} 2017-10-11 17:03:33,913 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'stateRootHash': None, 'viewNo': 0, 'digest': '6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff', 'ppSeqNo': 3, 'op': 'PREPARE', 'ppTime': 1507741413, 'txnRootHash': None}, 'Node1') 2017-10-11 17:03:33,913 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'instId': 1, 'stateRootHash': None, 'viewNo': 0, 'ppTime': 1507741413, 'ppSeqNo': 3, 'digest': '6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff', 'txnRootHash': None} 2017-10-11 17:03:33,916 | DEBUG | replica.py ( 790) | processPrePrepare | Node2:0 received PRE-PREPARE(0, 3) from Node1:0 at 2270499.373833943 2017-10-11 17:03:33,917 | DEBUG | replica.py (1002) | validate_pre_prepare | Node2:0 state root before processing PREPREPARE{'instId': 0, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'digest': '6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff', 'viewNo': 0, 'discarded': 1, 'ppSeqNo': 3, 'ledgerId': 2, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741413733229)], 'ppTime': 1507741413, 'txnRootHash': '4KLT8Jvfc9dEy91Tk8nsiWbFrvmwhBNu6avtVG3jcami'} is b'\xbc q\xa4\xde\x84o(W\x02D\x7f%\x89\xdd\x166x\xe0\x97*\x8a\x1b\r(\xb0N\xd5\xc0\x94T\x7f', GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn 2017-10-11 17:03:33,918 | DEBUG | replica.py ( 903) | doPrepare | Node2:0 Sending PREPARE(0, 3) at 2270499.376260179 2017-10-11 17:03:33,919 | INFO | replica.py (2064) | send | Node2:0 sending Prepare 2017-10-11 17:03:33,920 | TRACE | replica.py (2065) | send | Node2:0 sending PREPARE{'instId': 0, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'viewNo': 0, 'ppTime': 1507741413, 'ppSeqNo': 3, 'digest': '6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff', 'txnRootHash': '4KLT8Jvfc9dEy91Tk8nsiWbFrvmwhBNu6avtVG3jcami'} 2017-10-11 17:03:33,921 | DEBUG | replica.py ( 886) | tryCommit | Node2:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 0, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'viewNo': 0, 'ppTime': 1507741413, 'ppSeqNo': 3, 'digest': '6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff', 'txnRootHash': '4KLT8Jvfc9dEy91Tk8nsiWbFrvmwhBNu6avtVG3jcami'} 2017-10-11 17:03:33,921 | DEBUG | replica.py ( 594) | trackBatches | Node2:0 tracking batch for PREPREPARE{'instId': 0, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'digest': '6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff', 'viewNo': 0, 'discarded': 1, 'ppSeqNo': 3, 'ledgerId': 2, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741413733229)], 'ppTime': 1507741413, 'txnRootHash': '4KLT8Jvfc9dEy91Tk8nsiWbFrvmwhBNu6avtVG3jcami'} with state root b'\xbc q\xa4\xde\x84o(W\x02D\x7f%\x89\xdd\x166x\xe0\x97*\x8a\x1b\r(\xb0N\xd5\xc0\x94T\x7f' 2017-10-11 17:03:33,922 | DEBUG | replica.py ( 809) | processPrePrepare | Node2:0 processed incoming PRE-PREPARE(0, 3) 2017-10-11 17:03:33,923 | DEBUG | replica.py ( 834) | processPrepare | Node2:1 received PREPARE(0, 3) from Node1:1 2017-10-11 17:03:33,923 | DEBUG | replica.py ( 886) | tryCommit | Node2:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'instId': 1, 'stateRootHash': None, 'viewNo': 0, 'ppTime': 1507741413, 'ppSeqNo': 3, 'digest': '6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff', 'txnRootHash': None} 2017-10-11 17:03:33,923 | DEBUG | replica.py ( 845) | processPrepare | Node2:1 processed incoming PREPARE (0, 3) 2017-10-11 17:03:33,930 | DEBUG | node.py (2593) | send | Node2 sending message PREPARE{'instId': 0, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'viewNo': 0, 'ppTime': 1507741413, 'ppSeqNo': 3, 'digest': '6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff', 'txnRootHash': '4KLT8Jvfc9dEy91Tk8nsiWbFrvmwhBNu6avtVG3jcami'} to all recipients: ['Node1', 'Node4', 'Node3'] 2017-10-11 17:03:33,933 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":0,"viewNo":0,"ppSeqNo":3,"op":"PREPARE","ppTime":1507741413,"digest":"6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff","stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","txnRootHash":"4KLT8Jvfc9dEy91Tk8nsiWbFrvmwhBNu6avtVG3jcami"}' to Node1 2017-10-11 17:03:33,934 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":0,"viewNo":0,"ppSeqNo":3,"op":"PREPARE","ppTime":1507741413,"digest":"6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff","stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","txnRootHash":"4KLT8Jvfc9dEy91Tk8nsiWbFrvmwhBNu6avtVG3jcami"}' to Node1 2017-10-11 17:03:33,934 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":0,"viewNo":0,"ppSeqNo":3,"op":"PREPARE","ppTime":1507741413,"digest":"6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff","stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","txnRootHash":"4KLT8Jvfc9dEy91Tk8nsiWbFrvmwhBNu6avtVG3jcami"}' to Node4 2017-10-11 17:03:33,934 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":0,"viewNo":0,"ppSeqNo":3,"op":"PREPARE","ppTime":1507741413,"digest":"6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff","stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","txnRootHash":"4KLT8Jvfc9dEy91Tk8nsiWbFrvmwhBNu6avtVG3jcami"}' to Node4 2017-10-11 17:03:33,935 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":0,"viewNo":0,"ppSeqNo":3,"op":"PREPARE","ppTime":1507741413,"digest":"6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff","stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","txnRootHash":"4KLT8Jvfc9dEy91Tk8nsiWbFrvmwhBNu6avtVG3jcami"}' to Node3 2017-10-11 17:03:33,935 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":0,"viewNo":0,"ppSeqNo":3,"op":"PREPARE","ppTime":1507741413,"digest":"6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff","stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","txnRootHash":"4KLT8Jvfc9dEy91Tk8nsiWbFrvmwhBNu6avtVG3jcami"}' to Node3 2017-10-11 17:03:33,939 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.181 seconds 2017-10-11 17:03:33,941 | DEBUG | base_events.py ( 681) | create_connection | connect to ('127.0.0.1', 30003) 2017-10-11 17:03:33,943 | DEBUG | base_events.py (1270) | _run_once | poll 6.096 ms took 0.015 ms: 1 events 2017-10-11 17:03:33,945 | DEBUG | base_events.py ( 719) | create_connection | connected to 127.0.0.1:'30003': (<_SelectorSocketTransport fd=75 read=polling write=>, ) 2017-10-11 17:03:33,946 | TRACE | has_action_queue.py ( 34) | _schedule | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb scheduling action partial(_declareTimeoutExceeded) with id 1 to run in 600 seconds 2017-10-11 17:03:33,951 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:03:33,952 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'instId': 1, 'stateRootHash': None, 'viewNo': 0, 'ppTime': 1507741413, 'ppSeqNo': 3, 'digest': '6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff', 'txnRootHash': None} 2017-10-11 17:03:33,952 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'digest': '6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff', 'viewNo': 0, 'txnRootHash': None, 'ppSeqNo': 3, 'ppTime': 1507741413, 'op': 'PREPARE', 'stateRootHash': None}, 'Node4') 2017-10-11 17:03:33,952 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'instId': 1, 'stateRootHash': None, 'viewNo': 0, 'ppTime': 1507741413, 'ppSeqNo': 3, 'digest': '6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff', 'txnRootHash': None} 2017-10-11 17:03:33,957 | DEBUG | replica.py ( 834) | processPrepare | Node2:1 received PREPARE(0, 3) from Node4:1 2017-10-11 17:03:33,959 | DEBUG | replica.py ( 922) | doCommit | Node2:1 Sending COMMIT(0, 3) at 2270499.416487113 2017-10-11 17:03:33,959 | INFO | replica.py (2064) | send | Node2:1 sending Commit 2017-10-11 17:03:33,959 | TRACE | replica.py (2065) | send | Node2:1 sending COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 3} 2017-10-11 17:03:33,960 | DEBUG | replica.py ( 898) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 3} commits where f is 1 2017-10-11 17:03:33,960 | DEBUG | replica.py ( 845) | processPrepare | Node2:1 processed incoming PREPARE (0, 3) 2017-10-11 17:03:33,963 | DEBUG | node.py (2593) | send | Node2 sending message COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 3} to all recipients: ['Node1', 'Node4', 'Node3'] 2017-10-11 17:03:33,966 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node1 2017-10-11 17:03:33,967 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node1 2017-10-11 17:03:33,967 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node4 2017-10-11 17:03:33,967 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node4 2017-10-11 17:03:33,968 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node3 2017-10-11 17:03:33,968 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":1,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node3 2017-10-11 17:03:33,973 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:03:33,974 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'instId': 1, 'stateRootHash': None, 'viewNo': 0, 'ppTime': 1507741413, 'ppSeqNo': 3, 'digest': '6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff', 'txnRootHash': None} 2017-10-11 17:03:33,974 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'digest': '6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff', 'viewNo': 0, 'ppTime': 1507741413, 'ppSeqNo': 3, 'op': 'PREPARE', 'stateRootHash': None, 'txnRootHash': None}, 'Node3') 2017-10-11 17:03:33,974 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'instId': 1, 'stateRootHash': None, 'viewNo': 0, 'ppTime': 1507741413, 'ppSeqNo': 3, 'digest': '6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff', 'txnRootHash': None} 2017-10-11 17:03:33,978 | DEBUG | replica.py ( 834) | processPrepare | Node2:1 received PREPARE(0, 3) from Node3:1 2017-10-11 17:03:33,979 | DEBUG | replica.py ( 886) | tryCommit | Node2:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'instId': 1, 'stateRootHash': None, 'viewNo': 0, 'ppTime': 1507741413, 'ppSeqNo': 3, 'digest': '6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff', 'txnRootHash': None} 2017-10-11 17:03:33,980 | DEBUG | replica.py ( 845) | processPrepare | Node2:1 processed incoming PREPARE (0, 3) 2017-10-11 17:03:33,984 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:03:33,985 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node4: BATCH{'messages': ['{"viewNo":0,"instId":0,"op":"PREPARE","stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","txnRootHash":"4KLT8Jvfc9dEy91Tk8nsiWbFrvmwhBNu6avtVG3jcami","ppTime":1507741413,"digest":"6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff","ppSeqNo":3}', '{"instId":1,"op":"COMMIT","ppSeqNo":3,"viewNo":0}'], 'signature': None} 2017-10-11 17:03:33,986 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"viewNo":0,"instId":0,"op":"PREPARE","stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","txnRootHash":"4KLT8Jvfc9dEy91Tk8nsiWbFrvmwhBNu6avtVG3jcami","ppTime":1507741413,"digest":"6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff","ppSeqNo":3}', '{"instId":1,"op":"COMMIT","ppSeqNo":3,"viewNo":0}'], 'op': 'BATCH', 'signature': None}, 'Node4') 2017-10-11 17:03:33,986 | DEBUG | node.py (1328) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"viewNo":0,"instId":0,"op":"PREPARE","stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","txnRootHash":"4KLT8Jvfc9dEy91Tk8nsiWbFrvmwhBNu6avtVG3jcami","ppTime":1507741413,"digest":"6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff","ppSeqNo":3}', '{"instId":1,"op":"COMMIT","ppSeqNo":3,"viewNo":0}'], 'signature': None} 2017-10-11 17:03:33,987 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'instId': 0, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'viewNo': 0, 'ppTime': 1507741413, 'ppSeqNo': 3, 'digest': '6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff', 'txnRootHash': '4KLT8Jvfc9dEy91Tk8nsiWbFrvmwhBNu6avtVG3jcami'} 2017-10-11 17:03:33,987 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'digest': '6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff', 'viewNo': 0, 'txnRootHash': '4KLT8Jvfc9dEy91Tk8nsiWbFrvmwhBNu6avtVG3jcami', 'ppSeqNo': 3, 'ppTime': 1507741413, 'op': 'PREPARE', 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA'}, 'Node4') 2017-10-11 17:03:33,988 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'instId': 0, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'viewNo': 0, 'ppTime': 1507741413, 'ppSeqNo': 3, 'digest': '6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff', 'txnRootHash': '4KLT8Jvfc9dEy91Tk8nsiWbFrvmwhBNu6avtVG3jcami'} 2017-10-11 17:03:33,988 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 3} 2017-10-11 17:03:33,988 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 3}, 'Node4') 2017-10-11 17:03:33,989 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 3} 2017-10-11 17:03:33,992 | DEBUG | replica.py ( 834) | processPrepare | Node2:0 received PREPARE(0, 3) from Node4:0 2017-10-11 17:03:33,992 | DEBUG | replica.py ( 922) | doCommit | Node2:0 Sending COMMIT(0, 3) at 2270499.450008981 2017-10-11 17:03:33,992 | INFO | replica.py (2064) | send | Node2:0 sending Commit 2017-10-11 17:03:33,993 | TRACE | replica.py (2065) | send | Node2:0 sending COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 3} 2017-10-11 17:03:33,993 | DEBUG | replica.py ( 898) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 3} commits where f is 1 2017-10-11 17:03:33,993 | DEBUG | replica.py ( 845) | processPrepare | Node2:0 processed incoming PREPARE (0, 3) 2017-10-11 17:03:33,994 | DEBUG | replica.py ( 863) | processCommit | Node2:1 received COMMIT(0, 3) from Node4:1 2017-10-11 17:03:33,994 | DEBUG | replica.py ( 898) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 3} commits where f is 1 2017-10-11 17:03:33,994 | DEBUG | replica.py ( 874) | processCommit | Node2:1 processed incoming COMMIT(0, 3) 2017-10-11 17:03:33,995 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:03:33,996 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 3} 2017-10-11 17:03:33,996 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 3}, 'Node1') 2017-10-11 17:03:33,997 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 3} 2017-10-11 17:03:33,999 | DEBUG | node.py (2593) | send | Node2 sending message COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 3} to all recipients: ['Node1', 'Node4', 'Node3'] 2017-10-11 17:03:34,000 | DEBUG | replica.py ( 863) | processCommit | Node2:1 received COMMIT(0, 3) from Node1:1 2017-10-11 17:03:34,000 | TRACE | replica.py ( 894) | tryOrder | Node2:1 returning request to node 2017-10-11 17:03:34,000 | INFO | replica.py (1395) | doOrder | Node2:1 ordering COMMIT(0, 3) 2017-10-11 17:03:34,000 | DEBUG | replica.py ( 310) | last_ordered_3pc | Node2:1 set last ordered as (0, 3) 2017-10-11 17:03:34,001 | INFO | replica.py (2064) | send | Node2:1 sending Ordered 2017-10-11 17:03:34,001 | TRACE | replica.py (2065) | send | Node2:1 sending ORDERED{'instId': 1, 'viewNo': 0, 'stateRootHash': None, 'ppSeqNo': 3, 'ledgerId': 2, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741413733229)], 'ppTime': 1507741413, 'txnRootHash': None} 2017-10-11 17:03:34,001 | DEBUG | replica.py (1426) | order_3pc_key | Node2:1 ordered request (0, 3) 2017-10-11 17:03:34,005 | DEBUG | replica.py ( 874) | processCommit | Node2:1 processed incoming COMMIT(0, 3) 2017-10-11 17:03:34,006 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:03:34,007 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: BATCH{'messages': ['{"ppSeqNo":3,"op":"PREPARE","ppTime":1507741413,"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","viewNo":0,"digest":"6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff","txnRootHash":"4KLT8Jvfc9dEy91Tk8nsiWbFrvmwhBNu6avtVG3jcami","instId":0}', '{"op":"COMMIT","ppSeqNo":3,"viewNo":0,"instId":1}'], 'signature': None} 2017-10-11 17:03:34,007 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"ppSeqNo":3,"op":"PREPARE","ppTime":1507741413,"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","viewNo":0,"digest":"6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff","txnRootHash":"4KLT8Jvfc9dEy91Tk8nsiWbFrvmwhBNu6avtVG3jcami","instId":0}', '{"op":"COMMIT","ppSeqNo":3,"viewNo":0,"instId":1}'], 'op': 'BATCH', 'signature': None}, 'Node3') 2017-10-11 17:03:34,007 | DEBUG | node.py (1328) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"ppSeqNo":3,"op":"PREPARE","ppTime":1507741413,"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","viewNo":0,"digest":"6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff","txnRootHash":"4KLT8Jvfc9dEy91Tk8nsiWbFrvmwhBNu6avtVG3jcami","instId":0}', '{"op":"COMMIT","ppSeqNo":3,"viewNo":0,"instId":1}'], 'signature': None} 2017-10-11 17:03:34,008 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'instId': 0, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'viewNo': 0, 'ppTime': 1507741413, 'ppSeqNo': 3, 'digest': '6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff', 'txnRootHash': '4KLT8Jvfc9dEy91Tk8nsiWbFrvmwhBNu6avtVG3jcami'} 2017-10-11 17:03:34,012 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'digest': '6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff', 'viewNo': 0, 'ppTime': 1507741413, 'ppSeqNo': 3, 'op': 'PREPARE', 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': '4KLT8Jvfc9dEy91Tk8nsiWbFrvmwhBNu6avtVG3jcami'}, 'Node3') 2017-10-11 17:03:34,012 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'instId': 0, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'viewNo': 0, 'ppTime': 1507741413, 'ppSeqNo': 3, 'digest': '6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff', 'txnRootHash': '4KLT8Jvfc9dEy91Tk8nsiWbFrvmwhBNu6avtVG3jcami'} 2017-10-11 17:03:34,013 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 3} 2017-10-11 17:03:34,013 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 3}, 'Node3') 2017-10-11 17:03:34,013 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 3} 2017-10-11 17:03:34,015 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node1 2017-10-11 17:03:34,018 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node1 2017-10-11 17:03:34,019 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node4 2017-10-11 17:03:34,019 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node4 2017-10-11 17:03:34,020 | DEBUG | zstack.py ( 699) | transmit | Node2 transmitting message b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node3 2017-10-11 17:03:34,022 | TRACE | batched.py ( 85) | flushOutBoxes | Node2 sending msg b'{"instId":0,"op":"COMMIT","viewNo":0,"ppSeqNo":3}' to Node3 2017-10-11 17:03:34,023 | TRACE | node.py (1892) | processOrdered | Node2 got ordered requests from backup replica 1 2017-10-11 17:03:34,024 | DEBUG | replica.py ( 834) | processPrepare | Node2:0 received PREPARE(0, 3) from Node3:0 2017-10-11 17:03:34,027 | DEBUG | replica.py ( 886) | tryCommit | Node2:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'instId': 0, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'viewNo': 0, 'ppTime': 1507741413, 'ppSeqNo': 3, 'digest': '6d95b08f4d95a879e27b0cf519926b989fad4dbf0b42a44d2a496925fa45faff', 'txnRootHash': '4KLT8Jvfc9dEy91Tk8nsiWbFrvmwhBNu6avtVG3jcami'} 2017-10-11 17:03:34,027 | DEBUG | replica.py ( 845) | processPrepare | Node2:0 processed incoming PREPARE (0, 3) 2017-10-11 17:03:34,027 | TRACE | message_processor.py ( 28) | discard | Node2:1 discarding message COMMIT{'instId': 1, 'viewNo': 0, 'ppSeqNo': 3} because already ordered 3 phase message 2017-10-11 17:03:34,028 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 2 messages through listener 2017-10-11 17:03:34,029 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 3} 2017-10-11 17:03:34,029 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 3}, 'Node3') 2017-10-11 17:03:34,029 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 3} 2017-10-11 17:03:34,030 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 3} 2017-10-11 17:03:34,030 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 3}, 'Node4') 2017-10-11 17:03:34,031 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 3} 2017-10-11 17:03:34,044 | DEBUG | replica.py ( 863) | processCommit | Node2:0 received COMMIT(0, 3) from Node3:0 2017-10-11 17:03:34,045 | DEBUG | replica.py ( 898) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 3} commits where f is 1 2017-10-11 17:03:34,048 | DEBUG | replica.py ( 874) | processCommit | Node2:0 processed incoming COMMIT(0, 3) 2017-10-11 17:03:34,057 | DEBUG | replica.py ( 863) | processCommit | Node2:0 received COMMIT(0, 3) from Node4:0 2017-10-11 17:03:34,057 | TRACE | replica.py ( 894) | tryOrder | Node2:0 returning request to node 2017-10-11 17:03:34,057 | INFO | replica.py (1395) | doOrder | Node2:0 ordering COMMIT(0, 3) 2017-10-11 17:03:34,057 | DEBUG | replica.py ( 310) | last_ordered_3pc | Node2:0 set last ordered as (0, 3) 2017-10-11 17:03:34,058 | INFO | replica.py (2064) | send | Node2:0 sending Ordered 2017-10-11 17:03:34,059 | TRACE | replica.py (2065) | send | Node2:0 sending ORDERED{'instId': 0, 'viewNo': 0, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'ppSeqNo': 3, 'ledgerId': 2, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741413733229)], 'ppTime': 1507741413, 'txnRootHash': '4KLT8Jvfc9dEy91Tk8nsiWbFrvmwhBNu6avtVG3jcami'} 2017-10-11 17:03:34,059 | DEBUG | replica.py (1426) | order_3pc_key | Node2:0 ordered request (0, 3) 2017-10-11 17:03:34,060 | DEBUG | replica.py ( 874) | processCommit | Node2:0 processed incoming COMMIT(0, 3) 2017-10-11 17:03:34,063 | DEBUG | node.py (1881) | processOrdered | Node2 executing Ordered batch 0 3 of 1 requests 2017-10-11 17:03:34,068 | DEBUG | ledger.py ( 68) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-10-11 17:03:34,068 | INFO | upgrader.py ( 308) | _scheduleUpgrade | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb's upgrader processing upgrade for version 1.1.37 2017-10-11 17:03:34,069 | INFO | upgrader.py ( 369) | _callUpgradeAgent | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb's upgrader calling agent for upgrade 2017-10-11 17:03:34,070 | DEBUG | node.py (2374) | sendReplyToClient | Node2 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1507741413733229) to client 2017-10-11 17:03:34,071 | TRACE | zstack.py ( 726) | transmitThroughListener | Node2C transmitting b'{"op":"REPLY","result":{"name":"upgrade37","justification":null,"signature":"4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz","force":true,"version":"1.1.37","rootHash":"4KLT8Jvfc9dEy91Tk8nsiWbFrvmwhBNu6avtVG3jcami","type":"109","action":"start","seqNo":1,"txnTime":1507741413,"auditPath":[],"timeout":10,"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1507741413733229,"schedule":{"4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA":"2017-10-11T14:30:00.258870+00:00","8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb":"2017-10-11T14:20:00.258870+00:00","Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv":"2017-10-11T14:15:00.258870+00:00","DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya":"2017-10-11T14:25:00.258870+00:00"},"sha256":"ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f"}}' to b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' through listener socket 2017-10-11 17:03:34,071 | DEBUG | node.py (2295) | executeBatch | Node2 storing 3PC key (0, 3) for ledger 2 range (1, 1) 2017-10-11 17:03:34,072 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:03:34,078 | WARNING | base_events.py (1308) | _run_once | Executing wait_for= cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164]> took 0.128 seconds 2017-10-11 17:03:34,079 | DEBUG | base_events.py ( 681) | create_connection | connect to ('127.0.0.1', 30003) 2017-10-11 17:03:34,080 | DEBUG | base_events.py (1270) | _run_once | poll 7.732 ms took 0.011 ms: 1 events 2017-10-11 17:03:34,082 | DEBUG | base_events.py ( 719) | create_connection | connected to 127.0.0.1:'30003': (<_SelectorSocketTransport fd=75 read=polling write=>, ) 2017-10-11 17:03:34,082 | TRACE | has_action_queue.py ( 34) | _schedule | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb scheduling action partial(_declareTimeoutExceeded) with id 2 to run in 600 seconds 2017-10-11 17:03:34,104 | TRACE | zstack.py ( 472) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:03:34,105 | DEBUG | node.py (1313) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 3} 2017-10-11 17:03:34,106 | INFO | node.py (1275) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'op': 'COMMIT', 'viewNo': 0, 'ppSeqNo': 3}, 'Node1') 2017-10-11 17:03:34,106 | DEBUG | node.py (1342) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 3} 2017-10-11 17:03:34,108 | TRACE | message_processor.py ( 28) | discard | Node2:0 discarding message COMMIT{'instId': 0, 'viewNo': 0, 'ppSeqNo': 3} because already ordered 3 phase message 2017-10-11 17:03:37,022 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:03:37,023 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:03:37,023 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:03:37,023 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:03:42,125 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkPerformance with id 31 2017-10-11 17:03:42,125 | TRACE | node.py (2005) | checkPerformance | Node2 checking its performance 2017-10-11 17:03:42,141 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:03:42,141 | TRACE | monitor.py ( 299) | isMasterThroughputTooLow | Node2 master throughput ratio 0.9804845167957082 is acceptable. 2017-10-11 17:03:42,141 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:03:42,141 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:03:42,141 | DEBUG | node.py (2022) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:03:42,141 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 32 to run in 10 seconds 2017-10-11 17:03:52,038 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:03:52,039 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:03:52,039 | DEBUG | kit_zstack.py ( 65) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:03:52,039 | DEBUG | kit_zstack.py ( 47) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:03:52,144 | TRACE | has_action_queue.py ( 64) | _serviceActions | Node2 running action checkPerformance with id 32 2017-10-11 17:03:52,144 | TRACE | node.py (2005) | checkPerformance | Node2 checking its performance 2017-10-11 17:03:52,144 | DEBUG | notifier_plugin_manager.py ( 65) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:03:52,144 | TRACE | monitor.py ( 299) | isMasterThroughputTooLow | Node2 master throughput ratio 0.9804845167957082 is acceptable. 2017-10-11 17:03:52,145 | TRACE | monitor.py ( 315) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:03:52,145 | TRACE | monitor.py ( 345) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:03:52,145 | DEBUG | node.py (2022) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:03:52,145 | TRACE | has_action_queue.py ( 34) | _schedule | Node2 scheduling action checkPerformance with id 33 to run in 10 seconds 2017-10-11 17:04:03,077 | DEBUG | node_runner.py ( 18) | run_node | You can find logs in /home/sovrin/.sovrin/Node2.log 2017-10-11 17:04:03,077 | DEBUG | node_runner.py ( 21) | run_node | Sovrin related env vars: [] 2017-10-11 17:04:05,419 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: git 2017-10-11 17:04:05,434 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: hg 2017-10-11 17:04:05,476 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: svn 2017-10-11 17:04:05,477 | DEBUG | __init__.py ( 60) | register | Registered VCS backend: bzr 2017-10-11 17:04:06,362 | DEBUG | selector_events.py ( 53) | __init__ | Using selector: EpollSelector 2017-10-11 17:04:06,362 | DEBUG | looper.py ( 125) | __init__ | Setting handler for SIGINT 2017-10-11 17:04:06,403 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-10-11 17:04:06,413 | DEBUG | ledger.py ( 72) | recoverTree | Recovering tree from hash store of size 11 2017-10-11 17:04:06,414 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.0010682488791644573 seconds 2017-10-11 17:04:06,446 | DEBUG | idr_cache.py ( 25) | __init__ | Initializing identity cache Node2 2017-10-11 17:04:06,459 | INFO | node.py (2420) | initStateFromLedger | Node2 found state to be empty, recreating from ledger 2017-10-11 17:04:06,724 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-10-11 17:04:06,736 | DEBUG | ledger.py ( 72) | recoverTree | Recovering tree from hash store of size 4 2017-10-11 17:04:06,737 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.00043746083974838257 seconds 2017-10-11 17:04:06,737 | INFO | node.py (2420) | initStateFromLedger | Node2 found state to be empty, recreating from ledger 2017-10-11 17:04:06,794 | INFO | plugin_loader.py ( 117) | _load | plugin FirebaseStatsConsumer successfully loaded from module plugin_firebase_stats_consumer 2017-10-11 17:04:06,794 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-10-11 17:04:06,794 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-10-11 17:04:06,794 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-10-11 17:04:06,795 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: ] because it does not have a 'pluginType' attribute 2017-10-11 17:04:06,795 | DEBUG | plugin_loader.py ( 96) | _load | skipping plugin plugin_firebase_stats_consumer[class: typing.Dict<~KT, ~VT>] because it does not have a 'pluginType' attribute 2017-10-11 17:04:06,795 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkPerformance will be repeating every 60 seconds 2017-10-11 17:04:06,795 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 1 to run in 60 seconds 2017-10-11 17:04:06,795 | DEBUG | replica.py ( 313) | h | Node2:0 set watermarks as 0 300 2017-10-11 17:04:06,796 | DISPLAY | replicas.py ( 36) | grow | Node2 added replica Node2:0 to instance 0 (master) 2017-10-11 17:04:06,796 | DEBUG | replica.py ( 313) | h | Node2:1 set watermarks as 0 300 2017-10-11 17:04:06,796 | DISPLAY | replicas.py ( 36) | grow | Node2 added replica Node2:1 to instance 1 (backup) 2017-10-11 17:04:06,796 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkPerformance will be repeating every 10 seconds 2017-10-11 17:04:06,796 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 1 to run in 10 seconds 2017-10-11 17:04:06,796 | DEBUG | has_action_queue.py ( 79) | startRepeating | checkNodeRequestSpike will be repeating every 60 seconds 2017-10-11 17:04:06,796 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 2 to run in 60 seconds 2017-10-11 17:04:06,797 | DEBUG | plugin_helper.py ( 24) | loadPlugins | Plugin loading started to load plugins from basedir: /home/sovrin/.sovrin 2017-10-11 17:04:06,797 | DEBUG | plugin_helper.py ( 68) | loadPlugins | Total plugins loaded from basedir /home/sovrin/.sovrin are : 0 2017-10-11 17:04:06,797 | DEBUG | node.py ( 340) | __init__ | total plugins loaded in node: 0 2017-10-11 17:04:06,848 | DEBUG | ledger.py ( 200) | start | Starting ledger... 2017-10-11 17:04:06,859 | DEBUG | ledger.py ( 72) | recoverTree | Recovering tree from hash store of size 1 2017-10-11 17:04:06,859 | DEBUG | ledger.py ( 82) | recoverTree | Recovered tree in 0.00036070169880986214 seconds 2017-10-11 17:04:06,881 | WARNING | upgrader.py ( 126) | check_upgrade_succeeded | Upgrade for node Node2 was not scheduled. Last event is scheduled:2017-10-11 14:20:00.258870+00:00:1.1.37:None 2017-10-11 17:04:06,882 | INFO | node.py (2420) | initStateFromLedger | Node2 found state to be empty, recreating from ledger 2017-10-11 17:04:06,882 | DEBUG | motor.py ( 34) | set_status | Node2 changing status from stopped to starting 2017-10-11 17:04:06,882 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-10-11 17:04:06,882 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-10-11 17:04:06,882 | DEBUG | ledger.py ( 198) | start | Ledger already started. 2017-10-11 17:04:06,883 | DEBUG | zstack.py ( 319) | start | Node2 starting with restricted as True and reSetupAuth as True 2017-10-11 17:04:06,883 | DEBUG | authenticator.py ( 31) | start | Starting ZAP at inproc://zeromq.zap.1 2017-10-11 17:04:06,883 | DEBUG | base.py ( 72) | allow | Allowing 0.0.0.0 2017-10-11 17:04:06,883 | DEBUG | base.py ( 112) | configure_curve | Configure curve: *[/home/sovrin/.sovrin/Node2/public_keys] 2017-10-11 17:04:06,884 | DEBUG | zstack.py ( 347) | open | Node2 will bind its listener at 9703 2017-10-11 17:04:06,884 | INFO | stacks.py ( 84) | start | CONNECTION: Node2 listening for other nodes at 0.0.0.0:9703 2017-10-11 17:04:06,885 | DEBUG | zstack.py ( 319) | start | Node2C starting with restricted as False and reSetupAuth as True 2017-10-11 17:04:06,885 | DEBUG | authenticator.py ( 31) | start | Starting ZAP at inproc://zeromq.zap.2 2017-10-11 17:04:06,885 | DEBUG | base.py ( 72) | allow | Allowing 0.0.0.0 2017-10-11 17:04:06,885 | DEBUG | base.py ( 112) | configure_curve | Configure curve: *[*] 2017-10-11 17:04:06,885 | DEBUG | zstack.py ( 347) | open | Node2C will bind its listener at 9704 2017-10-11 17:04:06,886 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action propose_view_change with id 3 to run in 60 seconds 2017-10-11 17:04:06,886 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action dump_json_file with id 4 to run in 3 seconds 2017-10-11 17:04:06,886 | DEBUG | has_action_queue.py ( 79) | startRepeating | dump_json_file will be repeating every 60 seconds 2017-10-11 17:04:06,886 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action dump_json_file with id 5 to run in 60 seconds 2017-10-11 17:04:06,886 | INFO | node.py ( 621) | start | Node2 first time running... 2017-10-11 17:04:06,887 | DEBUG | kit_zstack.py ( 97) | connectToMissing | CONNECTION: Node2 found the following missing connections: Node1, Node3, Node4 2017-10-11 17:04:06,887 | TRACE | remote.py ( 86) | connect | connecting socket 78 49279248 to remote Node1:HA(host='10.0.0.2', port=9701) 2017-10-11 17:04:06,888 | INFO | zstack.py ( 590) | connect | CONNECTION: Node2 looking for Node1 at 10.0.0.2:9701 2017-10-11 17:04:06,889 | DEBUG | zstack.py ( 645) | sendPingPong | Node2 pinged Node1 2017-10-11 17:04:06,889 | TRACE | remote.py ( 86) | connect | connecting socket 79 49383280 to remote Node3:HA(host='10.0.0.4', port=9705) 2017-10-11 17:04:06,889 | INFO | zstack.py ( 590) | connect | CONNECTION: Node2 looking for Node3 at 10.0.0.4:9705 2017-10-11 17:04:06,889 | DEBUG | zstack.py ( 645) | sendPingPong | Node2 pinged Node3 2017-10-11 17:04:06,890 | TRACE | remote.py ( 86) | connect | connecting socket 81 49420800 to remote Node4:HA(host='10.0.0.5', port=9707) 2017-10-11 17:04:06,890 | INFO | zstack.py ( 590) | connect | CONNECTION: Node2 looking for Node4 at 10.0.0.5:9707 2017-10-11 17:04:06,890 | DEBUG | zstack.py ( 645) | sendPingPong | Node2 pinged Node4 2017-10-11 17:04:06,890 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:04:06,901 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'pi' to Node1 2017-10-11 17:04:06,902 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'pi' to Node1 2017-10-11 17:04:06,902 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'pi' to Node3 2017-10-11 17:04:06,902 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'pi' to Node3 2017-10-11 17:04:06,902 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'pi' to Node4 2017-10-11 17:04:06,902 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'pi' to Node4 2017-10-11 17:04:07,002 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-10-11 17:04:07,002 | DEBUG | zstack.py ( 663) | handlePingPong | Node2 got ping from Node3 2017-10-11 17:04:07,002 | DEBUG | zstack.py ( 645) | sendPingPong | Node2 ponged Node3 2017-10-11 17:04:07,003 | DEBUG | zstack.py ( 669) | handlePingPong | Node2 got pong from Node3 2017-10-11 17:04:07,003 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'po' to Node3 2017-10-11 17:04:07,003 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'po' to Node3 2017-10-11 17:04:07,003 | DEBUG | keep_in_touch.py ( 68) | conns | Node2's connections changed from set() to {'Node3'} 2017-10-11 17:04:07,003 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node2 now connected to Node3 2017-10-11 17:04:07,004 | DEBUG | node.py (2608) | send | Node2 sending message LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4} to 1 recipients: ['Node3'] 2017-10-11 17:04:07,005 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ledgerId":0,"ppSeqNo":null,"op":"LEDGER_STATUS","txnSeqNo":4,"viewNo":null}' to Node3 2017-10-11 17:04:07,005 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ledgerId":0,"ppSeqNo":null,"op":"LEDGER_STATUS","txnSeqNo":4,"viewNo":null}' to Node3 2017-10-11 17:04:07,030 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:04:07,032 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4} 2017-10-11 17:04:07,033 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'ppSeqNo': None, 'op': 'LEDGER_STATUS', 'ledgerId': 0, 'viewNo': None}, 'Node3') 2017-10-11 17:04:07,033 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4} 2017-10-11 17:04:07,034 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4} from Node3 2017-10-11 17:04:07,035 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 17:04:07,035 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 17:04:07,040 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2C got 1 messages through listener 2017-10-11 17:04:07,040 | DEBUG | zstack.py ( 663) | handlePingPong | Node2C got ping from b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' 2017-10-11 17:04:07,040 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'po' to b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' through listener socket 2017-10-11 17:04:07,040 | DEBUG | zstack.py ( 645) | sendPingPong | Node2C ponged b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' 2017-10-11 17:04:07,064 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2C got 1 messages through listener 2017-10-11 17:04:07,064 | TRACE | node.py (1388) | validateClientMsg | Node2C received CLIENT message: LEDGER_STATUS{'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4} 2017-10-11 17:04:07,065 | DEBUG | node.py (1434) | processClientInBox | Node2C processing b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' request LEDGER_STATUS{'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4} 2017-10-11 17:04:07,065 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '6mQmSGzvyAeSpp5E7rBcYyAwgim9pTdggDXwL4quw8HA', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4} from b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' 2017-10-11 17:04:07,065 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 17:04:07,065 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ledgerId":0,"ppSeqNo":null,"op":"LEDGER_STATUS","txnSeqNo":4,"viewNo":null}' to b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' through listener socket 2017-10-11 17:04:07,066 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 17:04:07,066 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 17:04:07,792 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:04:07,792 | DEBUG | zstack.py ( 663) | handlePingPong | Node2 got ping from Node4 2017-10-11 17:04:07,792 | DEBUG | zstack.py ( 645) | sendPingPong | Node2 ponged Node4 2017-10-11 17:04:07,793 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'po' to Node4 2017-10-11 17:04:07,793 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'po' to Node4 2017-10-11 17:04:07,815 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-10-11 17:04:07,816 | DEBUG | zstack.py ( 669) | handlePingPong | Node2 got pong from Node4 2017-10-11 17:04:07,816 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4} 2017-10-11 17:04:07,816 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'ppSeqNo': None, 'op': 'LEDGER_STATUS', 'txnSeqNo': 4, 'viewNo': None}, 'Node4') 2017-10-11 17:04:07,816 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4} 2017-10-11 17:04:07,816 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4} from Node4 2017-10-11 17:04:07,817 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 17:04:07,817 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 17:04:07,817 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node3', 'Node4'} that its ledger of type 0 is latest 2017-10-11 17:04:07,817 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node2 found from ledger status LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4} that it does not need catchup 2017-10-11 17:04:07,817 | DEBUG | node.py (1479) | preLedgerCatchUp | Node2 going to process any ordered requests before starting catchup. 2017-10-11 17:04:07,817 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node2:0 going to remove 0 Ordered messages from outbox 2017-10-11 17:04:07,817 | DEBUG | node.py (1888) | force_process_ordered | Node2 processed 0 Ordered batches for instance 0 before starting catch up 2017-10-11 17:04:07,817 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node2:1 going to remove 0 Ordered messages from outbox 2017-10-11 17:04:07,817 | DEBUG | node.py (1888) | force_process_ordered | Node2 processed 0 Ordered batches for instance 1 before starting catch up 2017-10-11 17:04:07,817 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-10-11 17:04:07,817 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-10-11 17:04:07,818 | INFO | node.py (1489) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 0 2017-10-11 17:04:07,818 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 17:04:07,818 | DEBUG | node.py ( 941) | checkInstances | Node2 choosing to start election on the basis of count 2 and nodes {'Node3'} 2017-10-11 17:04:07,818 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node1'] 2017-10-11 17:04:07,818 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node1 for ledger status of ledger 2 2017-10-11 17:04:07,818 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:04:07,818 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node2 for ledger status of ledger 2 2017-10-11 17:04:07,818 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-10-11 17:04:07,819 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node3 for ledger status of ledger 2 2017-10-11 17:04:07,819 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-10-11 17:04:07,819 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node4 for ledger status of ledger 2 2017-10-11 17:04:07,819 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node2 going to process 0 stashed ledger statuses for ledger 2 2017-10-11 17:04:07,819 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 0, caught up 0 in total 2017-10-11 17:04:07,826 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-10-11 17:04:07,826 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}']) 2017-10-11 17:04:07,826 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node1: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:04:07,826 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null,"op":"BATCH"}' to Node1 2017-10-11 17:04:07,826 | DEBUG | zstack.py ( 728) | transmit | Remote Node1 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:04:07,826 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-10-11 17:04:07,827 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}']) 2017-10-11 17:04:07,827 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node3: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:04:07,827 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null,"op":"BATCH"}' to Node3 2017-10-11 17:04:07,827 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node4 into one transmission 2017-10-11 17:04:07,827 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}']) 2017-10-11 17:04:07,827 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node4: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:04:07,827 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null,"op":"BATCH"}' to Node4 2017-10-11 17:04:07,828 | DEBUG | keep_in_touch.py ( 68) | conns | Node2's connections changed from {'Node3'} to {'Node3', 'Node4'} 2017-10-11 17:04:07,828 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node2 now connected to Node4 2017-10-11 17:04:07,828 | DEBUG | motor.py ( 34) | set_status | Node2 changing status from starting to started_hungry 2017-10-11 17:04:07,828 | DEBUG | node.py ( 941) | checkInstances | Node2 choosing to start election on the basis of count 3 and nodes {'Node3', 'Node4'} 2017-10-11 17:04:07,828 | DEBUG | primary_selector.py ( 77) | get_msgs_for_lagged_nodes | Node2 has no ViewChangeDone message to send for view 0 2017-10-11 17:04:07,828 | DEBUG | node.py ( 902) | send_current_state_to_lagging_node | Node2 sending current state CURRENT_STATE{'primary': [], 'viewNo': 0} to lagged node Node4 2017-10-11 17:04:07,828 | DEBUG | node.py (2608) | send | Node2 sending message CURRENT_STATE{'primary': [], 'viewNo': 0} to 1 recipients: ['Node4'] 2017-10-11 17:04:07,829 | DEBUG | node.py (2608) | send | Node2 sending message LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4} to 1 recipients: ['Node4'] 2017-10-11 17:04:07,829 | DEBUG | node.py (2608) | send | Node2 sending message LEDGER_STATUS{'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11} to 1 recipients: ['Node4'] 2017-10-11 17:04:07,830 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 3 msgs to Node4 into one transmission 2017-10-11 17:04:07,835 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"primary":[],"op":"CURRENT_STATE","viewNo":0}', b'{"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ledgerId":0,"ppSeqNo":null,"op":"LEDGER_STATUS","txnSeqNo":4,"viewNo":null}', b'{"merkleRoot":"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq","ledgerId":1,"ppSeqNo":null,"op":"LEDGER_STATUS","txnSeqNo":11,"viewNo":null}']) 2017-10-11 17:04:07,835 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node4: b'{"messages":["{\\"primary\\":[],\\"op\\":\\"CURRENT_STATE\\",\\"viewNo\\":0}","{\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ledgerId\\":0,\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"txnSeqNo\\":4,\\"viewNo\\":null}","{\\"merkleRoot\\":\\"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq\\",\\"ledgerId\\":1,\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"txnSeqNo\\":11,\\"viewNo\\":null}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:04:07,835 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"primary\\":[],\\"op\\":\\"CURRENT_STATE\\",\\"viewNo\\":0}","{\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ledgerId\\":0,\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"txnSeqNo\\":4,\\"viewNo\\":null}","{\\"merkleRoot\\":\\"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq\\",\\"ledgerId\\":1,\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"txnSeqNo\\":11,\\"viewNo\\":null}"],"signature":null,"op":"BATCH"}' to Node4 2017-10-11 17:04:07,846 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:04:07,847 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}'], 'signature': None} 2017-10-11 17:04:07,847 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}'], 'signature': None, 'op': 'BATCH'}, 'Node3') 2017-10-11 17:04:07,847 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}'], 'signature': None} 2017-10-11 17:04:07,848 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:04:07,848 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}}, 'Node3') 2017-10-11 17:04:07,848 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:04:07,848 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:04:07,848 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}}, 'Node3') 2017-10-11 17:04:07,848 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:04:07,849 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-10-11 17:04:07,849 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-10-11 17:04:07,849 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-10-11 17:04:07,849 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":2,"merkleRoot":"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ","ppSeqNo":null,"txnSeqNo":1,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":2,"merkleRoot":"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ","ppSeqNo":null,"txnSeqNo":1,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}']) 2017-10-11 17:04:07,850 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node3: b'{"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":1,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":1,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:04:07,850 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":1,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":1,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null,"op":"BATCH"}' to Node3 2017-10-11 17:04:07,861 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:04:07,862 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ","ppSeqNo":null,"txnSeqNo":1,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ","ppSeqNo":null,"txnSeqNo":1,"viewNo":null}}'], 'signature': None} 2017-10-11 17:04:07,862 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ","ppSeqNo":null,"txnSeqNo":1,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ","ppSeqNo":null,"txnSeqNo":1,"viewNo":null}}'], 'signature': None, 'op': 'BATCH'}, 'Node3') 2017-10-11 17:04:07,862 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ","ppSeqNo":null,"txnSeqNo":1,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ","ppSeqNo":null,"txnSeqNo":1,"viewNo":null}}'], 'signature': None} 2017-10-11 17:04:07,863 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:04:07,863 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}}, 'Node3') 2017-10-11 17:04:07,863 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:04:07,863 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:04:07,863 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}}, 'Node3') 2017-10-11 17:04:07,864 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:04:07,864 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1} from Node3 2017-10-11 17:04:07,864 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 1 with 1 2017-10-11 17:04:07,864 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 1 with 1 2017-10-11 17:04:07,864 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1} from Node3 2017-10-11 17:04:07,865 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 1 with 1 2017-10-11 17:04:07,865 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 1 with 1 2017-10-11 17:04:07,877 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:04:07,878 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ","ppSeqNo":null,"txnSeqNo":1,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ","ppSeqNo":null,"txnSeqNo":1,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}'], 'signature': None} 2017-10-11 17:04:07,878 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ","ppSeqNo":null,"txnSeqNo":1,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ","ppSeqNo":null,"txnSeqNo":1,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}'], 'signature': None, 'op': 'BATCH'}, 'Node4') 2017-10-11 17:04:07,878 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ","ppSeqNo":null,"txnSeqNo":1,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ","ppSeqNo":null,"txnSeqNo":1,"viewNo":null},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}'], 'signature': None} 2017-10-11 17:04:07,878 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:04:07,878 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}}, 'Node4') 2017-10-11 17:04:07,878 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:04:07,879 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:04:07,879 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}}, 'Node4') 2017-10-11 17:04:07,879 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:04:07,879 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:04:07,879 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}}, 'Node4') 2017-10-11 17:04:07,879 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:04:07,879 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:04:07,880 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}}, 'Node4') 2017-10-11 17:04:07,880 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:04:07,880 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1} from Node4 2017-10-11 17:04:07,880 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 1 with 1 2017-10-11 17:04:07,880 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 1 with 1 2017-10-11 17:04:07,880 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node3', 'Node4'} that its ledger of type 2 is latest 2017-10-11 17:04:07,880 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node2 found from ledger status LEDGER_STATUS{'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1} that it does not need catchup 2017-10-11 17:04:07,880 | DEBUG | node.py (1479) | preLedgerCatchUp | Node2 going to process any ordered requests before starting catchup. 2017-10-11 17:04:07,881 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node2:0 going to remove 0 Ordered messages from outbox 2017-10-11 17:04:07,881 | DEBUG | node.py (1888) | force_process_ordered | Node2 processed 0 Ordered batches for instance 0 before starting catch up 2017-10-11 17:04:07,881 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node2:1 going to remove 0 Ordered messages from outbox 2017-10-11 17:04:07,881 | DEBUG | node.py (1888) | force_process_ordered | Node2 processed 0 Ordered batches for instance 1 before starting catch up 2017-10-11 17:04:07,881 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-10-11 17:04:07,881 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-10-11 17:04:07,881 | INFO | node.py (1489) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 2 2017-10-11 17:04:07,882 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 1 with 1 2017-10-11 17:04:07,882 | DEBUG | pool_config.py ( 35) | processLedger | processing config ledger for any POOL_CONFIGs 2017-10-11 17:04:07,882 | DEBUG | upgrader.py ( 208) | processLedger | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb processing config ledger for any upgrades 2017-10-11 17:04:07,883 | INFO | upgrader.py ( 214) | processLedger | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb found upgrade START txn OrderedDict([('action', 'start'), ('force', True), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade37'), ('reqId', 1507741413733229), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-10-11T14:30:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-10-11T14:20:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-10-11T14:25:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-10-11T14:15:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz'), ('timeout', 10), ('txnTime', 1507741413), ('type', '109'), ('version', '1.1.37'), ('seqNo', 1)]) 2017-10-11 17:04:07,883 | INFO | upgrader.py ( 292) | handleUpgradeTxn | Node 'Node2' handles upgrade txn OrderedDict([('action', 'start'), ('force', True), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade37'), ('reqId', 1507741413733229), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-10-11T14:30:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-10-11T14:20:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-10-11T14:25:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-10-11T14:15:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', '4WCgZKWQtWJxH8rXhpJxKzyuWEuCDKLZqR8VbAHeb4Vfk9RjkQ5oSDz7FnD5tNYbXL97suiRXwySXdq2jvd18tXz'), ('timeout', 10), ('txnTime', 1507741413), ('type', '109'), ('version', '1.1.37'), ('seqNo', 1)]) 2017-10-11 17:04:07,883 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node1'] 2017-10-11 17:04:07,883 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node1 for ledger status of ledger 1 2017-10-11 17:04:07,883 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:04:07,884 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node2 for ledger status of ledger 1 2017-10-11 17:04:07,884 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-10-11 17:04:07,884 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node3 for ledger status of ledger 1 2017-10-11 17:04:07,884 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-10-11 17:04:07,884 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node4 for ledger status of ledger 1 2017-10-11 17:04:07,884 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node2 going to process 0 stashed ledger statuses for ledger 1 2017-10-11 17:04:07,884 | INFO | upgrader.py ( 150) | should_notify_about_upgrade_result | Node's 'Node2' last upgrade txn is None 2017-10-11 17:04:07,885 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 2, caught up 0 in total 2017-10-11 17:04:07,885 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1} from Node4 2017-10-11 17:04:07,885 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 1 with 1 2017-10-11 17:04:07,885 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 1 with 1 2017-10-11 17:04:07,886 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-10-11 17:04:07,886 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-10-11 17:04:07,886 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-10-11 17:04:07,886 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}']) 2017-10-11 17:04:07,887 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node1: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:04:07,887 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null,"op":"BATCH"}' to Node1 2017-10-11 17:04:07,887 | DEBUG | zstack.py ( 728) | transmit | Remote Node1 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:04:07,887 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-10-11 17:04:07,887 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}']) 2017-10-11 17:04:07,887 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node3: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:04:07,887 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null,"op":"BATCH"}' to Node3 2017-10-11 17:04:07,887 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 4 msgs to Node4 into one transmission 2017-10-11 17:04:07,888 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', b'{"msg":{"ledgerId":2,"merkleRoot":"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ","ppSeqNo":null,"txnSeqNo":1,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":2,"merkleRoot":"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ","ppSeqNo":null,"txnSeqNo":1,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}']) 2017-10-11 17:04:07,888 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node4: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":1,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":1,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:04:07,888 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":1,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":1,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null,"op":"BATCH"}' to Node4 2017-10-11 17:04:07,889 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:04:07,889 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}'], 'signature': None} 2017-10-11 17:04:07,889 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}'], 'signature': None, 'op': 'BATCH'}, 'Node3') 2017-10-11 17:04:07,889 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}'], 'signature': None} 2017-10-11 17:04:07,890 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:04:07,890 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}}, 'Node3') 2017-10-11 17:04:07,890 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:04:07,890 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:04:07,890 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}}, 'Node3') 2017-10-11 17:04:07,890 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:04:07,891 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-10-11 17:04:07,891 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-10-11 17:04:07,891 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-10-11 17:04:07,891 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":1,"merkleRoot":"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq","ppSeqNo":null,"txnSeqNo":11,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', b'{"msg":{"ledgerId":1,"merkleRoot":"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq","ppSeqNo":null,"txnSeqNo":11,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}']) 2017-10-11 17:04:07,892 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node3: b'{"messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":11,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":11,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:04:07,892 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":11,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":11,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null,"op":"BATCH"}' to Node3 2017-10-11 17:04:07,904 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:04:07,905 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq","ppSeqNo":null,"txnSeqNo":11,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq","ppSeqNo":null,"txnSeqNo":11,"viewNo":null}}'], 'signature': None} 2017-10-11 17:04:07,905 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq","ppSeqNo":null,"txnSeqNo":11,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq","ppSeqNo":null,"txnSeqNo":11,"viewNo":null}}'], 'signature': None, 'op': 'BATCH'}, 'Node3') 2017-10-11 17:04:07,905 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq","ppSeqNo":null,"txnSeqNo":11,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq","ppSeqNo":null,"txnSeqNo":11,"viewNo":null}}'], 'signature': None} 2017-10-11 17:04:07,905 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg': {'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:04:07,905 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}}, 'Node3') 2017-10-11 17:04:07,905 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:04:07,906 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg': {'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:04:07,906 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}}, 'Node3') 2017-10-11 17:04:07,906 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:04:07,906 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11} from Node3 2017-10-11 17:04:07,906 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 11 with 11 2017-10-11 17:04:07,906 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 11 with 11 2017-10-11 17:04:07,907 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11} from Node3 2017-10-11 17:04:07,907 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 11 with 11 2017-10-11 17:04:07,907 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 11 with 11 2017-10-11 17:04:07,930 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:04:07,931 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq","ppSeqNo":null,"txnSeqNo":11,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq","ppSeqNo":null,"txnSeqNo":11,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"name":"Node1","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,11,"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq"],[2,1,"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ"]],"viewNo":0,"op":"VIEW_CHANGE_DONE"}', '{"name":"Node1","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,11,"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq"],[2,1,"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ"]],"viewNo":0,"op":"VIEW_CHANGE_DONE"}'], 'signature': None} 2017-10-11 17:04:07,931 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq","ppSeqNo":null,"txnSeqNo":11,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq","ppSeqNo":null,"txnSeqNo":11,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"name":"Node1","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,11,"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq"],[2,1,"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ"]],"viewNo":0,"op":"VIEW_CHANGE_DONE"}', '{"name":"Node1","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,11,"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq"],[2,1,"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ"]],"viewNo":0,"op":"VIEW_CHANGE_DONE"}'], 'signature': None, 'op': 'BATCH'}, 'Node4') 2017-10-11 17:04:07,931 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq","ppSeqNo":null,"txnSeqNo":11,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq","ppSeqNo":null,"txnSeqNo":11,"viewNo":null},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"name":"Node1","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,11,"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq"],[2,1,"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ"]],"viewNo":0,"op":"VIEW_CHANGE_DONE"}', '{"name":"Node1","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,11,"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq"],[2,1,"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ"]],"viewNo":0,"op":"VIEW_CHANGE_DONE"}'], 'signature': None} 2017-10-11 17:04:07,931 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_RESPONSE{'msg': {'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:04:07,931 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}}, 'Node4') 2017-10-11 17:04:07,932 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:04:07,932 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_RESPONSE{'msg': {'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:04:07,932 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}}, 'Node4') 2017-10-11 17:04:07,932 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:04:07,932 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:04:07,932 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}}, 'Node4') 2017-10-11 17:04:07,932 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:04:07,932 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:04:07,933 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}}, 'Node4') 2017-10-11 17:04:07,933 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:04:07,933 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'], [2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ']], 'viewNo': 0} 2017-10-11 17:04:07,933 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'name': 'Node1', 'op': 'VIEW_CHANGE_DONE', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'], [2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ']], 'viewNo': 0}, 'Node4') 2017-10-11 17:04:07,933 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'], [2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ']], 'viewNo': 0} 2017-10-11 17:04:07,934 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'], [2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ']], 'viewNo': 0} 2017-10-11 17:04:07,934 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'name': 'Node1', 'op': 'VIEW_CHANGE_DONE', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'], [2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ']], 'viewNo': 0}, 'Node4') 2017-10-11 17:04:07,934 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'], [2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ']], 'viewNo': 0} 2017-10-11 17:04:07,934 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11} from Node4 2017-10-11 17:04:07,934 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 11 with 11 2017-10-11 17:04:07,934 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 11 with 11 2017-10-11 17:04:07,934 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node3', 'Node4'} that its ledger of type 1 is latest 2017-10-11 17:04:07,934 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node2 found from ledger status LEDGER_STATUS{'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11} that it does not need catchup 2017-10-11 17:04:07,935 | DEBUG | node.py (1479) | preLedgerCatchUp | Node2 going to process any ordered requests before starting catchup. 2017-10-11 17:04:07,935 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node2:0 going to remove 0 Ordered messages from outbox 2017-10-11 17:04:07,935 | DEBUG | node.py (1888) | force_process_ordered | Node2 processed 0 Ordered batches for instance 0 before starting catch up 2017-10-11 17:04:07,935 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node2:1 going to remove 0 Ordered messages from outbox 2017-10-11 17:04:07,935 | DEBUG | node.py (1888) | force_process_ordered | Node2 processed 0 Ordered batches for instance 1 before starting catch up 2017-10-11 17:04:07,935 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-10-11 17:04:07,935 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-10-11 17:04:07,935 | INFO | node.py (1489) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 1 2017-10-11 17:04:07,935 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 11 with 11 2017-10-11 17:04:07,935 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 1, caught up 0 in total 2017-10-11 17:04:07,936 | DEBUG | node.py (1609) | num_txns_caught_up_in_last_catchup | Node2 caught up to 0 txns in the last catchup 2017-10-11 17:04:07,936 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-10-11 17:04:07,936 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-10-11 17:04:07,936 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node2 needs 2 ViewChangeDone messages 2017-10-11 17:04:07,936 | DEBUG | node.py (1568) | caught_up_for_current_view | Node2 does not have view change quorum for view 0 2017-10-11 17:04:07,936 | DEBUG | node.py (1552) | is_catchup_needed | Node2 is not caught up for the current view 0 2017-10-11 17:04:07,936 | DEBUG | node.py (1609) | num_txns_caught_up_in_last_catchup | Node2 caught up to 0 txns in the last catchup 2017-10-11 17:04:07,936 | DEBUG | node.py (1557) | is_catchup_needed | Node2 ordered till last prepared certificate 2017-10-11 17:04:07,936 | INFO | node.py (1537) | allLedgersCaughtUp | CATCH-UP: Node2 does not need any more catchups 2017-10-11 17:04:07,937 | DEBUG | primary_decider.py ( 134) | send | Node2's elector sending VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'), (2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ')], 'viewNo': 0} 2017-10-11 17:04:07,937 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node2 needs 1 ViewChangeDone messages 2017-10-11 17:04:07,937 | DEBUG | primary_selector.py ( 266) | _startSelection | Node2 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-10-11 17:04:07,937 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11} from Node4 2017-10-11 17:04:07,937 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 11 with 11 2017-10-11 17:04:07,937 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 11 with 11 2017-10-11 17:04:07,938 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-10-11 17:04:07,939 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-10-11 17:04:07,940 | DEBUG | node.py (1205) | sendToElector | Node2 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'], [2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ']], 'viewNo': 0}, 'Node4') 2017-10-11 17:04:07,941 | DEBUG | node.py (1205) | sendToElector | Node2 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'], [2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ']], 'viewNo': 0}, 'Node4') 2017-10-11 17:04:07,942 | DEBUG | node.py (2608) | send | Node2 sending message VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'), (2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ')], 'viewNo': 0} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:04:07,942 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node2's primary selector started processing of ViewChangeDone msg from Node4 : VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'], [2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ']], 'viewNo': 0} 2017-10-11 17:04:07,942 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node2 got view change quorum (2 >= 2) 2017-10-11 17:04:07,943 | DEBUG | primary_selector.py ( 209) | has_view_change_from_primary | Node2 has not received ViewChangeDone from the next primary Node1 (viewNo: 0, totalNodes: 4) 2017-10-11 17:04:07,943 | DEBUG | primary_selector.py ( 266) | _startSelection | Node2 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-10-11 17:04:07,943 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node2's primary selector started processing of ViewChangeDone msg from Node4 : VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'], [2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ']], 'viewNo': 0} 2017-10-11 17:04:07,943 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node2 got view change quorum (2 >= 2) 2017-10-11 17:04:07,943 | DEBUG | primary_selector.py ( 209) | has_view_change_from_primary | Node2 has not received ViewChangeDone from the next primary Node1 (viewNo: 0, totalNodes: 4) 2017-10-11 17:04:07,943 | DEBUG | primary_selector.py ( 266) | _startSelection | Node2 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-10-11 17:04:07,943 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"name":"Node1","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,11,"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq"],[2,1,"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ"]],"op":"VIEW_CHANGE_DONE","viewNo":0}' to Node1 2017-10-11 17:04:07,943 | DEBUG | zstack.py ( 728) | transmit | Remote Node1 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:04:07,944 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"name":"Node1","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,11,"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq"],[2,1,"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ"]],"op":"VIEW_CHANGE_DONE","viewNo":0}' to Node1 2017-10-11 17:04:07,944 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"name":"Node1","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,11,"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq"],[2,1,"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ"]],"op":"VIEW_CHANGE_DONE","viewNo":0}' to Node3 2017-10-11 17:04:07,944 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"name":"Node1","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,11,"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq"],[2,1,"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ"]],"op":"VIEW_CHANGE_DONE","viewNo":0}' to Node3 2017-10-11 17:04:07,944 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 3 msgs to Node4 into one transmission 2017-10-11 17:04:07,947 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":1,"merkleRoot":"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq","ppSeqNo":null,"txnSeqNo":11,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', b'{"msg":{"ledgerId":1,"merkleRoot":"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq","ppSeqNo":null,"txnSeqNo":11,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', b'{"name":"Node1","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,11,"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq"],[2,1,"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ"]],"op":"VIEW_CHANGE_DONE","viewNo":0}']) 2017-10-11 17:04:07,947 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node4: b'{"messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":11,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":11,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"name\\":\\"Node1\\",\\"ledgerInfo\\":[[0,4,\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\"],[1,11,\\"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq\\"],[2,1,\\"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ\\"]],\\"op\\":\\"VIEW_CHANGE_DONE\\",\\"viewNo\\":0}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:04:07,947 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":11,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":11,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"name\\":\\"Node1\\",\\"ledgerInfo\\":[[0,4,\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\"],[1,11,\\"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq\\"],[2,1,\\"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ\\"]],\\"op\\":\\"VIEW_CHANGE_DONE\\",\\"viewNo\\":0}"],"signature":null,"op":"BATCH"}' to Node4 2017-10-11 17:04:07,948 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:04:07,949 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'], [2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ']], 'viewNo': 0} 2017-10-11 17:04:07,950 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'name': 'Node1', 'op': 'VIEW_CHANGE_DONE', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'], [2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ']], 'viewNo': 0}, 'Node3') 2017-10-11 17:04:07,951 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'], [2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ']], 'viewNo': 0} 2017-10-11 17:04:07,952 | DEBUG | node.py (1205) | sendToElector | Node2 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'], [2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ']], 'viewNo': 0}, 'Node3') 2017-10-11 17:04:07,952 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node2's primary selector started processing of ViewChangeDone msg from Node3 : VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'], [2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ']], 'viewNo': 0} 2017-10-11 17:04:07,952 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node2 got view change quorum (3 >= 2) 2017-10-11 17:04:07,952 | DEBUG | primary_selector.py ( 209) | has_view_change_from_primary | Node2 has not received ViewChangeDone from the next primary Node1 (viewNo: 0, totalNodes: 4) 2017-10-11 17:04:07,953 | DEBUG | primary_selector.py ( 266) | _startSelection | Node2 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-10-11 17:04:08,832 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:04:08,832 | DEBUG | zstack.py ( 663) | handlePingPong | Node2 got ping from Node1 2017-10-11 17:04:08,833 | DEBUG | zstack.py ( 645) | sendPingPong | Node2 ponged Node1 2017-10-11 17:04:08,833 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'po' to Node1 2017-10-11 17:04:08,833 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'po' to Node1 2017-10-11 17:04:08,991 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:04:08,991 | DEBUG | zstack.py ( 669) | handlePingPong | Node2 got pong from Node1 2017-10-11 17:04:08,991 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: BATCH{'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ","ppSeqNo":null,"txnSeqNo":1,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ","ppSeqNo":null,"txnSeqNo":1,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq","ppSeqNo":null,"txnSeqNo":11,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq","ppSeqNo":null,"txnSeqNo":11,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,11,"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq"],[2,1,"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ"]],"name":"Node1","viewNo":0,"op":"VIEW_CHANGE_DONE"}'], 'signature': None} 2017-10-11 17:04:08,992 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ","ppSeqNo":null,"txnSeqNo":1,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ","ppSeqNo":null,"txnSeqNo":1,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq","ppSeqNo":null,"txnSeqNo":11,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq","ppSeqNo":null,"txnSeqNo":11,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,11,"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq"],[2,1,"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ"]],"name":"Node1","viewNo":0,"op":"VIEW_CHANGE_DONE"}'], 'op': 'BATCH', 'signature': None}, 'Node1') 2017-10-11 17:04:08,992 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ","ppSeqNo":null,"txnSeqNo":1,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":2},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ","ppSeqNo":null,"txnSeqNo":1,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq","ppSeqNo":null,"txnSeqNo":11,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"params":{"ledgerId":1},"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq","ppSeqNo":null,"txnSeqNo":11,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,11,"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq"],[2,1,"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ"]],"name":"Node1","viewNo":0,"op":"VIEW_CHANGE_DONE"}'], 'signature': None} 2017-10-11 17:04:08,992 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:04:08,992 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}}, 'Node1') 2017-10-11 17:04:08,992 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:04:08,992 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:04:08,992 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}}, 'Node1') 2017-10-11 17:04:08,992 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:04:08,993 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:04:08,993 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}}, 'Node1') 2017-10-11 17:04:08,993 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:04:08,993 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:04:08,993 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}}, 'Node1') 2017-10-11 17:04:08,993 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:04:08,993 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:04:08,993 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}}, 'Node1') 2017-10-11 17:04:08,993 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:04:08,994 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:04:08,994 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}}, 'Node1') 2017-10-11 17:04:08,994 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:04:08,994 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg': {'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:04:08,994 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}}, 'Node1') 2017-10-11 17:04:08,994 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:04:08,994 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg': {'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:04:08,995 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}}, 'Node1') 2017-10-11 17:04:08,995 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:04:08,995 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'], [2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ']], 'viewNo': 0} 2017-10-11 17:04:08,995 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'name': 'Node1', 'op': 'VIEW_CHANGE_DONE', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'], [2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ']], 'viewNo': 0}, 'Node1') 2017-10-11 17:04:08,996 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'], [2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ']], 'viewNo': 0} 2017-10-11 17:04:08,996 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node1'] 2017-10-11 17:04:08,996 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node1'] 2017-10-11 17:04:08,996 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node1'] 2017-10-11 17:04:08,997 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node1'] 2017-10-11 17:04:08,997 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1} from Node1 2017-10-11 17:04:08,997 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 1 with 1 2017-10-11 17:04:08,997 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 1 with 1 2017-10-11 17:04:08,997 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node1', 'Node4'} that its ledger of type 2 is latest 2017-10-11 17:04:08,998 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1} from Node1 2017-10-11 17:04:08,998 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 1 with 1 2017-10-11 17:04:08,998 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 1 with 1 2017-10-11 17:04:08,998 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node1', 'Node4'} that its ledger of type 2 is latest 2017-10-11 17:04:08,998 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11} from Node1 2017-10-11 17:04:08,998 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 11 with 11 2017-10-11 17:04:08,999 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 11 with 11 2017-10-11 17:04:08,999 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node1', 'Node4'} that its ledger of type 1 is latest 2017-10-11 17:04:08,999 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11} from Node1 2017-10-11 17:04:08,999 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 11 with 11 2017-10-11 17:04:08,999 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 11 with 11 2017-10-11 17:04:08,999 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node1', 'Node4'} that its ledger of type 1 is latest 2017-10-11 17:04:08,999 | DEBUG | node.py (1205) | sendToElector | Node2 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'], [2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ']], 'viewNo': 0}, 'Node1') 2017-10-11 17:04:09,000 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node2's primary selector started processing of ViewChangeDone msg from Node1 : VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'], [2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ']], 'viewNo': 0} 2017-10-11 17:04:09,000 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node2 got view change quorum (4 >= 2) 2017-10-11 17:04:09,000 | DEBUG | primary_selector.py ( 215) | has_view_change_from_primary | Node2 received ViewChangeDone from primary Node1 2017-10-11 17:04:09,000 | DEBUG | primary_selector.py ( 238) | has_sufficient_same_view_change_done_messages | Node2 found acceptable primary Node1 and ledger info ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'), (2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ')) 2017-10-11 17:04:09,000 | DEBUG | primary_selector.py ( 281) | _startSelection | Node2 starting selection 2017-10-11 17:04:09,001 | DISPLAY | primary_selector.py ( 291) | _startSelection | PRIMARY SELECTION: Node2:0 selected primary Node1:0 for instance 0 (view 0) 2017-10-11 17:04:09,001 | INFO | node.py ( 510) | start_participating | Node2 started participating 2017-10-11 17:04:09,001 | DEBUG | replica.py ( 408) | primaryName | Node2:0 setting primaryName for view no 0 to: Node1:0 2017-10-11 17:04:09,001 | DEBUG | replica.py (1694) | _gc | Node2:0 cleaning up till (0, 0) 2017-10-11 17:04:09,005 | DEBUG | replica.py (1709) | _gc | Node2:0 found 0 3-phase keys to clean 2017-10-11 17:04:09,005 | DEBUG | replica.py (1711) | _gc | Node2:0 found 0 request keys to clean 2017-10-11 17:04:09,005 | DEBUG | replica.py ( 313) | h | Node2:0 set watermarks as 0 300 2017-10-11 17:04:09,006 | DISPLAY | primary_selector.py ( 315) | _startSelection | VIEW CHANGE: Node2:0 declares view change 0 as completed for instance 0, new primary is Node1:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'), (2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ')] 2017-10-11 17:04:09,006 | DISPLAY | primary_selector.py ( 291) | _startSelection | PRIMARY SELECTION: Node2:1 selected primary Node2:1 for instance 1 (view 0) 2017-10-11 17:04:09,006 | DEBUG | replica.py ( 408) | primaryName | Node2:1 setting primaryName for view no 0 to: Node2:1 2017-10-11 17:04:09,006 | DEBUG | replica.py (1694) | _gc | Node2:1 cleaning up till (0, 0) 2017-10-11 17:04:09,006 | DEBUG | replica.py (1709) | _gc | Node2:1 found 0 3-phase keys to clean 2017-10-11 17:04:09,006 | DEBUG | replica.py (1711) | _gc | Node2:1 found 0 request keys to clean 2017-10-11 17:04:09,007 | DEBUG | replica.py ( 313) | h | Node2:1 set watermarks as 0 300 2017-10-11 17:04:09,007 | DEBUG | replica.py ( 514) | _setup_for_non_master | Node2:1 Setting last ordered for non-master as (0, 0) 2017-10-11 17:04:09,007 | DEBUG | replica.py ( 323) | last_ordered_3pc | Node2:1 set last ordered as (0, 0) 2017-10-11 17:04:09,007 | DISPLAY | primary_selector.py ( 315) | _startSelection | VIEW CHANGE: Node2:1 declares view change 0 as completed for instance 1, new primary is Node2:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'), (2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ')] 2017-10-11 17:04:09,007 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 4 msgs to Node1 into one transmission 2017-10-11 17:04:09,007 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":2,"merkleRoot":"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ","ppSeqNo":null,"txnSeqNo":1,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":2,"merkleRoot":"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ","ppSeqNo":null,"txnSeqNo":1,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":1,"merkleRoot":"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq","ppSeqNo":null,"txnSeqNo":11,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', b'{"msg":{"ledgerId":1,"merkleRoot":"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq","ppSeqNo":null,"txnSeqNo":11,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}']) 2017-10-11 17:04:09,007 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node1: b'{"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":1,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":1,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":11,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":11,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:04:09,008 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":1,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":1,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":11,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":11,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null,"op":"BATCH"}' to Node1 2017-10-11 17:04:09,009 | DEBUG | keep_in_touch.py ( 68) | conns | Node2's connections changed from {'Node3', 'Node4'} to {'Node1', 'Node3', 'Node4'} 2017-10-11 17:04:09,009 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node2 now connected to Node1 2017-10-11 17:04:09,009 | DEBUG | motor.py ( 34) | set_status | Node2 changing status from started_hungry to started 2017-10-11 17:04:09,009 | DEBUG | node.py ( 941) | checkInstances | Node2 choosing to start election on the basis of count 4 and nodes {'Node1', 'Node3', 'Node4'} 2017-10-11 17:04:09,010 | DEBUG | node.py ( 902) | send_current_state_to_lagging_node | Node2 sending current state CURRENT_STATE{'primary': [VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'), (2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ')), 'viewNo': 0}], 'viewNo': 0} to lagged node Node1 2017-10-11 17:04:09,010 | DEBUG | node.py (2608) | send | Node2 sending message CURRENT_STATE{'primary': [VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'), (2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ')), 'viewNo': 0}], 'viewNo': 0} to 1 recipients: ['Node1'] 2017-10-11 17:04:09,010 | DEBUG | node.py (2608) | send | Node2 sending message LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4} to 1 recipients: ['Node1'] 2017-10-11 17:04:09,011 | DEBUG | node.py (2608) | send | Node2 sending message LEDGER_STATUS{'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11} to 1 recipients: ['Node1'] 2017-10-11 17:04:09,011 | DEBUG | node.py (2608) | send | Node2 sending message LEDGER_STATUS{'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1} to 1 recipients: ['Node1'] 2017-10-11 17:04:09,011 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:04:09,012 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: BATCH{'messages': ['{"viewNo":0,"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,11,"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq"],[2,1,"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ"]],"name":"Node1","viewNo":0}],"op":"CURRENT_STATE"}', '{"txnSeqNo":4,"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":null,"ppSeqNo":null,"op":"LEDGER_STATUS"}', '{"txnSeqNo":11,"ledgerId":1,"merkleRoot":"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq","viewNo":null,"ppSeqNo":null,"op":"LEDGER_STATUS"}', '{"txnSeqNo":1,"ledgerId":2,"merkleRoot":"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ","viewNo":null,"ppSeqNo":null,"op":"LEDGER_STATUS"}'], 'signature': None} 2017-10-11 17:04:09,012 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"viewNo":0,"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,11,"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq"],[2,1,"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ"]],"name":"Node1","viewNo":0}],"op":"CURRENT_STATE"}', '{"txnSeqNo":4,"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":null,"ppSeqNo":null,"op":"LEDGER_STATUS"}', '{"txnSeqNo":11,"ledgerId":1,"merkleRoot":"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq","viewNo":null,"ppSeqNo":null,"op":"LEDGER_STATUS"}', '{"txnSeqNo":1,"ledgerId":2,"merkleRoot":"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ","viewNo":null,"ppSeqNo":null,"op":"LEDGER_STATUS"}'], 'op': 'BATCH', 'signature': None}, 'Node1') 2017-10-11 17:04:09,012 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"viewNo":0,"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,11,"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq"],[2,1,"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ"]],"name":"Node1","viewNo":0}],"op":"CURRENT_STATE"}', '{"txnSeqNo":4,"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","viewNo":null,"ppSeqNo":null,"op":"LEDGER_STATUS"}', '{"txnSeqNo":11,"ledgerId":1,"merkleRoot":"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq","viewNo":null,"ppSeqNo":null,"op":"LEDGER_STATUS"}', '{"txnSeqNo":1,"ledgerId":2,"merkleRoot":"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ","viewNo":null,"ppSeqNo":null,"op":"LEDGER_STATUS"}'], 'signature': None} 2017-10-11 17:04:09,013 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: CURRENT_STATE{'primary': [{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'], [2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ']], 'viewNo': 0}], 'viewNo': 0} 2017-10-11 17:04:09,013 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'primary': [{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'], [2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ']], 'viewNo': 0}], 'op': 'CURRENT_STATE', 'viewNo': 0}, 'Node1') 2017-10-11 17:04:09,013 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox CURRENT_STATE{'primary': [{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'], [2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ']], 'viewNo': 0}], 'viewNo': 0} 2017-10-11 17:04:09,013 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4} 2017-10-11 17:04:09,013 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'ppSeqNo': None, 'op': 'LEDGER_STATUS', 'ledgerId': 0, 'viewNo': None}, 'Node1') 2017-10-11 17:04:09,013 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4} 2017-10-11 17:04:09,014 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: LEDGER_STATUS{'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11} 2017-10-11 17:04:09,014 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'txnSeqNo': 11, 'ppSeqNo': None, 'op': 'LEDGER_STATUS', 'ledgerId': 1, 'viewNo': None}, 'Node1') 2017-10-11 17:04:09,014 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox LEDGER_STATUS{'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11} 2017-10-11 17:04:09,014 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: LEDGER_STATUS{'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1} 2017-10-11 17:04:09,014 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'txnSeqNo': 1, 'ppSeqNo': None, 'op': 'LEDGER_STATUS', 'ledgerId': 2, 'viewNo': None}, 'Node1') 2017-10-11 17:04:09,014 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox LEDGER_STATUS{'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1} 2017-10-11 17:04:09,015 | DEBUG | node.py ( 907) | process_current_state_message | Node2 processing current state CURRENT_STATE{'primary': [{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'], [2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ']], 'viewNo': 0}], 'viewNo': 0} from Node1 2017-10-11 17:04:09,015 | DEBUG | node.py (1205) | sendToElector | Node2 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'], [2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ']], 'viewNo': 0}, 'Node1') 2017-10-11 17:04:09,015 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4} from Node1 2017-10-11 17:04:09,017 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 17:04:09,017 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 17:04:09,017 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 11} from Node1 2017-10-11 17:04:09,017 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 11 with 11 2017-10-11 17:04:09,017 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 11 with 11 2017-10-11 17:04:09,017 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node1', 'Node4'} that its ledger of type 1 is latest 2017-10-11 17:04:09,018 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 1} from Node1 2017-10-11 17:04:09,018 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 1 with 1 2017-10-11 17:04:09,018 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 1 with 1 2017-10-11 17:04:09,018 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node1', 'Node4'} that its ledger of type 2 is latest 2017-10-11 17:04:09,018 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node2's primary selector started processing of ViewChangeDone msg from Node1 : VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'], [2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ']], 'viewNo': 0} 2017-10-11 17:04:09,018 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message VIEW_CHANGE_DONE{'name': 'Node1', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 11, '6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq'], [2, 1, 'AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ']], 'viewNo': 0} because it already decided primary which is Node1:0 2017-10-11 17:04:09,019 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 4 msgs to Node1 into one transmission 2017-10-11 17:04:09,019 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,11,"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq"],[2,1,"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ"]],"name":"Node1","viewNo":0}],"op":"CURRENT_STATE","viewNo":0}', b'{"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ledgerId":0,"ppSeqNo":null,"op":"LEDGER_STATUS","txnSeqNo":4,"viewNo":null}', b'{"merkleRoot":"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq","ledgerId":1,"ppSeqNo":null,"op":"LEDGER_STATUS","txnSeqNo":11,"viewNo":null}', b'{"merkleRoot":"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ","ledgerId":2,"ppSeqNo":null,"op":"LEDGER_STATUS","txnSeqNo":1,"viewNo":null}']) 2017-10-11 17:04:09,019 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node1: b'{"messages":["{\\"primary\\":[{\\"ledgerInfo\\":[[0,4,\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\"],[1,11,\\"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq\\"],[2,1,\\"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ\\"]],\\"name\\":\\"Node1\\",\\"viewNo\\":0}],\\"op\\":\\"CURRENT_STATE\\",\\"viewNo\\":0}","{\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ledgerId\\":0,\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"txnSeqNo\\":4,\\"viewNo\\":null}","{\\"merkleRoot\\":\\"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq\\",\\"ledgerId\\":1,\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"txnSeqNo\\":11,\\"viewNo\\":null}","{\\"merkleRoot\\":\\"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ\\",\\"ledgerId\\":2,\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"txnSeqNo\\":1,\\"viewNo\\":null}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:04:09,019 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"primary\\":[{\\"ledgerInfo\\":[[0,4,\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\"],[1,11,\\"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq\\"],[2,1,\\"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ\\"]],\\"name\\":\\"Node1\\",\\"viewNo\\":0}],\\"op\\":\\"CURRENT_STATE\\",\\"viewNo\\":0}","{\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ledgerId\\":0,\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"txnSeqNo\\":4,\\"viewNo\\":null}","{\\"merkleRoot\\":\\"6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq\\",\\"ledgerId\\":1,\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"txnSeqNo\\":11,\\"viewNo\\":null}","{\\"merkleRoot\\":\\"AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ\\",\\"ledgerId\\":2,\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"txnSeqNo\\":1,\\"viewNo\\":null}"],"signature":null,"op":"BATCH"}' to Node1 2017-10-11 17:04:09,887 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action dump_json_file with id 4 2017-10-11 17:04:16,809 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 1 2017-10-11 17:04:16,809 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:04:16,810 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:04:16,810 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-10-11 17:04:16,810 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:04:16,810 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:04:16,810 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:04:16,811 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 6 to run in 10 seconds 2017-10-11 17:04:21,896 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:04:21,897 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:04:21,897 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:04:21,898 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:04:26,822 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 6 2017-10-11 17:04:26,822 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:04:26,822 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:04:26,822 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-10-11 17:04:26,823 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:04:26,823 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:04:26,823 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:04:26,823 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 7 to run in 10 seconds 2017-10-11 17:04:36,827 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 7 2017-10-11 17:04:36,827 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:04:36,827 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:04:36,827 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-10-11 17:04:36,827 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:04:36,827 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:04:36,827 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:04:36,828 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 8 to run in 10 seconds 2017-10-11 17:04:36,898 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:04:36,898 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:04:36,898 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:04:36,899 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:04:46,831 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 8 2017-10-11 17:04:46,832 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:04:46,832 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:04:46,832 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-10-11 17:04:46,832 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:04:46,832 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:04:46,832 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:04:46,832 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 9 to run in 10 seconds 2017-10-11 17:04:51,900 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:04:51,900 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:04:51,900 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:04:51,900 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:04:56,845 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 9 2017-10-11 17:04:56,845 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:04:56,845 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:04:56,846 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-10-11 17:04:56,846 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:04:56,846 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:04:56,846 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:04:56,846 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 10 to run in 10 seconds 2017-10-11 17:05:00,992 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2C got 1 messages through listener 2017-10-11 17:05:01,007 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1507741500981867 2017-10-11 17:05:01,008 | TRACE | node.py (1388) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT3'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU', 'reqId': 1507741500981867} 2017-10-11 17:05:01,008 | DEBUG | node.py (1434) | processClientInBox | Node2C processing b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' request SafeRequest: {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT3'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU', 'reqId': 1507741500981867} 2017-10-11 17:05:01,008 | DEBUG | node.py (1708) | processRequest | Node2 received client request: SafeRequest: {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT3'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU', 'reqId': 1507741500981867} from b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' 2017-10-11 17:05:01,010 | TRACE | propagator.py ( 165) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT3'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU', 'reqId': 1507741500981867} 2017-10-11 17:05:01,011 | INFO | propagator.py ( 148) | propagate | Node2 propagating request ('V4SGRU86Z58d6TV7PBUe6f', 1507741500981867) from client b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' 2017-10-11 17:05:01,012 | DEBUG | node.py (2608) | send | Node2 sending message PROPAGATE{'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%', 'request': {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT3'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU', 'reqId': 1507741500981867}} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:05:01,012 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT3'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU', 'reqId': 1507741500981867} to its replicas since not finalised 2017-10-11 17:05:01,012 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1507741500981867,"op":"REQACK"}' to b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' through listener socket 2017-10-11 17:05:01,013 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"senderClient":"fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%","request":{"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT3"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU","reqId":1507741500981867},"op":"PROPAGATE"}' to Node1 2017-10-11 17:05:01,013 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"senderClient":"fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%","request":{"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT3"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU","reqId":1507741500981867},"op":"PROPAGATE"}' to Node1 2017-10-11 17:05:01,013 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"senderClient":"fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%","request":{"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT3"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU","reqId":1507741500981867},"op":"PROPAGATE"}' to Node3 2017-10-11 17:05:01,014 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"senderClient":"fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%","request":{"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT3"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU","reqId":1507741500981867},"op":"PROPAGATE"}' to Node3 2017-10-11 17:05:01,014 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"senderClient":"fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%","request":{"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT3"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU","reqId":1507741500981867},"op":"PROPAGATE"}' to Node4 2017-10-11 17:05:01,014 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"senderClient":"fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%","request":{"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT3"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU","reqId":1507741500981867},"op":"PROPAGATE"}' to Node4 2017-10-11 17:05:01,015 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:05:01,017 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1507741500981867 2017-10-11 17:05:01,018 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: PROPAGATE{'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%', 'request': {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT3'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU', 'reqId': 1507741500981867}} 2017-10-11 17:05:01,018 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%', 'request': {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT3'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU', 'reqId': 1507741500981867}, 'op': 'PROPAGATE'}, 'Node4') 2017-10-11 17:05:01,018 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%', 'request': {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT3'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU', 'reqId': 1507741500981867}} 2017-10-11 17:05:01,018 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%', 'request': {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT3'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU', 'reqId': 1507741500981867}} 2017-10-11 17:05:01,018 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT3'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU', 'reqId': 1507741500981867} 2017-10-11 17:05:01,018 | DEBUG | propagator.py ( 214) | forward | Node2 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1507741500981867) to 0 replicas 2017-10-11 17:05:01,019 | DEBUG | replica.py ( 667) | create3PCBatch | Node2:1 creating batch 1 for ledger 1 with state root None 2017-10-11 17:05:01,020 | DEBUG | replica.py ( 700) | create3PCBatch | Node2:1 created a PRE-PREPARE with 1 requests for ledger 1 2017-10-11 17:05:01,020 | DEBUG | replica.py (2193) | send | Node2:1 sending PrePrepare 2017-10-11 17:05:01,020 | TRACE | replica.py (2194) | send | Node2:1 sending PREPREPARE{'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741500981867)], 'ppSeqNo': 1, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741501, 'discarded': 1, 'digest': '6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897', 'viewNo': 0} 2017-10-11 17:05:01,020 | DEBUG | replica.py ( 614) | trackBatches | Node2:1 tracking batch for PREPREPARE{'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741500981867)], 'ppSeqNo': 1, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741501, 'discarded': 1, 'digest': '6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897', 'viewNo': 0} with state root None 2017-10-11 17:05:01,020 | DEBUG | node.py (2608) | send | Node2 sending message PREPREPARE{'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741500981867)], 'ppSeqNo': 1, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741501, 'discarded': 1, 'digest': '6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897', 'viewNo': 0} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:05:01,021 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":null,"digest":"6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897","discarded":1,"ledgerId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741500981867]],"ppSeqNo":1,"op":"PREPREPARE","instId":1,"ppTime":1507741501,"viewNo":0,"txnRootHash":null}' to Node1 2017-10-11 17:05:01,022 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":null,"digest":"6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897","discarded":1,"ledgerId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741500981867]],"ppSeqNo":1,"op":"PREPREPARE","instId":1,"ppTime":1507741501,"viewNo":0,"txnRootHash":null}' to Node1 2017-10-11 17:05:01,022 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":null,"digest":"6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897","discarded":1,"ledgerId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741500981867]],"ppSeqNo":1,"op":"PREPREPARE","instId":1,"ppTime":1507741501,"viewNo":0,"txnRootHash":null}' to Node3 2017-10-11 17:05:01,022 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":null,"digest":"6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897","discarded":1,"ledgerId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741500981867]],"ppSeqNo":1,"op":"PREPREPARE","instId":1,"ppTime":1507741501,"viewNo":0,"txnRootHash":null}' to Node3 2017-10-11 17:05:01,023 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":null,"digest":"6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897","discarded":1,"ledgerId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741500981867]],"ppSeqNo":1,"op":"PREPREPARE","instId":1,"ppTime":1507741501,"viewNo":0,"txnRootHash":null}' to Node4 2017-10-11 17:05:01,028 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":null,"digest":"6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897","discarded":1,"ledgerId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741500981867]],"ppSeqNo":1,"op":"PREPREPARE","instId":1,"ppTime":1507741501,"viewNo":0,"txnRootHash":null}' to Node4 2017-10-11 17:05:01,029 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-10-11 17:05:01,031 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1507741500981867 2017-10-11 17:05:01,031 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PROPAGATE{'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%', 'request': {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT3'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU', 'reqId': 1507741500981867}} 2017-10-11 17:05:01,031 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%', 'request': {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT3'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU', 'reqId': 1507741500981867}, 'op': 'PROPAGATE'}, 'Node3') 2017-10-11 17:05:01,032 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%', 'request': {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT3'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU', 'reqId': 1507741500981867}} 2017-10-11 17:05:01,032 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1507741500981867 2017-10-11 17:05:01,032 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PROPAGATE{'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%', 'request': {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT3'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU', 'reqId': 1507741500981867}} 2017-10-11 17:05:01,033 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%', 'request': {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT3'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU', 'reqId': 1507741500981867}, 'op': 'PROPAGATE'}, 'Node1') 2017-10-11 17:05:01,033 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%', 'request': {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT3'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU', 'reqId': 1507741500981867}} 2017-10-11 17:05:01,033 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%', 'request': {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT3'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU', 'reqId': 1507741500981867}} 2017-10-11 17:05:01,033 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT3'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU', 'reqId': 1507741500981867} 2017-10-11 17:05:01,033 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT3'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU', 'reqId': 1507741500981867} to its replicas since already forwarded 2017-10-11 17:05:01,033 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%', 'request': {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT3'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU', 'reqId': 1507741500981867}} 2017-10-11 17:05:01,034 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT3'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU', 'reqId': 1507741500981867} 2017-10-11 17:05:01,034 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT3'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU', 'reqId': 1507741500981867} to its replicas since already forwarded 2017-10-11 17:05:01,058 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-10-11 17:05:01,059 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PREPREPARE{'ledgerId': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1507741500981867]], 'ppSeqNo': 1, 'stateRootHash': '24sLDAYWjNXuoxUhsmcVPxLg8NK9vaWZAW75AQGrCtfE', 'txnRootHash': 'wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2', 'instId': 0, 'ppTime': 1507741501, 'discarded': 1, 'digest': '6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897', 'viewNo': 0} 2017-10-11 17:05:01,059 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'ledgerId': 1, 'digest': '6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897', 'ppSeqNo': 1, 'stateRootHash': '24sLDAYWjNXuoxUhsmcVPxLg8NK9vaWZAW75AQGrCtfE', 'op': 'PREPREPARE', 'txnRootHash': 'wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2', 'instId': 0, 'ppTime': 1507741501, 'discarded': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1507741500981867]], 'viewNo': 0}, 'Node1') 2017-10-11 17:05:01,059 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPREPARE{'ledgerId': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1507741500981867]], 'ppSeqNo': 1, 'stateRootHash': '24sLDAYWjNXuoxUhsmcVPxLg8NK9vaWZAW75AQGrCtfE', 'txnRootHash': 'wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2', 'instId': 0, 'ppTime': 1507741501, 'discarded': 1, 'digest': '6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897', 'viewNo': 0} 2017-10-11 17:05:01,060 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PREPARE{'digest': '6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897', 'ppSeqNo': 1, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741501, 'viewNo': 0} 2017-10-11 17:05:01,060 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'digest': '6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897', 'ppSeqNo': 1, 'stateRootHash': None, 'op': 'PREPARE', 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741501, 'viewNo': 0}, 'Node1') 2017-10-11 17:05:01,060 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'digest': '6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897', 'ppSeqNo': 1, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741501, 'viewNo': 0} 2017-10-11 17:05:01,062 | DEBUG | replica.py ( 823) | processPrePrepare | Node2:0 received PRE-PREPARE(0, 1) from Node1:0 at 2270586.519549229 2017-10-11 17:05:01,062 | DEBUG | replica.py (1034) | validate_pre_prepare | Node2:0 state root before processing PREPREPARE{'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741500981867)], 'ppSeqNo': 1, 'stateRootHash': '24sLDAYWjNXuoxUhsmcVPxLg8NK9vaWZAW75AQGrCtfE', 'txnRootHash': 'wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2', 'instId': 0, 'ppTime': 1507741501, 'discarded': 1, 'digest': '6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897', 'viewNo': 0} is b'\x93\xdf\xc1\x87\x14\xea\xbfa}i\x80\x90\xd6\xf2\\6K~\x81x=4\xe4\xf0o\x81G.^Y\xb5\x07', 6L8BXEMTzhyfkb3A6rx5qEZyraHJQeJ8d3RWWLwK8Hvq 2017-10-11 17:05:01,065 | DEBUG | replica.py ( 934) | doPrepare | Node2:0 Sending PREPARE(0, 1) at 2270586.522532099 2017-10-11 17:05:01,065 | DEBUG | replica.py (2193) | send | Node2:0 sending Prepare 2017-10-11 17:05:01,065 | TRACE | replica.py (2194) | send | Node2:0 sending PREPARE{'digest': '6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897', 'ppSeqNo': 1, 'stateRootHash': '24sLDAYWjNXuoxUhsmcVPxLg8NK9vaWZAW75AQGrCtfE', 'txnRootHash': 'wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2', 'instId': 0, 'ppTime': 1507741501, 'viewNo': 0} 2017-10-11 17:05:01,065 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': '6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897', 'ppSeqNo': 1, 'stateRootHash': '24sLDAYWjNXuoxUhsmcVPxLg8NK9vaWZAW75AQGrCtfE', 'txnRootHash': 'wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2', 'instId': 0, 'ppTime': 1507741501, 'viewNo': 0} 2017-10-11 17:05:01,065 | DEBUG | replica.py ( 614) | trackBatches | Node2:0 tracking batch for PREPREPARE{'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741500981867)], 'ppSeqNo': 1, 'stateRootHash': '24sLDAYWjNXuoxUhsmcVPxLg8NK9vaWZAW75AQGrCtfE', 'txnRootHash': 'wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2', 'instId': 0, 'ppTime': 1507741501, 'discarded': 1, 'digest': '6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897', 'viewNo': 0} with state root b'\x93\xdf\xc1\x87\x14\xea\xbfa}i\x80\x90\xd6\xf2\\6K~\x81x=4\xe4\xf0o\x81G.^Y\xb5\x07' 2017-10-11 17:05:01,066 | DEBUG | replica.py ( 842) | processPrePrepare | Node2:0 processed incoming PRE-PREPARE(0, 1) 2017-10-11 17:05:01,066 | DEBUG | replica.py ( 917) | tryCommit | Node2:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': '6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897', 'ppSeqNo': 1, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741501, 'viewNo': 0} 2017-10-11 17:05:01,066 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (0, 1) 2017-10-11 17:05:01,066 | DEBUG | node.py (2608) | send | Node2 sending message PREPARE{'digest': '6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897', 'ppSeqNo': 1, 'stateRootHash': '24sLDAYWjNXuoxUhsmcVPxLg8NK9vaWZAW75AQGrCtfE', 'txnRootHash': 'wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2', 'instId': 0, 'ppTime': 1507741501, 'viewNo': 0} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:05:01,067 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":"24sLDAYWjNXuoxUhsmcVPxLg8NK9vaWZAW75AQGrCtfE","viewNo":0,"digest":"6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897","ppSeqNo":1,"op":"PREPARE","instId":0,"ppTime":1507741501,"txnRootHash":"wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2"}' to Node1 2017-10-11 17:05:01,067 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":"24sLDAYWjNXuoxUhsmcVPxLg8NK9vaWZAW75AQGrCtfE","viewNo":0,"digest":"6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897","ppSeqNo":1,"op":"PREPARE","instId":0,"ppTime":1507741501,"txnRootHash":"wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2"}' to Node1 2017-10-11 17:05:01,067 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":"24sLDAYWjNXuoxUhsmcVPxLg8NK9vaWZAW75AQGrCtfE","viewNo":0,"digest":"6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897","ppSeqNo":1,"op":"PREPARE","instId":0,"ppTime":1507741501,"txnRootHash":"wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2"}' to Node3 2017-10-11 17:05:01,067 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":"24sLDAYWjNXuoxUhsmcVPxLg8NK9vaWZAW75AQGrCtfE","viewNo":0,"digest":"6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897","ppSeqNo":1,"op":"PREPARE","instId":0,"ppTime":1507741501,"txnRootHash":"wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2"}' to Node3 2017-10-11 17:05:01,068 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":"24sLDAYWjNXuoxUhsmcVPxLg8NK9vaWZAW75AQGrCtfE","viewNo":0,"digest":"6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897","ppSeqNo":1,"op":"PREPARE","instId":0,"ppTime":1507741501,"txnRootHash":"wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2"}' to Node4 2017-10-11 17:05:01,068 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":"24sLDAYWjNXuoxUhsmcVPxLg8NK9vaWZAW75AQGrCtfE","viewNo":0,"digest":"6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897","ppSeqNo":1,"op":"PREPARE","instId":0,"ppTime":1507741501,"txnRootHash":"wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2"}' to Node4 2017-10-11 17:05:01,069 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-10-11 17:05:01,074 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'digest': '6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897', 'ppSeqNo': 1, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741501, 'viewNo': 0} 2017-10-11 17:05:01,075 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'ppSeqNo': 1, 'digest': '6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897', 'stateRootHash': None, 'txnRootHash': None, 'op': 'PREPARE', 'instId': 1, 'ppTime': 1507741501, 'viewNo': 0}, 'Node4') 2017-10-11 17:05:01,075 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'digest': '6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897', 'ppSeqNo': 1, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741501, 'viewNo': 0} 2017-10-11 17:05:01,075 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'digest': '6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897', 'ppSeqNo': 1, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741501, 'viewNo': 0} 2017-10-11 17:05:01,076 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'digest': '6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897', 'ppSeqNo': 1, 'stateRootHash': None, 'op': 'PREPARE', 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741501, 'viewNo': 0}, 'Node3') 2017-10-11 17:05:01,076 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'digest': '6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897', 'ppSeqNo': 1, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741501, 'viewNo': 0} 2017-10-11 17:05:01,077 | DEBUG | replica.py ( 953) | doCommit | Node2:1 Sending COMMIT(0, 1) at 2270586.534769717 2017-10-11 17:05:01,077 | DEBUG | replica.py (2193) | send | Node2:1 sending Commit 2017-10-11 17:05:01,077 | TRACE | replica.py (2194) | send | Node2:1 sending COMMIT{'instId': 1, 'ppSeqNo': 1, 'viewNo': 0} 2017-10-11 17:05:01,078 | DEBUG | replica.py ( 929) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'ppSeqNo': 1, 'viewNo': 0} commits where f is 1 2017-10-11 17:05:01,078 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (0, 1) 2017-10-11 17:05:01,078 | DEBUG | replica.py ( 917) | tryCommit | Node2:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'digest': '6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897', 'ppSeqNo': 1, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741501, 'viewNo': 0} 2017-10-11 17:05:01,078 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (0, 1) 2017-10-11 17:05:01,078 | DEBUG | node.py (2608) | send | Node2 sending message COMMIT{'instId': 1, 'ppSeqNo': 1, 'viewNo': 0} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:05:01,079 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-10-11 17:05:01,079 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: BATCH{'messages': ['{"txnRootHash":"wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2","ppSeqNo":1,"digest":"6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897","op":"PREPARE","viewNo":0,"ppTime":1507741501,"stateRootHash":"24sLDAYWjNXuoxUhsmcVPxLg8NK9vaWZAW75AQGrCtfE","instId":0}', '{"op":"COMMIT","instId":1,"viewNo":0,"ppSeqNo":1}'], 'signature': None} 2017-10-11 17:05:01,079 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"txnRootHash":"wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2","ppSeqNo":1,"digest":"6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897","op":"PREPARE","viewNo":0,"ppTime":1507741501,"stateRootHash":"24sLDAYWjNXuoxUhsmcVPxLg8NK9vaWZAW75AQGrCtfE","instId":0}', '{"op":"COMMIT","instId":1,"viewNo":0,"ppSeqNo":1}'], 'signature': None, 'op': 'BATCH'}, 'Node4') 2017-10-11 17:05:01,080 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"txnRootHash":"wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2","ppSeqNo":1,"digest":"6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897","op":"PREPARE","viewNo":0,"ppTime":1507741501,"stateRootHash":"24sLDAYWjNXuoxUhsmcVPxLg8NK9vaWZAW75AQGrCtfE","instId":0}', '{"op":"COMMIT","instId":1,"viewNo":0,"ppSeqNo":1}'], 'signature': None} 2017-10-11 17:05:01,087 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'digest': '6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897', 'ppSeqNo': 1, 'stateRootHash': '24sLDAYWjNXuoxUhsmcVPxLg8NK9vaWZAW75AQGrCtfE', 'txnRootHash': 'wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2', 'instId': 0, 'ppTime': 1507741501, 'viewNo': 0} 2017-10-11 17:05:01,087 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'ppSeqNo': 1, 'digest': '6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897', 'stateRootHash': '24sLDAYWjNXuoxUhsmcVPxLg8NK9vaWZAW75AQGrCtfE', 'txnRootHash': 'wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2', 'op': 'PREPARE', 'instId': 0, 'ppTime': 1507741501, 'viewNo': 0}, 'Node4') 2017-10-11 17:05:01,087 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'digest': '6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897', 'ppSeqNo': 1, 'stateRootHash': '24sLDAYWjNXuoxUhsmcVPxLg8NK9vaWZAW75AQGrCtfE', 'txnRootHash': 'wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2', 'instId': 0, 'ppTime': 1507741501, 'viewNo': 0} 2017-10-11 17:05:01,087 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 1, 'ppSeqNo': 1, 'viewNo': 0} 2017-10-11 17:05:01,087 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'ppSeqNo': 1, 'op': 'COMMIT', 'viewNo': 0}, 'Node4') 2017-10-11 17:05:01,087 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'ppSeqNo': 1, 'viewNo': 0} 2017-10-11 17:05:01,087 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 0} 2017-10-11 17:05:01,088 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'ppSeqNo': 1, 'op': 'COMMIT', 'viewNo': 0}, 'Node4') 2017-10-11 17:05:01,088 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 0} 2017-10-11 17:05:01,088 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":1,"ppSeqNo":1,"op":"COMMIT","viewNo":0}' to Node1 2017-10-11 17:05:01,088 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":1,"ppSeqNo":1,"op":"COMMIT","viewNo":0}' to Node1 2017-10-11 17:05:01,089 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":1,"ppSeqNo":1,"op":"COMMIT","viewNo":0}' to Node3 2017-10-11 17:05:01,089 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":1,"ppSeqNo":1,"op":"COMMIT","viewNo":0}' to Node3 2017-10-11 17:05:01,089 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":1,"ppSeqNo":1,"op":"COMMIT","viewNo":0}' to Node4 2017-10-11 17:05:01,089 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":1,"ppSeqNo":1,"op":"COMMIT","viewNo":0}' to Node4 2017-10-11 17:05:01,090 | DEBUG | replica.py ( 953) | doCommit | Node2:0 Sending COMMIT(0, 1) at 2270586.548200602 2017-10-11 17:05:01,091 | DEBUG | replica.py (2193) | send | Node2:0 sending Commit 2017-10-11 17:05:01,091 | TRACE | replica.py (2194) | send | Node2:0 sending COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 0} 2017-10-11 17:05:01,091 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 0} commits where f is 1 2017-10-11 17:05:01,091 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (0, 1) 2017-10-11 17:05:01,091 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(0, 1) from Node4:0 2017-10-11 17:05:01,091 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 0} commits where f is 1 2017-10-11 17:05:01,091 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(0, 1) 2017-10-11 17:05:01,091 | DEBUG | replica.py ( 894) | processCommit | Node2:1 received COMMIT(0, 1) from Node4:1 2017-10-11 17:05:01,091 | DEBUG | replica.py ( 929) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'ppSeqNo': 1, 'viewNo': 0} commits where f is 1 2017-10-11 17:05:01,092 | DEBUG | replica.py ( 905) | processCommit | Node2:1 processed incoming COMMIT(0, 1) 2017-10-11 17:05:01,092 | DEBUG | node.py (2608) | send | Node2 sending message COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 0} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:05:01,093 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 3 messages through listener 2017-10-11 17:05:01,093 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: BATCH{'messages': ['{"ppSeqNo":1,"viewNo":0,"instId":0,"op":"PREPARE","stateRootHash":"24sLDAYWjNXuoxUhsmcVPxLg8NK9vaWZAW75AQGrCtfE","ppTime":1507741501,"digest":"6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897","txnRootHash":"wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2"}', '{"viewNo":0,"instId":1,"op":"COMMIT","ppSeqNo":1}'], 'signature': None} 2017-10-11 17:05:01,093 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"ppSeqNo":1,"viewNo":0,"instId":0,"op":"PREPARE","stateRootHash":"24sLDAYWjNXuoxUhsmcVPxLg8NK9vaWZAW75AQGrCtfE","ppTime":1507741501,"digest":"6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897","txnRootHash":"wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2"}', '{"viewNo":0,"instId":1,"op":"COMMIT","ppSeqNo":1}'], 'signature': None, 'op': 'BATCH'}, 'Node3') 2017-10-11 17:05:01,093 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"ppSeqNo":1,"viewNo":0,"instId":0,"op":"PREPARE","stateRootHash":"24sLDAYWjNXuoxUhsmcVPxLg8NK9vaWZAW75AQGrCtfE","ppTime":1507741501,"digest":"6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897","txnRootHash":"wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2"}', '{"viewNo":0,"instId":1,"op":"COMMIT","ppSeqNo":1}'], 'signature': None} 2017-10-11 17:05:01,094 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'digest': '6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897', 'ppSeqNo': 1, 'stateRootHash': '24sLDAYWjNXuoxUhsmcVPxLg8NK9vaWZAW75AQGrCtfE', 'txnRootHash': 'wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2', 'instId': 0, 'ppTime': 1507741501, 'viewNo': 0} 2017-10-11 17:05:01,094 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'digest': '6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897', 'ppSeqNo': 1, 'stateRootHash': '24sLDAYWjNXuoxUhsmcVPxLg8NK9vaWZAW75AQGrCtfE', 'op': 'PREPARE', 'txnRootHash': 'wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2', 'instId': 0, 'ppTime': 1507741501, 'viewNo': 0}, 'Node3') 2017-10-11 17:05:01,094 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'digest': '6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897', 'ppSeqNo': 1, 'stateRootHash': '24sLDAYWjNXuoxUhsmcVPxLg8NK9vaWZAW75AQGrCtfE', 'txnRootHash': 'wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2', 'instId': 0, 'ppTime': 1507741501, 'viewNo': 0} 2017-10-11 17:05:01,094 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 1, 'ppSeqNo': 1, 'viewNo': 0} 2017-10-11 17:05:01,094 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'ppSeqNo': 1, 'op': 'COMMIT', 'viewNo': 0}, 'Node3') 2017-10-11 17:05:01,094 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'ppSeqNo': 1, 'viewNo': 0} 2017-10-11 17:05:01,094 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 1, 'ppSeqNo': 1, 'viewNo': 0} 2017-10-11 17:05:01,094 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'ppSeqNo': 1, 'op': 'COMMIT', 'viewNo': 0}, 'Node1') 2017-10-11 17:05:01,096 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'ppSeqNo': 1, 'viewNo': 0} 2017-10-11 17:05:01,096 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 0} 2017-10-11 17:05:01,096 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'ppSeqNo': 1, 'op': 'COMMIT', 'viewNo': 0}, 'Node3') 2017-10-11 17:05:01,096 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 0} 2017-10-11 17:05:01,097 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":0,"ppSeqNo":1,"op":"COMMIT","viewNo":0}' to Node1 2017-10-11 17:05:01,097 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":0,"ppSeqNo":1,"op":"COMMIT","viewNo":0}' to Node1 2017-10-11 17:05:01,098 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":0,"ppSeqNo":1,"op":"COMMIT","viewNo":0}' to Node3 2017-10-11 17:05:01,098 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":0,"ppSeqNo":1,"op":"COMMIT","viewNo":0}' to Node3 2017-10-11 17:05:01,098 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":0,"ppSeqNo":1,"op":"COMMIT","viewNo":0}' to Node4 2017-10-11 17:05:01,098 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":0,"ppSeqNo":1,"op":"COMMIT","viewNo":0}' to Node4 2017-10-11 17:05:01,099 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'digest': '6ea68488f1f1b7bd5690b63f741538fc871a0fd01740d645a253b285d4d72897', 'ppSeqNo': 1, 'stateRootHash': '24sLDAYWjNXuoxUhsmcVPxLg8NK9vaWZAW75AQGrCtfE', 'txnRootHash': 'wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2', 'instId': 0, 'ppTime': 1507741501, 'viewNo': 0} 2017-10-11 17:05:01,099 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (0, 1) 2017-10-11 17:05:01,099 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(0, 1) from Node3:0 2017-10-11 17:05:01,099 | TRACE | replica.py ( 925) | tryOrder | Node2:0 returning request to node 2017-10-11 17:05:01,099 | DEBUG | replica.py (1444) | doOrder | Node2:0 ordering COMMIT (0, 1) 2017-10-11 17:05:01,099 | DEBUG | replica.py ( 323) | last_ordered_3pc | Node2:0 set last ordered as (0, 1) 2017-10-11 17:05:01,100 | DEBUG | replica.py (2193) | send | Node2:0 sending Ordered 2017-10-11 17:05:01,101 | TRACE | replica.py (2194) | send | Node2:0 sending ORDERED{'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741500981867)], 'ppSeqNo': 1, 'stateRootHash': '24sLDAYWjNXuoxUhsmcVPxLg8NK9vaWZAW75AQGrCtfE', 'txnRootHash': 'wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2', 'instId': 0, 'ppTime': 1507741501, 'viewNo': 0} 2017-10-11 17:05:01,101 | INFO | replica.py (1483) | order_3pc_key | Node2:0 ordered batch request, view no 0, ppSeqNo 1, ledger 1, state root 24sLDAYWjNXuoxUhsmcVPxLg8NK9vaWZAW75AQGrCtfE, txn root wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1507741500981867)], discarded [] 2017-10-11 17:05:01,101 | DEBUG | replica.py (1556) | _newCheckpointState | Node2:0 adding new checkpoint state for (1, 100) 2017-10-11 17:05:01,102 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(0, 1) 2017-10-11 17:05:01,104 | DEBUG | replica.py ( 894) | processCommit | Node2:1 received COMMIT(0, 1) from Node3:1 2017-10-11 17:05:01,105 | TRACE | replica.py ( 925) | tryOrder | Node2:1 returning request to node 2017-10-11 17:05:01,105 | DEBUG | replica.py (1444) | doOrder | Node2:1 ordering COMMIT (0, 1) 2017-10-11 17:05:01,106 | DEBUG | replica.py ( 323) | last_ordered_3pc | Node2:1 set last ordered as (0, 1) 2017-10-11 17:05:01,106 | DEBUG | replica.py (2193) | send | Node2:1 sending Ordered 2017-10-11 17:05:01,107 | TRACE | replica.py (2194) | send | Node2:1 sending ORDERED{'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741500981867)], 'ppSeqNo': 1, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741501, 'viewNo': 0} 2017-10-11 17:05:01,107 | INFO | replica.py (1483) | order_3pc_key | Node2:1 ordered batch request, view no 0, ppSeqNo 1, ledger 1, state root None, txn root None, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1507741500981867)], discarded [] 2017-10-11 17:05:01,107 | DEBUG | replica.py (1556) | _newCheckpointState | Node2:1 adding new checkpoint state for (1, 100) 2017-10-11 17:05:01,107 | DEBUG | replica.py ( 905) | processCommit | Node2:1 processed incoming COMMIT(0, 1) 2017-10-11 17:05:01,107 | TRACE | message_processor.py ( 29) | discard | Node2:1 discarding message COMMIT{'instId': 1, 'ppSeqNo': 1, 'viewNo': 0} because already ordered 3 phase message 2017-10-11 17:05:01,107 | DEBUG | node.py (1847) | processOrdered | Node2 executing Ordered batch 0 1 of 1 requests 2017-10-11 17:05:01,108 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-10-11 17:05:01,108 | DEBUG | node.py (2386) | sendReplyToClient | Node2 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1507741500981867) to client 2017-10-11 17:05:01,108 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"result":{"dest":"542MVr22zcHbVyGzaXmbT3","seqNo":12,"auditPath":["D9h1kJ9Egbq3d9BNRpAgZQmRvDTACpiyucoes3URbZWF","7LVA4w9Yeaprrpy8NTv4yitZ7hrpsXCeduCMbuh7B936","Gi2HwXVBFyDVgs7rwus6Mvr4PTuRgvyN3tR5izEkMLcE"],"type":"1","signature":"MfpjZjjhLbGuZwmjAMjFbKrZeitkMsLdmrXNK6yP62J3tDcbcvrtXp7dqqYhdZcvCw2DnwTt4FG5932Mg2Z1fnU","txnTime":1507741501,"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1507741500981867,"rootHash":"wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2"},"op":"REPLY"}' to b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' through listener socket 2017-10-11 17:05:01,109 | INFO | node.py (2290) | executeBatch | Node2 committed batch request, view no 0, ppSeqNo 1, ledger 1, state root 24sLDAYWjNXuoxUhsmcVPxLg8NK9vaWZAW75AQGrCtfE, txn root wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2, requests: [('V4SGRU86Z58d6TV7PBUe6f', 1507741500981867)] 2017-10-11 17:05:01,109 | DEBUG | node.py (2305) | executeBatch | Node2 storing 3PC key (0, 1) for ledger 1 range (12, 12) 2017-10-11 17:05:01,109 | TRACE | node.py (1858) | processOrdered | Node2 got ordered requests from backup replica 1 2017-10-11 17:05:01,110 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:05:01,110 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:05:01,111 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 0} 2017-10-11 17:05:01,112 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'ppSeqNo': 1, 'op': 'COMMIT', 'viewNo': 0}, 'Node1') 2017-10-11 17:05:01,112 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 0} 2017-10-11 17:05:01,112 | TRACE | message_processor.py ( 29) | discard | Node2:0 discarding message COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 0} because already ordered 3 phase message 2017-10-11 17:05:04,012 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2C got 1 messages through listener 2017-10-11 17:05:04,013 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1507741503995435 2017-10-11 17:05:04,013 | TRACE | node.py (1388) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT4'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ', 'reqId': 1507741503995435} 2017-10-11 17:05:04,013 | DEBUG | node.py (1434) | processClientInBox | Node2C processing b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' request SafeRequest: {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT4'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ', 'reqId': 1507741503995435} 2017-10-11 17:05:04,013 | DEBUG | node.py (1708) | processRequest | Node2 received client request: SafeRequest: {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT4'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ', 'reqId': 1507741503995435} from b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' 2017-10-11 17:05:04,013 | TRACE | propagator.py ( 165) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT4'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ', 'reqId': 1507741503995435} 2017-10-11 17:05:04,013 | INFO | propagator.py ( 148) | propagate | Node2 propagating request ('V4SGRU86Z58d6TV7PBUe6f', 1507741503995435) from client b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' 2017-10-11 17:05:04,014 | DEBUG | node.py (2608) | send | Node2 sending message PROPAGATE{'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%', 'request': {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT4'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ', 'reqId': 1507741503995435}} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:05:04,014 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT4'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ', 'reqId': 1507741503995435} to its replicas since not finalised 2017-10-11 17:05:04,014 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1507741503995435,"op":"REQACK"}' to b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' through listener socket 2017-10-11 17:05:04,020 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"senderClient":"fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%","request":{"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT4"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ","reqId":1507741503995435},"op":"PROPAGATE"}' to Node1 2017-10-11 17:05:04,020 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"senderClient":"fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%","request":{"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT4"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ","reqId":1507741503995435},"op":"PROPAGATE"}' to Node1 2017-10-11 17:05:04,020 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"senderClient":"fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%","request":{"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT4"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ","reqId":1507741503995435},"op":"PROPAGATE"}' to Node3 2017-10-11 17:05:04,020 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"senderClient":"fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%","request":{"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT4"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ","reqId":1507741503995435},"op":"PROPAGATE"}' to Node3 2017-10-11 17:05:04,021 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"senderClient":"fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%","request":{"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT4"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ","reqId":1507741503995435},"op":"PROPAGATE"}' to Node4 2017-10-11 17:05:04,021 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"senderClient":"fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%","request":{"operation":{"type":"1","dest":"542MVr22zcHbVyGzaXmbT4"},"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ","reqId":1507741503995435},"op":"PROPAGATE"}' to Node4 2017-10-11 17:05:04,039 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 3 messages through listener 2017-10-11 17:05:04,040 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1507741503995435 2017-10-11 17:05:04,040 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PROPAGATE{'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%', 'request': {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT4'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ', 'reqId': 1507741503995435}} 2017-10-11 17:05:04,040 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%', 'request': {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT4'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ', 'reqId': 1507741503995435}, 'op': 'PROPAGATE'}, 'Node3') 2017-10-11 17:05:04,040 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%', 'request': {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT4'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ', 'reqId': 1507741503995435}} 2017-10-11 17:05:04,041 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1507741503995435 2017-10-11 17:05:04,041 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PROPAGATE{'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%', 'request': {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT4'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ', 'reqId': 1507741503995435}} 2017-10-11 17:05:04,041 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%', 'request': {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT4'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ', 'reqId': 1507741503995435}, 'op': 'PROPAGATE'}, 'Node1') 2017-10-11 17:05:04,041 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%', 'request': {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT4'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ', 'reqId': 1507741503995435}} 2017-10-11 17:05:04,042 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1507741503995435 2017-10-11 17:05:04,042 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: PROPAGATE{'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%', 'request': {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT4'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ', 'reqId': 1507741503995435}} 2017-10-11 17:05:04,042 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%', 'request': {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT4'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ', 'reqId': 1507741503995435}, 'op': 'PROPAGATE'}, 'Node4') 2017-10-11 17:05:04,042 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%', 'request': {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT4'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ', 'reqId': 1507741503995435}} 2017-10-11 17:05:04,043 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%', 'request': {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT4'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ', 'reqId': 1507741503995435}} 2017-10-11 17:05:04,043 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT4'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ', 'reqId': 1507741503995435} 2017-10-11 17:05:04,043 | DEBUG | propagator.py ( 214) | forward | Node2 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1507741503995435) to 0 replicas 2017-10-11 17:05:04,043 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%', 'request': {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT4'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ', 'reqId': 1507741503995435}} 2017-10-11 17:05:04,043 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT4'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ', 'reqId': 1507741503995435} 2017-10-11 17:05:04,044 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT4'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ', 'reqId': 1507741503995435} to its replicas since already forwarded 2017-10-11 17:05:04,044 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': 'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%', 'request': {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT4'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ', 'reqId': 1507741503995435}} 2017-10-11 17:05:04,044 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT4'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ', 'reqId': 1507741503995435} 2017-10-11 17:05:04,044 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'type': '1', 'dest': '542MVr22zcHbVyGzaXmbT4'}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ', 'reqId': 1507741503995435} to its replicas since already forwarded 2017-10-11 17:05:04,045 | DEBUG | replica.py ( 667) | create3PCBatch | Node2:1 creating batch 2 for ledger 1 with state root None 2017-10-11 17:05:04,047 | DEBUG | replica.py ( 700) | create3PCBatch | Node2:1 created a PRE-PREPARE with 1 requests for ledger 1 2017-10-11 17:05:04,047 | DEBUG | replica.py (2193) | send | Node2:1 sending PrePrepare 2017-10-11 17:05:04,048 | TRACE | replica.py (2194) | send | Node2:1 sending PREPREPARE{'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741503995435)], 'ppSeqNo': 2, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741504, 'discarded': 1, 'digest': '79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23', 'viewNo': 0} 2017-10-11 17:05:04,048 | DEBUG | replica.py ( 614) | trackBatches | Node2:1 tracking batch for PREPREPARE{'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741503995435)], 'ppSeqNo': 2, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741504, 'discarded': 1, 'digest': '79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23', 'viewNo': 0} with state root None 2017-10-11 17:05:04,048 | DEBUG | node.py (2608) | send | Node2 sending message PREPREPARE{'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741503995435)], 'ppSeqNo': 2, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741504, 'discarded': 1, 'digest': '79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23', 'viewNo': 0} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:05:04,049 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":null,"digest":"79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23","discarded":1,"ledgerId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741503995435]],"ppSeqNo":2,"op":"PREPREPARE","instId":1,"ppTime":1507741504,"viewNo":0,"txnRootHash":null}' to Node1 2017-10-11 17:05:04,049 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":null,"digest":"79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23","discarded":1,"ledgerId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741503995435]],"ppSeqNo":2,"op":"PREPREPARE","instId":1,"ppTime":1507741504,"viewNo":0,"txnRootHash":null}' to Node1 2017-10-11 17:05:04,049 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":null,"digest":"79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23","discarded":1,"ledgerId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741503995435]],"ppSeqNo":2,"op":"PREPREPARE","instId":1,"ppTime":1507741504,"viewNo":0,"txnRootHash":null}' to Node3 2017-10-11 17:05:04,051 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":null,"digest":"79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23","discarded":1,"ledgerId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741503995435]],"ppSeqNo":2,"op":"PREPREPARE","instId":1,"ppTime":1507741504,"viewNo":0,"txnRootHash":null}' to Node3 2017-10-11 17:05:04,051 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":null,"digest":"79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23","discarded":1,"ledgerId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741503995435]],"ppSeqNo":2,"op":"PREPREPARE","instId":1,"ppTime":1507741504,"viewNo":0,"txnRootHash":null}' to Node4 2017-10-11 17:05:04,052 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":null,"digest":"79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23","discarded":1,"ledgerId":1,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741503995435]],"ppSeqNo":2,"op":"PREPREPARE","instId":1,"ppTime":1507741504,"viewNo":0,"txnRootHash":null}' to Node4 2017-10-11 17:05:04,071 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 4 messages through listener 2017-10-11 17:05:04,071 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PREPREPARE{'ledgerId': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1507741503995435]], 'ppSeqNo': 2, 'stateRootHash': 'CZrJjTQqN9QFaEpGiGRpkxiZG3FA43MbDKuVxiqvhDeY', 'txnRootHash': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'instId': 0, 'ppTime': 1507741504, 'discarded': 1, 'digest': '79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23', 'viewNo': 0} 2017-10-11 17:05:04,071 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'ledgerId': 1, 'digest': '79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23', 'ppSeqNo': 2, 'stateRootHash': 'CZrJjTQqN9QFaEpGiGRpkxiZG3FA43MbDKuVxiqvhDeY', 'op': 'PREPREPARE', 'txnRootHash': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'instId': 0, 'ppTime': 1507741504, 'discarded': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1507741503995435]], 'viewNo': 0}, 'Node1') 2017-10-11 17:05:04,071 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPREPARE{'ledgerId': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1507741503995435]], 'ppSeqNo': 2, 'stateRootHash': 'CZrJjTQqN9QFaEpGiGRpkxiZG3FA43MbDKuVxiqvhDeY', 'txnRootHash': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'instId': 0, 'ppTime': 1507741504, 'discarded': 1, 'digest': '79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23', 'viewNo': 0} 2017-10-11 17:05:04,072 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'digest': '79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23', 'ppSeqNo': 2, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741504, 'viewNo': 0} 2017-10-11 17:05:04,072 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'digest': '79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23', 'ppSeqNo': 2, 'stateRootHash': None, 'op': 'PREPARE', 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741504, 'viewNo': 0}, 'Node3') 2017-10-11 17:05:04,072 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'digest': '79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23', 'ppSeqNo': 2, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741504, 'viewNo': 0} 2017-10-11 17:05:04,072 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'digest': '79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23', 'ppSeqNo': 2, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741504, 'viewNo': 0} 2017-10-11 17:05:04,072 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'ppSeqNo': 2, 'digest': '79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23', 'stateRootHash': None, 'txnRootHash': None, 'op': 'PREPARE', 'instId': 1, 'ppTime': 1507741504, 'viewNo': 0}, 'Node4') 2017-10-11 17:05:04,072 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'digest': '79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23', 'ppSeqNo': 2, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741504, 'viewNo': 0} 2017-10-11 17:05:04,072 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PREPARE{'digest': '79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23', 'ppSeqNo': 2, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741504, 'viewNo': 0} 2017-10-11 17:05:04,073 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'digest': '79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23', 'ppSeqNo': 2, 'stateRootHash': None, 'op': 'PREPARE', 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741504, 'viewNo': 0}, 'Node1') 2017-10-11 17:05:04,073 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'digest': '79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23', 'ppSeqNo': 2, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741504, 'viewNo': 0} 2017-10-11 17:05:04,080 | DEBUG | replica.py ( 823) | processPrePrepare | Node2:0 received PRE-PREPARE(0, 2) from Node1:0 at 2270589.537984472 2017-10-11 17:05:04,081 | DEBUG | replica.py (1034) | validate_pre_prepare | Node2:0 state root before processing PREPREPARE{'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741503995435)], 'ppSeqNo': 2, 'stateRootHash': 'CZrJjTQqN9QFaEpGiGRpkxiZG3FA43MbDKuVxiqvhDeY', 'txnRootHash': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'instId': 0, 'ppTime': 1507741504, 'discarded': 1, 'digest': '79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23', 'viewNo': 0} is b'\x0f\xd9V\xca_4\x81I\x98\r\xf0\xc7F%\x1a\xb2PM\xd92*O\xe6\xa5o\xc6!h\r\xb7U=', wLJkMsXcnqqii7GSsqKxSB4SC79yzrLi5Rx6dBv2ch2 2017-10-11 17:05:04,082 | DEBUG | replica.py ( 934) | doPrepare | Node2:0 Sending PREPARE(0, 2) at 2270589.540020744 2017-10-11 17:05:04,082 | DEBUG | replica.py (2193) | send | Node2:0 sending Prepare 2017-10-11 17:05:04,083 | TRACE | replica.py (2194) | send | Node2:0 sending PREPARE{'digest': '79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23', 'ppSeqNo': 2, 'stateRootHash': 'CZrJjTQqN9QFaEpGiGRpkxiZG3FA43MbDKuVxiqvhDeY', 'txnRootHash': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'instId': 0, 'ppTime': 1507741504, 'viewNo': 0} 2017-10-11 17:05:04,083 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': '79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23', 'ppSeqNo': 2, 'stateRootHash': 'CZrJjTQqN9QFaEpGiGRpkxiZG3FA43MbDKuVxiqvhDeY', 'txnRootHash': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'instId': 0, 'ppTime': 1507741504, 'viewNo': 0} 2017-10-11 17:05:04,083 | DEBUG | replica.py ( 614) | trackBatches | Node2:0 tracking batch for PREPREPARE{'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741503995435)], 'ppSeqNo': 2, 'stateRootHash': 'CZrJjTQqN9QFaEpGiGRpkxiZG3FA43MbDKuVxiqvhDeY', 'txnRootHash': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'instId': 0, 'ppTime': 1507741504, 'discarded': 1, 'digest': '79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23', 'viewNo': 0} with state root b'\x0f\xd9V\xca_4\x81I\x98\r\xf0\xc7F%\x1a\xb2PM\xd92*O\xe6\xa5o\xc6!h\r\xb7U=' 2017-10-11 17:05:04,083 | DEBUG | replica.py ( 842) | processPrePrepare | Node2:0 processed incoming PRE-PREPARE(0, 2) 2017-10-11 17:05:04,083 | DEBUG | replica.py ( 917) | tryCommit | Node2:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': '79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23', 'ppSeqNo': 2, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741504, 'viewNo': 0} 2017-10-11 17:05:04,083 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (0, 2) 2017-10-11 17:05:04,084 | DEBUG | replica.py ( 953) | doCommit | Node2:1 Sending COMMIT(0, 2) at 2270589.54139956 2017-10-11 17:05:04,084 | DEBUG | replica.py (2193) | send | Node2:1 sending Commit 2017-10-11 17:05:04,084 | TRACE | replica.py (2194) | send | Node2:1 sending COMMIT{'instId': 1, 'ppSeqNo': 2, 'viewNo': 0} 2017-10-11 17:05:04,084 | DEBUG | replica.py ( 929) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'ppSeqNo': 2, 'viewNo': 0} commits where f is 1 2017-10-11 17:05:04,084 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (0, 2) 2017-10-11 17:05:04,084 | DEBUG | replica.py ( 917) | tryCommit | Node2:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'digest': '79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23', 'ppSeqNo': 2, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741504, 'viewNo': 0} 2017-10-11 17:05:04,084 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (0, 2) 2017-10-11 17:05:04,084 | DEBUG | node.py (2608) | send | Node2 sending message PREPARE{'digest': '79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23', 'ppSeqNo': 2, 'stateRootHash': 'CZrJjTQqN9QFaEpGiGRpkxiZG3FA43MbDKuVxiqvhDeY', 'txnRootHash': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'instId': 0, 'ppTime': 1507741504, 'viewNo': 0} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:05:04,085 | DEBUG | node.py (2608) | send | Node2 sending message COMMIT{'instId': 1, 'ppSeqNo': 2, 'viewNo': 0} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:05:04,085 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-10-11 17:05:04,092 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 1, 'ppSeqNo': 2, 'viewNo': 0} 2017-10-11 17:05:04,093 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'ppSeqNo': 2, 'op': 'COMMIT', 'viewNo': 0}, 'Node1') 2017-10-11 17:05:04,093 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'ppSeqNo': 2, 'viewNo': 0} 2017-10-11 17:05:04,093 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'digest': '79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23', 'ppSeqNo': 2, 'stateRootHash': 'CZrJjTQqN9QFaEpGiGRpkxiZG3FA43MbDKuVxiqvhDeY', 'txnRootHash': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'instId': 0, 'ppTime': 1507741504, 'viewNo': 0} 2017-10-11 17:05:04,093 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'digest': '79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23', 'ppSeqNo': 2, 'stateRootHash': 'CZrJjTQqN9QFaEpGiGRpkxiZG3FA43MbDKuVxiqvhDeY', 'op': 'PREPARE', 'txnRootHash': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'instId': 0, 'ppTime': 1507741504, 'viewNo': 0}, 'Node3') 2017-10-11 17:05:04,093 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'digest': '79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23', 'ppSeqNo': 2, 'stateRootHash': 'CZrJjTQqN9QFaEpGiGRpkxiZG3FA43MbDKuVxiqvhDeY', 'txnRootHash': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'instId': 0, 'ppTime': 1507741504, 'viewNo': 0} 2017-10-11 17:05:04,094 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-10-11 17:05:04,094 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"stateRootHash":"CZrJjTQqN9QFaEpGiGRpkxiZG3FA43MbDKuVxiqvhDeY","viewNo":0,"digest":"79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23","ppSeqNo":2,"op":"PREPARE","instId":0,"ppTime":1507741504,"txnRootHash":"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn"}', b'{"instId":1,"ppSeqNo":2,"op":"COMMIT","viewNo":0}']) 2017-10-11 17:05:04,098 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node1: b'{"messages":["{\\"stateRootHash\\":\\"CZrJjTQqN9QFaEpGiGRpkxiZG3FA43MbDKuVxiqvhDeY\\",\\"viewNo\\":0,\\"digest\\":\\"79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23\\",\\"ppSeqNo\\":2,\\"op\\":\\"PREPARE\\",\\"instId\\":0,\\"ppTime\\":1507741504,\\"txnRootHash\\":\\"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn\\"}","{\\"instId\\":1,\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\",\\"viewNo\\":0}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:05:04,098 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"stateRootHash\\":\\"CZrJjTQqN9QFaEpGiGRpkxiZG3FA43MbDKuVxiqvhDeY\\",\\"viewNo\\":0,\\"digest\\":\\"79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23\\",\\"ppSeqNo\\":2,\\"op\\":\\"PREPARE\\",\\"instId\\":0,\\"ppTime\\":1507741504,\\"txnRootHash\\":\\"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn\\"}","{\\"instId\\":1,\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\",\\"viewNo\\":0}"],"signature":null,"op":"BATCH"}' to Node1 2017-10-11 17:05:04,098 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-10-11 17:05:04,098 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"stateRootHash":"CZrJjTQqN9QFaEpGiGRpkxiZG3FA43MbDKuVxiqvhDeY","viewNo":0,"digest":"79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23","ppSeqNo":2,"op":"PREPARE","instId":0,"ppTime":1507741504,"txnRootHash":"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn"}', b'{"instId":1,"ppSeqNo":2,"op":"COMMIT","viewNo":0}']) 2017-10-11 17:05:04,098 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node3: b'{"messages":["{\\"stateRootHash\\":\\"CZrJjTQqN9QFaEpGiGRpkxiZG3FA43MbDKuVxiqvhDeY\\",\\"viewNo\\":0,\\"digest\\":\\"79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23\\",\\"ppSeqNo\\":2,\\"op\\":\\"PREPARE\\",\\"instId\\":0,\\"ppTime\\":1507741504,\\"txnRootHash\\":\\"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn\\"}","{\\"instId\\":1,\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\",\\"viewNo\\":0}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:05:04,099 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"stateRootHash\\":\\"CZrJjTQqN9QFaEpGiGRpkxiZG3FA43MbDKuVxiqvhDeY\\",\\"viewNo\\":0,\\"digest\\":\\"79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23\\",\\"ppSeqNo\\":2,\\"op\\":\\"PREPARE\\",\\"instId\\":0,\\"ppTime\\":1507741504,\\"txnRootHash\\":\\"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn\\"}","{\\"instId\\":1,\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\",\\"viewNo\\":0}"],"signature":null,"op":"BATCH"}' to Node3 2017-10-11 17:05:04,099 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node4 into one transmission 2017-10-11 17:05:04,099 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"stateRootHash":"CZrJjTQqN9QFaEpGiGRpkxiZG3FA43MbDKuVxiqvhDeY","viewNo":0,"digest":"79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23","ppSeqNo":2,"op":"PREPARE","instId":0,"ppTime":1507741504,"txnRootHash":"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn"}', b'{"instId":1,"ppSeqNo":2,"op":"COMMIT","viewNo":0}']) 2017-10-11 17:05:04,099 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node4: b'{"messages":["{\\"stateRootHash\\":\\"CZrJjTQqN9QFaEpGiGRpkxiZG3FA43MbDKuVxiqvhDeY\\",\\"viewNo\\":0,\\"digest\\":\\"79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23\\",\\"ppSeqNo\\":2,\\"op\\":\\"PREPARE\\",\\"instId\\":0,\\"ppTime\\":1507741504,\\"txnRootHash\\":\\"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn\\"}","{\\"instId\\":1,\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\",\\"viewNo\\":0}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:05:04,099 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"stateRootHash\\":\\"CZrJjTQqN9QFaEpGiGRpkxiZG3FA43MbDKuVxiqvhDeY\\",\\"viewNo\\":0,\\"digest\\":\\"79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23\\",\\"ppSeqNo\\":2,\\"op\\":\\"PREPARE\\",\\"instId\\":0,\\"ppTime\\":1507741504,\\"txnRootHash\\":\\"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn\\"}","{\\"instId\\":1,\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\",\\"viewNo\\":0}"],"signature":null,"op":"BATCH"}' to Node4 2017-10-11 17:05:04,104 | DEBUG | replica.py ( 953) | doCommit | Node2:0 Sending COMMIT(0, 2) at 2270589.558089849 2017-10-11 17:05:04,105 | DEBUG | replica.py (2193) | send | Node2:0 sending Commit 2017-10-11 17:05:04,105 | TRACE | replica.py (2194) | send | Node2:0 sending COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 0} 2017-10-11 17:05:04,105 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 0} commits where f is 1 2017-10-11 17:05:04,105 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (0, 2) 2017-10-11 17:05:04,105 | DEBUG | replica.py ( 894) | processCommit | Node2:1 received COMMIT(0, 2) from Node1:1 2017-10-11 17:05:04,106 | DEBUG | replica.py ( 929) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'ppSeqNo': 2, 'viewNo': 0} commits where f is 1 2017-10-11 17:05:04,106 | DEBUG | replica.py ( 905) | processCommit | Node2:1 processed incoming COMMIT(0, 2) 2017-10-11 17:05:04,106 | DEBUG | node.py (2608) | send | Node2 sending message COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 0} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:05:04,106 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 4 messages through listener 2017-10-11 17:05:04,107 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 1, 'ppSeqNo': 2, 'viewNo': 0} 2017-10-11 17:05:04,107 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'ppSeqNo': 2, 'op': 'COMMIT', 'viewNo': 0}, 'Node3') 2017-10-11 17:05:04,107 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'ppSeqNo': 2, 'viewNo': 0} 2017-10-11 17:05:04,112 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: BATCH{'messages': ['{"txnRootHash":"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn","ppSeqNo":2,"digest":"79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23","op":"PREPARE","viewNo":0,"ppTime":1507741504,"stateRootHash":"CZrJjTQqN9QFaEpGiGRpkxiZG3FA43MbDKuVxiqvhDeY","instId":0}', '{"op":"COMMIT","instId":1,"viewNo":0,"ppSeqNo":2}'], 'signature': None} 2017-10-11 17:05:04,112 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"txnRootHash":"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn","ppSeqNo":2,"digest":"79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23","op":"PREPARE","viewNo":0,"ppTime":1507741504,"stateRootHash":"CZrJjTQqN9QFaEpGiGRpkxiZG3FA43MbDKuVxiqvhDeY","instId":0}', '{"op":"COMMIT","instId":1,"viewNo":0,"ppSeqNo":2}'], 'signature': None, 'op': 'BATCH'}, 'Node4') 2017-10-11 17:05:04,112 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"txnRootHash":"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn","ppSeqNo":2,"digest":"79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23","op":"PREPARE","viewNo":0,"ppTime":1507741504,"stateRootHash":"CZrJjTQqN9QFaEpGiGRpkxiZG3FA43MbDKuVxiqvhDeY","instId":0}', '{"op":"COMMIT","instId":1,"viewNo":0,"ppSeqNo":2}'], 'signature': None} 2017-10-11 17:05:04,113 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'digest': '79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23', 'ppSeqNo': 2, 'stateRootHash': 'CZrJjTQqN9QFaEpGiGRpkxiZG3FA43MbDKuVxiqvhDeY', 'txnRootHash': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'instId': 0, 'ppTime': 1507741504, 'viewNo': 0} 2017-10-11 17:05:04,113 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'ppSeqNo': 2, 'digest': '79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23', 'stateRootHash': 'CZrJjTQqN9QFaEpGiGRpkxiZG3FA43MbDKuVxiqvhDeY', 'txnRootHash': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'op': 'PREPARE', 'instId': 0, 'ppTime': 1507741504, 'viewNo': 0}, 'Node4') 2017-10-11 17:05:04,113 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'digest': '79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23', 'ppSeqNo': 2, 'stateRootHash': 'CZrJjTQqN9QFaEpGiGRpkxiZG3FA43MbDKuVxiqvhDeY', 'txnRootHash': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'instId': 0, 'ppTime': 1507741504, 'viewNo': 0} 2017-10-11 17:05:04,113 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 1, 'ppSeqNo': 2, 'viewNo': 0} 2017-10-11 17:05:04,113 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'ppSeqNo': 2, 'op': 'COMMIT', 'viewNo': 0}, 'Node4') 2017-10-11 17:05:04,113 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'ppSeqNo': 2, 'viewNo': 0} 2017-10-11 17:05:04,114 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 0} 2017-10-11 17:05:04,114 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'ppSeqNo': 2, 'op': 'COMMIT', 'viewNo': 0}, 'Node3') 2017-10-11 17:05:04,114 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 0} 2017-10-11 17:05:04,114 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 0} 2017-10-11 17:05:04,114 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'ppSeqNo': 2, 'op': 'COMMIT', 'viewNo': 0}, 'Node4') 2017-10-11 17:05:04,114 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 0} 2017-10-11 17:05:04,115 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":0,"ppSeqNo":2,"op":"COMMIT","viewNo":0}' to Node1 2017-10-11 17:05:04,116 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":0,"ppSeqNo":2,"op":"COMMIT","viewNo":0}' to Node1 2017-10-11 17:05:04,116 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":0,"ppSeqNo":2,"op":"COMMIT","viewNo":0}' to Node3 2017-10-11 17:05:04,116 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":0,"ppSeqNo":2,"op":"COMMIT","viewNo":0}' to Node3 2017-10-11 17:05:04,116 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":0,"ppSeqNo":2,"op":"COMMIT","viewNo":0}' to Node4 2017-10-11 17:05:04,117 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":0,"ppSeqNo":2,"op":"COMMIT","viewNo":0}' to Node4 2017-10-11 17:05:04,118 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'digest': '79dcf7fb12e16a9769b789d9cd302b8143ea89ba185e26c1cac9cc0be5558b23', 'ppSeqNo': 2, 'stateRootHash': 'CZrJjTQqN9QFaEpGiGRpkxiZG3FA43MbDKuVxiqvhDeY', 'txnRootHash': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'instId': 0, 'ppTime': 1507741504, 'viewNo': 0} 2017-10-11 17:05:04,118 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (0, 2) 2017-10-11 17:05:04,118 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(0, 2) from Node3:0 2017-10-11 17:05:04,118 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 0} commits where f is 1 2017-10-11 17:05:04,118 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(0, 2) 2017-10-11 17:05:04,118 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(0, 2) from Node4:0 2017-10-11 17:05:04,118 | TRACE | replica.py ( 925) | tryOrder | Node2:0 returning request to node 2017-10-11 17:05:04,118 | DEBUG | replica.py (1444) | doOrder | Node2:0 ordering COMMIT (0, 2) 2017-10-11 17:05:04,118 | DEBUG | replica.py ( 323) | last_ordered_3pc | Node2:0 set last ordered as (0, 2) 2017-10-11 17:05:04,119 | DEBUG | replica.py (2193) | send | Node2:0 sending Ordered 2017-10-11 17:05:04,119 | TRACE | replica.py (2194) | send | Node2:0 sending ORDERED{'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741503995435)], 'ppSeqNo': 2, 'stateRootHash': 'CZrJjTQqN9QFaEpGiGRpkxiZG3FA43MbDKuVxiqvhDeY', 'txnRootHash': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'instId': 0, 'ppTime': 1507741504, 'viewNo': 0} 2017-10-11 17:05:04,119 | INFO | replica.py (1483) | order_3pc_key | Node2:0 ordered batch request, view no 0, ppSeqNo 2, ledger 1, state root CZrJjTQqN9QFaEpGiGRpkxiZG3FA43MbDKuVxiqvhDeY, txn root DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1507741503995435)], discarded [] 2017-10-11 17:05:04,119 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(0, 2) 2017-10-11 17:05:04,119 | DEBUG | replica.py ( 894) | processCommit | Node2:1 received COMMIT(0, 2) from Node3:1 2017-10-11 17:05:04,120 | TRACE | replica.py ( 925) | tryOrder | Node2:1 returning request to node 2017-10-11 17:05:04,120 | DEBUG | replica.py (1444) | doOrder | Node2:1 ordering COMMIT (0, 2) 2017-10-11 17:05:04,120 | DEBUG | replica.py ( 323) | last_ordered_3pc | Node2:1 set last ordered as (0, 2) 2017-10-11 17:05:04,120 | DEBUG | replica.py (2193) | send | Node2:1 sending Ordered 2017-10-11 17:05:04,120 | TRACE | replica.py (2194) | send | Node2:1 sending ORDERED{'ledgerId': 1, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741503995435)], 'ppSeqNo': 2, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741504, 'viewNo': 0} 2017-10-11 17:05:04,120 | INFO | replica.py (1483) | order_3pc_key | Node2:1 ordered batch request, view no 0, ppSeqNo 2, ledger 1, state root None, txn root None, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1507741503995435)], discarded [] 2017-10-11 17:05:04,120 | DEBUG | replica.py ( 905) | processCommit | Node2:1 processed incoming COMMIT(0, 2) 2017-10-11 17:05:04,121 | TRACE | message_processor.py ( 29) | discard | Node2:1 discarding message COMMIT{'instId': 1, 'ppSeqNo': 2, 'viewNo': 0} because already ordered 3 phase message 2017-10-11 17:05:04,121 | DEBUG | node.py (1847) | processOrdered | Node2 executing Ordered batch 0 2 of 1 requests 2017-10-11 17:05:04,121 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-10-11 17:05:04,123 | DEBUG | node.py (2386) | sendReplyToClient | Node2 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1507741503995435) to client 2017-10-11 17:05:04,123 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"result":{"dest":"542MVr22zcHbVyGzaXmbT4","seqNo":13,"auditPath":["CFFMfapJTbk3rWENJa7Brjm5cVQTXaJE3P6UAGdyyZwC","Gi2HwXVBFyDVgs7rwus6Mvr4PTuRgvyN3tR5izEkMLcE"],"type":"1","signature":"4qkeA5A8cXU3cTWFSuYeqTLsK2D6Lqnjt7HxJAkkHLrV8jxsZbybuZBgTf6BMwSL7KCnjwduyXgz6ugSj3LZDqmQ","txnTime":1507741504,"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1507741503995435,"rootHash":"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn"},"op":"REPLY"}' to b'fUW0HqcUbaEUwY&4ITjkWTgST8Ypps&uh46af]S%' through listener socket 2017-10-11 17:05:04,123 | INFO | node.py (2290) | executeBatch | Node2 committed batch request, view no 0, ppSeqNo 2, ledger 1, state root CZrJjTQqN9QFaEpGiGRpkxiZG3FA43MbDKuVxiqvhDeY, txn root DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn, requests: [('V4SGRU86Z58d6TV7PBUe6f', 1507741503995435)] 2017-10-11 17:05:04,124 | DEBUG | node.py (2305) | executeBatch | Node2 storing 3PC key (0, 2) for ledger 1 range (13, 13) 2017-10-11 17:05:04,124 | TRACE | node.py (1858) | processOrdered | Node2 got ordered requests from backup replica 1 2017-10-11 17:05:04,124 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:05:04,137 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:05:04,138 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 0} 2017-10-11 17:05:04,138 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'ppSeqNo': 2, 'op': 'COMMIT', 'viewNo': 0}, 'Node1') 2017-10-11 17:05:04,138 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 0} 2017-10-11 17:05:04,139 | TRACE | message_processor.py ( 29) | discard | Node2:0 discarding message COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 0} because already ordered 3 phase message 2017-10-11 17:05:06,795 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 1 2017-10-11 17:05:06,796 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:05:06,796 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 2 to run in 60 seconds 2017-10-11 17:05:06,808 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 2 2017-10-11 17:05:06,809 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node2 checking its request amount 2017-10-11 17:05:06,809 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:05:06,809 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 11 to run in 60 seconds 2017-10-11 17:05:06,850 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 10 2017-10-11 17:05:06,850 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:05:06,850 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:05:06,850 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0016907315409305 is acceptable. 2017-10-11 17:05:06,851 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:05:06,851 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:05:06,851 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:05:06,852 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 12 to run in 10 seconds 2017-10-11 17:05:06,893 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action dump_json_file with id 5 2017-10-11 17:05:06,894 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action dump_json_file with id 13 to run in 60 seconds 2017-10-11 17:05:06,895 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action propose_view_change with id 3 2017-10-11 17:05:06,895 | TRACE | node.py (2097) | propose_view_change | The primary is already connected so view change will not be proposed 2017-10-11 17:05:06,903 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:05:06,903 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:05:06,903 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:05:06,904 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:05:08,786 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:05:08,788 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} 2017-10-11 17:05:08,788 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'reason': 26, 'op': 'INSTANCE_CHANGE', 'viewNo': 1}, 'Node1') 2017-10-11 17:05:08,788 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} 2017-10-11 17:05:08,788 | DEBUG | node.py (1915) | processInstanceChange | Node2 received instance change request: INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} from Node1 2017-10-11 17:05:08,788 | DEBUG | node.py (1958) | do_view_change_if_possible | Node2 has no quorum for view 1 2017-10-11 17:05:08,788 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0016907315409305 is acceptable. 2017-10-11 17:05:08,788 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:05:08,789 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:05:08,789 | DEBUG | node.py (1946) | processInstanceChange | Node2 received instance change message INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} but did not find the master to be slow or has already sent an instance change message 2017-10-11 17:05:16,856 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 12 2017-10-11 17:05:16,857 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:05:16,857 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:05:16,857 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0016907315409305 is acceptable. 2017-10-11 17:05:16,857 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:05:16,857 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:05:16,857 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:05:16,858 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 14 to run in 10 seconds 2017-10-11 17:05:21,913 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:05:21,913 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:05:21,913 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:05:21,913 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:05:26,864 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 14 2017-10-11 17:05:26,864 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:05:26,864 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:05:26,864 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0016907315409305 is acceptable. 2017-10-11 17:05:26,864 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:05:26,865 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:05:26,865 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:05:26,865 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 15 to run in 10 seconds 2017-10-11 17:05:36,877 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 15 2017-10-11 17:05:36,877 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:05:36,877 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:05:36,877 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0016907315409305 is acceptable. 2017-10-11 17:05:36,877 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:05:36,878 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:05:36,878 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:05:36,878 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 16 to run in 10 seconds 2017-10-11 17:05:36,914 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:05:36,914 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:05:36,914 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:05:36,915 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:05:46,886 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 16 2017-10-11 17:05:46,886 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:05:46,886 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:05:46,886 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0016907315409305 is acceptable. 2017-10-11 17:05:46,886 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:05:46,887 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:05:46,887 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:05:46,887 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 17 to run in 10 seconds 2017-10-11 17:05:51,918 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:05:51,919 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:05:51,919 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:05:51,919 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:05:56,893 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 17 2017-10-11 17:05:56,893 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:05:56,893 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:05:56,893 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0016907315409305 is acceptable. 2017-10-11 17:05:56,894 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:05:56,894 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:05:56,894 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:05:56,894 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 18 to run in 10 seconds 2017-10-11 17:06:06,798 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 2 2017-10-11 17:06:06,798 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:06:06,799 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 3 to run in 60 seconds 2017-10-11 17:06:06,817 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 11 2017-10-11 17:06:06,817 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node2 checking its request amount 2017-10-11 17:06:06,817 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:06:06,818 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 19 to run in 60 seconds 2017-10-11 17:06:06,902 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 18 2017-10-11 17:06:06,903 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:06:06,903 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:06:06,903 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0016907315409305 is acceptable. 2017-10-11 17:06:06,903 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:06:06,903 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:06:06,903 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:06:06,903 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 20 to run in 10 seconds 2017-10-11 17:06:06,903 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action dump_json_file with id 13 2017-10-11 17:06:06,904 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action dump_json_file with id 21 to run in 60 seconds 2017-10-11 17:06:06,921 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:06:06,921 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:06:06,921 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:06:06,922 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:06:16,911 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 20 2017-10-11 17:06:16,911 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:06:16,911 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:06:16,911 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0016907315409305 is acceptable. 2017-10-11 17:06:16,911 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:06:16,911 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:06:16,911 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:06:16,911 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 22 to run in 10 seconds 2017-10-11 17:06:21,923 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:06:21,923 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:06:21,924 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:06:21,924 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:06:26,914 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 22 2017-10-11 17:06:26,915 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:06:26,915 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:06:26,915 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0016907315409305 is acceptable. 2017-10-11 17:06:26,915 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:06:26,915 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:06:26,916 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:06:26,916 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 23 to run in 10 seconds 2017-10-11 17:06:32,784 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2C got 1 messages through listener 2017-10-11 17:06:32,785 | DEBUG | zstack.py ( 663) | handlePingPong | Node2C got ping from b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' 2017-10-11 17:06:32,785 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'po' to b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' through listener socket 2017-10-11 17:06:32,785 | DEBUG | zstack.py ( 645) | sendPingPong | Node2C ponged b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' 2017-10-11 17:06:32,830 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2C got 1 messages through listener 2017-10-11 17:06:32,830 | TRACE | node.py (1388) | validateClientMsg | Node2C received CLIENT message: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4} 2017-10-11 17:06:32,831 | DEBUG | node.py (1434) | processClientInBox | Node2C processing b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' request LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4} 2017-10-11 17:06:32,831 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4} from b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' 2017-10-11 17:06:32,831 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 17:06:32,831 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ledgerId":0,"ppSeqNo":null,"op":"LEDGER_STATUS","txnSeqNo":4,"viewNo":null}' to b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' through listener socket 2017-10-11 17:06:32,831 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 17:06:32,832 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 17:06:36,921 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 23 2017-10-11 17:06:36,921 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:06:36,921 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:06:36,921 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0016907315409305 is acceptable. 2017-10-11 17:06:36,922 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:06:36,922 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:06:36,922 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:06:36,922 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 24 to run in 10 seconds 2017-10-11 17:06:36,936 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:06:36,936 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:06:36,936 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:06:36,937 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:06:46,925 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 24 2017-10-11 17:06:46,925 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:06:46,926 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:06:46,926 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0016907315409305 is acceptable. 2017-10-11 17:06:46,926 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:06:46,927 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:06:46,927 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:06:46,927 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 25 to run in 10 seconds 2017-10-11 17:06:47,585 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2C got 1 messages through listener 2017-10-11 17:06:47,591 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on request 1507741607555952 2017-10-11 17:06:47,591 | TRACE | node.py (1388) | validateClientMsg | Node2C received CLIENT message: SafeRequest: {'operation': {'action': 'start', 'name': 'upgrade40', 'reinstall': False, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'type': '109', 'version': '1.1.40', 'force': False, 'justification': None, 'timeout': 10, 'schedule': {'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T17:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T17:10:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T17:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T17:25:00.258870+00:00'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i', 'reqId': 1507741607555952} 2017-10-11 17:06:47,591 | DEBUG | node.py (1434) | processClientInBox | Node2C processing b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' request SafeRequest: {'operation': {'action': 'start', 'name': 'upgrade40', 'reinstall': False, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'type': '109', 'version': '1.1.40', 'force': False, 'justification': None, 'timeout': 10, 'schedule': {'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T17:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T17:10:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T17:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T17:25:00.258870+00:00'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i', 'reqId': 1507741607555952} 2017-10-11 17:06:47,591 | DEBUG | node.py (1708) | processRequest | Node2 received client request: SafeRequest: {'operation': {'action': 'start', 'name': 'upgrade40', 'reinstall': False, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'type': '109', 'version': '1.1.40', 'force': False, 'justification': None, 'timeout': 10, 'schedule': {'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T17:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T17:10:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T17:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T17:25:00.258870+00:00'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i', 'reqId': 1507741607555952} from b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' 2017-10-11 17:06:47,592 | TRACE | propagator.py ( 165) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'action': 'start', 'name': 'upgrade40', 'reinstall': False, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'type': '109', 'version': '1.1.40', 'force': False, 'justification': None, 'timeout': 10, 'schedule': {'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T17:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T17:10:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T17:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T17:25:00.258870+00:00'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i', 'reqId': 1507741607555952} 2017-10-11 17:06:47,592 | INFO | propagator.py ( 148) | propagate | Node2 propagating request ('V4SGRU86Z58d6TV7PBUe6f', 1507741607555952) from client b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' 2017-10-11 17:06:47,592 | DEBUG | node.py (2608) | send | Node2 sending message PROPAGATE{'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a', 'request': {'operation': {'action': 'start', 'name': 'upgrade40', 'reinstall': False, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'type': '109', 'version': '1.1.40', 'force': False, 'justification': None, 'timeout': 10, 'schedule': {'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T17:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T17:10:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T17:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T17:25:00.258870+00:00'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i', 'reqId': 1507741607555952}} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:06:47,592 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'action': 'start', 'name': 'upgrade40', 'reinstall': False, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'type': '109', 'version': '1.1.40', 'force': False, 'justification': None, 'timeout': 10, 'schedule': {'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T17:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T17:10:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T17:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T17:25:00.258870+00:00'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i', 'reqId': 1507741607555952} to its replicas since not finalised 2017-10-11 17:06:47,593 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"identifier":"V4SGRU86Z58d6TV7PBUe6f","reqId":1507741607555952,"op":"REQACK"}' to b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' through listener socket 2017-10-11 17:06:47,593 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"senderClient":"+>]}>OKfYQDWB?@taDN+?+{nc\\/ZoKRv9iH*\\/^F$a","request":{"operation":{"action":"start","name":"upgrade40","reinstall":false,"sha256":"ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f","type":"109","version":"1.1.40","force":false,"justification":null,"timeout":10,"schedule":{"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya":"2017-10-11T17:20:00.258870+00:00","Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv":"2017-10-11T17:10:00.258870+00:00","8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb":"2017-10-11T17:15:00.258870+00:00","4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA":"2017-10-11T17:25:00.258870+00:00"}},"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i","reqId":1507741607555952},"op":"PROPAGATE"}' to Node1 2017-10-11 17:06:47,593 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"senderClient":"+>]}>OKfYQDWB?@taDN+?+{nc\\/ZoKRv9iH*\\/^F$a","request":{"operation":{"action":"start","name":"upgrade40","reinstall":false,"sha256":"ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f","type":"109","version":"1.1.40","force":false,"justification":null,"timeout":10,"schedule":{"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya":"2017-10-11T17:20:00.258870+00:00","Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv":"2017-10-11T17:10:00.258870+00:00","8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb":"2017-10-11T17:15:00.258870+00:00","4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA":"2017-10-11T17:25:00.258870+00:00"}},"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i","reqId":1507741607555952},"op":"PROPAGATE"}' to Node1 2017-10-11 17:06:47,593 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"senderClient":"+>]}>OKfYQDWB?@taDN+?+{nc\\/ZoKRv9iH*\\/^F$a","request":{"operation":{"action":"start","name":"upgrade40","reinstall":false,"sha256":"ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f","type":"109","version":"1.1.40","force":false,"justification":null,"timeout":10,"schedule":{"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya":"2017-10-11T17:20:00.258870+00:00","Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv":"2017-10-11T17:10:00.258870+00:00","8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb":"2017-10-11T17:15:00.258870+00:00","4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA":"2017-10-11T17:25:00.258870+00:00"}},"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i","reqId":1507741607555952},"op":"PROPAGATE"}' to Node3 2017-10-11 17:06:47,594 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"senderClient":"+>]}>OKfYQDWB?@taDN+?+{nc\\/ZoKRv9iH*\\/^F$a","request":{"operation":{"action":"start","name":"upgrade40","reinstall":false,"sha256":"ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f","type":"109","version":"1.1.40","force":false,"justification":null,"timeout":10,"schedule":{"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya":"2017-10-11T17:20:00.258870+00:00","Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv":"2017-10-11T17:10:00.258870+00:00","8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb":"2017-10-11T17:15:00.258870+00:00","4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA":"2017-10-11T17:25:00.258870+00:00"}},"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i","reqId":1507741607555952},"op":"PROPAGATE"}' to Node3 2017-10-11 17:06:47,602 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"senderClient":"+>]}>OKfYQDWB?@taDN+?+{nc\\/ZoKRv9iH*\\/^F$a","request":{"operation":{"action":"start","name":"upgrade40","reinstall":false,"sha256":"ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f","type":"109","version":"1.1.40","force":false,"justification":null,"timeout":10,"schedule":{"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya":"2017-10-11T17:20:00.258870+00:00","Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv":"2017-10-11T17:10:00.258870+00:00","8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb":"2017-10-11T17:15:00.258870+00:00","4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA":"2017-10-11T17:25:00.258870+00:00"}},"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i","reqId":1507741607555952},"op":"PROPAGATE"}' to Node4 2017-10-11 17:06:47,603 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"senderClient":"+>]}>OKfYQDWB?@taDN+?+{nc\\/ZoKRv9iH*\\/^F$a","request":{"operation":{"action":"start","name":"upgrade40","reinstall":false,"sha256":"ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f","type":"109","version":"1.1.40","force":false,"justification":null,"timeout":10,"schedule":{"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya":"2017-10-11T17:20:00.258870+00:00","Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv":"2017-10-11T17:10:00.258870+00:00","8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb":"2017-10-11T17:15:00.258870+00:00","4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA":"2017-10-11T17:25:00.258870+00:00"}},"identifier":"V4SGRU86Z58d6TV7PBUe6f","signature":"3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i","reqId":1507741607555952},"op":"PROPAGATE"}' to Node4 2017-10-11 17:06:47,604 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 3 messages through listener 2017-10-11 17:06:47,605 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1507741607555952 2017-10-11 17:06:47,605 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: PROPAGATE{'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a', 'request': {'operation': {'action': 'start', 'name': 'upgrade40', 'reinstall': False, 'force': False, 'type': '109', 'version': '1.1.40', 'timeout': 10, 'justification': None, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T17:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T17:25:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T17:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T17:10:00.258870+00:00'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i', 'reqId': 1507741607555952}} 2017-10-11 17:06:47,605 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a', 'request': {'operation': {'action': 'start', 'name': 'upgrade40', 'reinstall': False, 'force': False, 'type': '109', 'version': '1.1.40', 'timeout': 10, 'justification': None, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T17:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T17:25:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T17:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T17:10:00.258870+00:00'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i', 'reqId': 1507741607555952}, 'op': 'PROPAGATE'}, 'Node4') 2017-10-11 17:06:47,605 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a', 'request': {'operation': {'action': 'start', 'name': 'upgrade40', 'reinstall': False, 'force': False, 'type': '109', 'version': '1.1.40', 'timeout': 10, 'justification': None, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T17:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T17:25:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T17:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T17:10:00.258870+00:00'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i', 'reqId': 1507741607555952}} 2017-10-11 17:06:47,606 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1507741607555952 2017-10-11 17:06:47,606 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PROPAGATE{'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a', 'request': {'operation': {'action': 'start', 'name': 'upgrade40', 'reinstall': False, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'type': '109', 'version': '1.1.40', 'force': False, 'justification': None, 'timeout': 10, 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T17:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T17:25:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T17:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T17:10:00.258870+00:00'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i', 'reqId': 1507741607555952}} 2017-10-11 17:06:47,606 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a', 'request': {'operation': {'action': 'start', 'name': 'upgrade40', 'reinstall': False, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'type': '109', 'version': '1.1.40', 'force': False, 'justification': None, 'timeout': 10, 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T17:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T17:25:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T17:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T17:10:00.258870+00:00'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i', 'reqId': 1507741607555952}, 'op': 'PROPAGATE'}, 'Node3') 2017-10-11 17:06:47,606 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a', 'request': {'operation': {'action': 'start', 'name': 'upgrade40', 'reinstall': False, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'type': '109', 'version': '1.1.40', 'force': False, 'justification': None, 'timeout': 10, 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T17:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T17:25:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T17:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T17:10:00.258870+00:00'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i', 'reqId': 1507741607555952}} 2017-10-11 17:06:47,611 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated V4SGRU86Z58d6TV7PBUe6f signature on propagate request 1507741607555952 2017-10-11 17:06:47,612 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PROPAGATE{'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a', 'request': {'operation': {'action': 'start', 'name': 'upgrade40', 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'reinstall': False, 'timeout': 10, 'version': '1.1.40', 'schedule': {'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T17:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T17:10:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T17:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T17:25:00.258870+00:00'}, 'justification': None, 'type': '109', 'force': False}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i', 'reqId': 1507741607555952}} 2017-10-11 17:06:47,612 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a', 'request': {'operation': {'action': 'start', 'name': 'upgrade40', 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'reinstall': False, 'timeout': 10, 'version': '1.1.40', 'schedule': {'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T17:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T17:10:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T17:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T17:25:00.258870+00:00'}, 'justification': None, 'type': '109', 'force': False}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i', 'reqId': 1507741607555952}, 'op': 'PROPAGATE'}, 'Node1') 2017-10-11 17:06:47,612 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a', 'request': {'operation': {'action': 'start', 'name': 'upgrade40', 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'reinstall': False, 'timeout': 10, 'version': '1.1.40', 'schedule': {'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T17:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T17:10:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T17:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T17:25:00.258870+00:00'}, 'justification': None, 'type': '109', 'force': False}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i', 'reqId': 1507741607555952}} 2017-10-11 17:06:47,612 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a', 'request': {'operation': {'action': 'start', 'name': 'upgrade40', 'reinstall': False, 'force': False, 'type': '109', 'version': '1.1.40', 'timeout': 10, 'justification': None, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T17:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T17:25:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T17:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T17:10:00.258870+00:00'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i', 'reqId': 1507741607555952}} 2017-10-11 17:06:47,613 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'action': 'start', 'name': 'upgrade40', 'reinstall': False, 'force': False, 'type': '109', 'version': '1.1.40', 'timeout': 10, 'justification': None, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T17:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T17:25:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T17:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T17:10:00.258870+00:00'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i', 'reqId': 1507741607555952} 2017-10-11 17:06:47,613 | DEBUG | propagator.py ( 214) | forward | Node2 forwarding request ('V4SGRU86Z58d6TV7PBUe6f', 1507741607555952) to 0 replicas 2017-10-11 17:06:47,613 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a', 'request': {'operation': {'action': 'start', 'name': 'upgrade40', 'reinstall': False, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'type': '109', 'version': '1.1.40', 'force': False, 'justification': None, 'timeout': 10, 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T17:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T17:25:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T17:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T17:10:00.258870+00:00'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i', 'reqId': 1507741607555952}} 2017-10-11 17:06:47,613 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'action': 'start', 'name': 'upgrade40', 'reinstall': False, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'type': '109', 'version': '1.1.40', 'force': False, 'justification': None, 'timeout': 10, 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T17:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T17:25:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T17:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T17:10:00.258870+00:00'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i', 'reqId': 1507741607555952} 2017-10-11 17:06:47,614 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'action': 'start', 'name': 'upgrade40', 'reinstall': False, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'type': '109', 'version': '1.1.40', 'force': False, 'justification': None, 'timeout': 10, 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T17:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T17:25:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T17:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T17:10:00.258870+00:00'}}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i', 'reqId': 1507741607555952} to its replicas since already forwarded 2017-10-11 17:06:47,614 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': '+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a', 'request': {'operation': {'action': 'start', 'name': 'upgrade40', 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'reinstall': False, 'timeout': 10, 'version': '1.1.40', 'schedule': {'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T17:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T17:10:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T17:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T17:25:00.258870+00:00'}, 'justification': None, 'type': '109', 'force': False}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i', 'reqId': 1507741607555952}} 2017-10-11 17:06:47,616 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'action': 'start', 'name': 'upgrade40', 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'reinstall': False, 'timeout': 10, 'version': '1.1.40', 'schedule': {'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T17:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T17:10:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T17:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T17:25:00.258870+00:00'}, 'justification': None, 'type': '109', 'force': False}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i', 'reqId': 1507741607555952} 2017-10-11 17:06:47,616 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'action': 'start', 'name': 'upgrade40', 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'reinstall': False, 'timeout': 10, 'version': '1.1.40', 'schedule': {'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T17:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T17:10:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T17:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T17:25:00.258870+00:00'}, 'justification': None, 'type': '109', 'force': False}, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i', 'reqId': 1507741607555952} to its replicas since already forwarded 2017-10-11 17:06:47,617 | DEBUG | replica.py ( 667) | create3PCBatch | Node2:1 creating batch 3 for ledger 2 with state root None 2017-10-11 17:06:47,617 | DEBUG | replica.py ( 700) | create3PCBatch | Node2:1 created a PRE-PREPARE with 1 requests for ledger 2 2017-10-11 17:06:47,617 | DEBUG | replica.py (2193) | send | Node2:1 sending PrePrepare 2017-10-11 17:06:47,617 | TRACE | replica.py (2194) | send | Node2:1 sending PREPREPARE{'ledgerId': 2, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741607555952)], 'ppSeqNo': 3, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741607, 'discarded': 1, 'digest': 'e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c', 'viewNo': 0} 2017-10-11 17:06:47,617 | DEBUG | replica.py ( 614) | trackBatches | Node2:1 tracking batch for PREPREPARE{'ledgerId': 2, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741607555952)], 'ppSeqNo': 3, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741607, 'discarded': 1, 'digest': 'e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c', 'viewNo': 0} with state root None 2017-10-11 17:06:47,617 | DEBUG | node.py (2608) | send | Node2 sending message PREPREPARE{'ledgerId': 2, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741607555952)], 'ppSeqNo': 3, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741607, 'discarded': 1, 'digest': 'e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c', 'viewNo': 0} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:06:47,618 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":null,"digest":"e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c","discarded":1,"ledgerId":2,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741607555952]],"ppSeqNo":3,"op":"PREPREPARE","instId":1,"ppTime":1507741607,"viewNo":0,"txnRootHash":null}' to Node1 2017-10-11 17:06:47,618 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":null,"digest":"e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c","discarded":1,"ledgerId":2,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741607555952]],"ppSeqNo":3,"op":"PREPREPARE","instId":1,"ppTime":1507741607,"viewNo":0,"txnRootHash":null}' to Node1 2017-10-11 17:06:47,618 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":null,"digest":"e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c","discarded":1,"ledgerId":2,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741607555952]],"ppSeqNo":3,"op":"PREPREPARE","instId":1,"ppTime":1507741607,"viewNo":0,"txnRootHash":null}' to Node3 2017-10-11 17:06:47,619 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":null,"digest":"e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c","discarded":1,"ledgerId":2,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741607555952]],"ppSeqNo":3,"op":"PREPREPARE","instId":1,"ppTime":1507741607,"viewNo":0,"txnRootHash":null}' to Node3 2017-10-11 17:06:47,619 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":null,"digest":"e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c","discarded":1,"ledgerId":2,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741607555952]],"ppSeqNo":3,"op":"PREPREPARE","instId":1,"ppTime":1507741607,"viewNo":0,"txnRootHash":null}' to Node4 2017-10-11 17:06:47,619 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":null,"digest":"e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c","discarded":1,"ledgerId":2,"reqIdr":[["V4SGRU86Z58d6TV7PBUe6f",1507741607555952]],"ppSeqNo":3,"op":"PREPREPARE","instId":1,"ppTime":1507741607,"viewNo":0,"txnRootHash":null}' to Node4 2017-10-11 17:06:47,620 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:06:47,621 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PREPREPARE{'ledgerId': 2, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1507741607555952]], 'ppSeqNo': 3, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': '4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7', 'instId': 0, 'ppTime': 1507741607, 'discarded': 1, 'digest': 'e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c', 'viewNo': 0} 2017-10-11 17:06:47,621 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'ledgerId': 2, 'digest': 'e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c', 'ppSeqNo': 3, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'op': 'PREPREPARE', 'txnRootHash': '4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7', 'instId': 0, 'ppTime': 1507741607, 'discarded': 1, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1507741607555952]], 'viewNo': 0}, 'Node1') 2017-10-11 17:06:47,630 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPREPARE{'ledgerId': 2, 'reqIdr': [['V4SGRU86Z58d6TV7PBUe6f', 1507741607555952]], 'ppSeqNo': 3, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': '4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7', 'instId': 0, 'ppTime': 1507741607, 'discarded': 1, 'digest': 'e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c', 'viewNo': 0} 2017-10-11 17:06:47,631 | DEBUG | replica.py ( 823) | processPrePrepare | Node2:0 received PRE-PREPARE(0, 3) from Node1:0 at 2270693.089141047 2017-10-11 17:06:47,632 | DEBUG | replica.py (1034) | validate_pre_prepare | Node2:0 state root before processing PREPREPARE{'ledgerId': 2, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741607555952)], 'ppSeqNo': 3, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': '4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7', 'instId': 0, 'ppTime': 1507741607, 'discarded': 1, 'digest': 'e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c', 'viewNo': 0} is b'\xbc q\xa4\xde\x84o(W\x02D\x7f%\x89\xdd\x166x\xe0\x97*\x8a\x1b\r(\xb0N\xd5\xc0\x94T\x7f', AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ 2017-10-11 17:06:47,633 | DEBUG | replica.py ( 934) | doPrepare | Node2:0 Sending PREPARE(0, 3) at 2270693.090526517 2017-10-11 17:06:47,634 | DEBUG | replica.py (2193) | send | Node2:0 sending Prepare 2017-10-11 17:06:47,634 | TRACE | replica.py (2194) | send | Node2:0 sending PREPARE{'digest': 'e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c', 'ppSeqNo': 3, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': '4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7', 'instId': 0, 'ppTime': 1507741607, 'viewNo': 0} 2017-10-11 17:06:47,635 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': 'e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c', 'ppSeqNo': 3, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': '4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7', 'instId': 0, 'ppTime': 1507741607, 'viewNo': 0} 2017-10-11 17:06:47,635 | DEBUG | replica.py ( 614) | trackBatches | Node2:0 tracking batch for PREPREPARE{'ledgerId': 2, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741607555952)], 'ppSeqNo': 3, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': '4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7', 'instId': 0, 'ppTime': 1507741607, 'discarded': 1, 'digest': 'e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c', 'viewNo': 0} with state root b'\xbc q\xa4\xde\x84o(W\x02D\x7f%\x89\xdd\x166x\xe0\x97*\x8a\x1b\r(\xb0N\xd5\xc0\x94T\x7f' 2017-10-11 17:06:47,635 | DEBUG | replica.py ( 842) | processPrePrepare | Node2:0 processed incoming PRE-PREPARE(0, 3) 2017-10-11 17:06:47,635 | DEBUG | node.py (2608) | send | Node2 sending message PREPARE{'digest': 'e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c', 'ppSeqNo': 3, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': '4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7', 'instId': 0, 'ppTime': 1507741607, 'viewNo': 0} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:06:47,635 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:06:47,636 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: BATCH{'messages': ['{"txnRootHash":"4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7","ppSeqNo":3,"digest":"e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c","op":"PREPARE","viewNo":0,"ppTime":1507741607,"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","instId":0}', '{"txnRootHash":null,"ppSeqNo":3,"digest":"e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c","op":"PREPARE","viewNo":0,"ppTime":1507741607,"stateRootHash":null,"instId":1}'], 'signature': None} 2017-10-11 17:06:47,636 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"txnRootHash":"4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7","ppSeqNo":3,"digest":"e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c","op":"PREPARE","viewNo":0,"ppTime":1507741607,"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","instId":0}', '{"txnRootHash":null,"ppSeqNo":3,"digest":"e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c","op":"PREPARE","viewNo":0,"ppTime":1507741607,"stateRootHash":null,"instId":1}'], 'signature': None, 'op': 'BATCH'}, 'Node4') 2017-10-11 17:06:47,636 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"txnRootHash":"4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7","ppSeqNo":3,"digest":"e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c","op":"PREPARE","viewNo":0,"ppTime":1507741607,"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","instId":0}', '{"txnRootHash":null,"ppSeqNo":3,"digest":"e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c","op":"PREPARE","viewNo":0,"ppTime":1507741607,"stateRootHash":null,"instId":1}'], 'signature': None} 2017-10-11 17:06:47,637 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'digest': 'e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c', 'ppSeqNo': 3, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': '4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7', 'instId': 0, 'ppTime': 1507741607, 'viewNo': 0} 2017-10-11 17:06:47,637 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'ppSeqNo': 3, 'digest': 'e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c', 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': '4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7', 'op': 'PREPARE', 'instId': 0, 'ppTime': 1507741607, 'viewNo': 0}, 'Node4') 2017-10-11 17:06:47,637 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'digest': 'e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c', 'ppSeqNo': 3, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': '4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7', 'instId': 0, 'ppTime': 1507741607, 'viewNo': 0} 2017-10-11 17:06:47,637 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'digest': 'e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c', 'ppSeqNo': 3, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741607, 'viewNo': 0} 2017-10-11 17:06:47,637 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'ppSeqNo': 3, 'digest': 'e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c', 'stateRootHash': None, 'txnRootHash': None, 'op': 'PREPARE', 'instId': 1, 'ppTime': 1507741607, 'viewNo': 0}, 'Node4') 2017-10-11 17:06:47,637 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'digest': 'e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c', 'ppSeqNo': 3, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741607, 'viewNo': 0} 2017-10-11 17:06:47,638 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","viewNo":0,"digest":"e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c","ppSeqNo":3,"op":"PREPARE","instId":0,"ppTime":1507741607,"txnRootHash":"4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7"}' to Node1 2017-10-11 17:06:47,638 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","viewNo":0,"digest":"e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c","ppSeqNo":3,"op":"PREPARE","instId":0,"ppTime":1507741607,"txnRootHash":"4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7"}' to Node1 2017-10-11 17:06:47,638 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","viewNo":0,"digest":"e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c","ppSeqNo":3,"op":"PREPARE","instId":0,"ppTime":1507741607,"txnRootHash":"4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7"}' to Node3 2017-10-11 17:06:47,638 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","viewNo":0,"digest":"e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c","ppSeqNo":3,"op":"PREPARE","instId":0,"ppTime":1507741607,"txnRootHash":"4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7"}' to Node3 2017-10-11 17:06:47,639 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","viewNo":0,"digest":"e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c","ppSeqNo":3,"op":"PREPARE","instId":0,"ppTime":1507741607,"txnRootHash":"4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7"}' to Node4 2017-10-11 17:06:47,639 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","viewNo":0,"digest":"e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c","ppSeqNo":3,"op":"PREPARE","instId":0,"ppTime":1507741607,"txnRootHash":"4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7"}' to Node4 2017-10-11 17:06:47,639 | DEBUG | replica.py ( 953) | doCommit | Node2:0 Sending COMMIT(0, 3) at 2270693.097315668 2017-10-11 17:06:47,640 | DEBUG | replica.py (2193) | send | Node2:0 sending Commit 2017-10-11 17:06:47,640 | TRACE | replica.py (2194) | send | Node2:0 sending COMMIT{'instId': 0, 'ppSeqNo': 3, 'viewNo': 0} 2017-10-11 17:06:47,640 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'ppSeqNo': 3, 'viewNo': 0} commits where f is 1 2017-10-11 17:06:47,640 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (0, 3) 2017-10-11 17:06:47,640 | DEBUG | replica.py ( 917) | tryCommit | Node2:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': 'e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c', 'ppSeqNo': 3, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741607, 'viewNo': 0} 2017-10-11 17:06:47,640 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (0, 3) 2017-10-11 17:06:47,640 | DEBUG | node.py (2608) | send | Node2 sending message COMMIT{'instId': 0, 'ppSeqNo': 3, 'viewNo': 0} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:06:47,641 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":0,"ppSeqNo":3,"op":"COMMIT","viewNo":0}' to Node1 2017-10-11 17:06:47,641 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":0,"ppSeqNo":3,"op":"COMMIT","viewNo":0}' to Node1 2017-10-11 17:06:47,641 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":0,"ppSeqNo":3,"op":"COMMIT","viewNo":0}' to Node3 2017-10-11 17:06:47,642 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":0,"ppSeqNo":3,"op":"COMMIT","viewNo":0}' to Node3 2017-10-11 17:06:47,642 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":0,"ppSeqNo":3,"op":"COMMIT","viewNo":0}' to Node4 2017-10-11 17:06:47,642 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":0,"ppSeqNo":3,"op":"COMMIT","viewNo":0}' to Node4 2017-10-11 17:06:47,655 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:06:47,666 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PREPARE{'digest': 'e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c', 'ppSeqNo': 3, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741607, 'viewNo': 0} 2017-10-11 17:06:47,667 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'digest': 'e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c', 'ppSeqNo': 3, 'stateRootHash': None, 'op': 'PREPARE', 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741607, 'viewNo': 0}, 'Node1') 2017-10-11 17:06:47,667 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'digest': 'e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c', 'ppSeqNo': 3, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741607, 'viewNo': 0} 2017-10-11 17:06:47,668 | DEBUG | replica.py ( 953) | doCommit | Node2:1 Sending COMMIT(0, 3) at 2270693.125781659 2017-10-11 17:06:47,668 | DEBUG | replica.py (2193) | send | Node2:1 sending Commit 2017-10-11 17:06:47,668 | TRACE | replica.py (2194) | send | Node2:1 sending COMMIT{'instId': 1, 'ppSeqNo': 3, 'viewNo': 0} 2017-10-11 17:06:47,668 | DEBUG | replica.py ( 929) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'ppSeqNo': 3, 'viewNo': 0} commits where f is 1 2017-10-11 17:06:47,668 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (0, 3) 2017-10-11 17:06:47,669 | DEBUG | node.py (2608) | send | Node2 sending message COMMIT{'instId': 1, 'ppSeqNo': 3, 'viewNo': 0} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:06:47,669 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 4 messages through listener 2017-10-11 17:06:47,670 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 1, 'ppSeqNo': 3, 'viewNo': 0} 2017-10-11 17:06:47,670 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'ppSeqNo': 3, 'op': 'COMMIT', 'viewNo': 0}, 'Node1') 2017-10-11 17:06:47,670 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'ppSeqNo': 3, 'viewNo': 0} 2017-10-11 17:06:47,675 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: BATCH{'messages': ['{"ppSeqNo":3,"viewNo":0,"instId":0,"op":"PREPARE","stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","ppTime":1507741607,"digest":"e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c","txnRootHash":"4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7"}', '{"viewNo":0,"instId":0,"op":"COMMIT","ppSeqNo":3}', '{"ppSeqNo":3,"viewNo":0,"instId":1,"op":"PREPARE","stateRootHash":null,"ppTime":1507741607,"digest":"e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c","txnRootHash":null}', '{"viewNo":0,"instId":1,"op":"COMMIT","ppSeqNo":3}'], 'signature': None} 2017-10-11 17:06:47,675 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"ppSeqNo":3,"viewNo":0,"instId":0,"op":"PREPARE","stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","ppTime":1507741607,"digest":"e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c","txnRootHash":"4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7"}', '{"viewNo":0,"instId":0,"op":"COMMIT","ppSeqNo":3}', '{"ppSeqNo":3,"viewNo":0,"instId":1,"op":"PREPARE","stateRootHash":null,"ppTime":1507741607,"digest":"e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c","txnRootHash":null}', '{"viewNo":0,"instId":1,"op":"COMMIT","ppSeqNo":3}'], 'signature': None, 'op': 'BATCH'}, 'Node3') 2017-10-11 17:06:47,675 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"ppSeqNo":3,"viewNo":0,"instId":0,"op":"PREPARE","stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","ppTime":1507741607,"digest":"e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c","txnRootHash":"4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7"}', '{"viewNo":0,"instId":0,"op":"COMMIT","ppSeqNo":3}', '{"ppSeqNo":3,"viewNo":0,"instId":1,"op":"PREPARE","stateRootHash":null,"ppTime":1507741607,"digest":"e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c","txnRootHash":null}', '{"viewNo":0,"instId":1,"op":"COMMIT","ppSeqNo":3}'], 'signature': None} 2017-10-11 17:06:47,675 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'digest': 'e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c', 'ppSeqNo': 3, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': '4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7', 'instId': 0, 'ppTime': 1507741607, 'viewNo': 0} 2017-10-11 17:06:47,675 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'digest': 'e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c', 'ppSeqNo': 3, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'op': 'PREPARE', 'txnRootHash': '4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7', 'instId': 0, 'ppTime': 1507741607, 'viewNo': 0}, 'Node3') 2017-10-11 17:06:47,675 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'digest': 'e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c', 'ppSeqNo': 3, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': '4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7', 'instId': 0, 'ppTime': 1507741607, 'viewNo': 0} 2017-10-11 17:06:47,676 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 0, 'ppSeqNo': 3, 'viewNo': 0} 2017-10-11 17:06:47,676 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'ppSeqNo': 3, 'op': 'COMMIT', 'viewNo': 0}, 'Node3') 2017-10-11 17:06:47,676 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 3, 'viewNo': 0} 2017-10-11 17:06:47,676 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'digest': 'e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c', 'ppSeqNo': 3, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741607, 'viewNo': 0} 2017-10-11 17:06:47,676 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'digest': 'e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c', 'ppSeqNo': 3, 'stateRootHash': None, 'op': 'PREPARE', 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741607, 'viewNo': 0}, 'Node3') 2017-10-11 17:06:47,677 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'digest': 'e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c', 'ppSeqNo': 3, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741607, 'viewNo': 0} 2017-10-11 17:06:47,677 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 1, 'ppSeqNo': 3, 'viewNo': 0} 2017-10-11 17:06:47,677 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'ppSeqNo': 3, 'op': 'COMMIT', 'viewNo': 0}, 'Node3') 2017-10-11 17:06:47,677 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'ppSeqNo': 3, 'viewNo': 0} 2017-10-11 17:06:47,677 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: BATCH{'messages': ['{"op":"COMMIT","instId":0,"viewNo":0,"ppSeqNo":3}', '{"op":"COMMIT","instId":1,"viewNo":0,"ppSeqNo":3}'], 'signature': None} 2017-10-11 17:06:47,677 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"op":"COMMIT","instId":0,"viewNo":0,"ppSeqNo":3}', '{"op":"COMMIT","instId":1,"viewNo":0,"ppSeqNo":3}'], 'signature': None, 'op': 'BATCH'}, 'Node4') 2017-10-11 17:06:47,677 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"op":"COMMIT","instId":0,"viewNo":0,"ppSeqNo":3}', '{"op":"COMMIT","instId":1,"viewNo":0,"ppSeqNo":3}'], 'signature': None} 2017-10-11 17:06:47,678 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 0, 'ppSeqNo': 3, 'viewNo': 0} 2017-10-11 17:06:47,678 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'ppSeqNo': 3, 'op': 'COMMIT', 'viewNo': 0}, 'Node4') 2017-10-11 17:06:47,678 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 3, 'viewNo': 0} 2017-10-11 17:06:47,678 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 1, 'ppSeqNo': 3, 'viewNo': 0} 2017-10-11 17:06:47,678 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'ppSeqNo': 3, 'op': 'COMMIT', 'viewNo': 0}, 'Node4') 2017-10-11 17:06:47,678 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'ppSeqNo': 3, 'viewNo': 0} 2017-10-11 17:06:47,679 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 0, 'ppSeqNo': 3, 'viewNo': 0} 2017-10-11 17:06:47,679 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'ppSeqNo': 3, 'op': 'COMMIT', 'viewNo': 0}, 'Node1') 2017-10-11 17:06:47,679 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 3, 'viewNo': 0} 2017-10-11 17:06:47,680 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":1,"ppSeqNo":3,"op":"COMMIT","viewNo":0}' to Node1 2017-10-11 17:06:47,680 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":1,"ppSeqNo":3,"op":"COMMIT","viewNo":0}' to Node1 2017-10-11 17:06:47,680 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":1,"ppSeqNo":3,"op":"COMMIT","viewNo":0}' to Node3 2017-10-11 17:06:47,683 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":1,"ppSeqNo":3,"op":"COMMIT","viewNo":0}' to Node3 2017-10-11 17:06:47,687 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":1,"ppSeqNo":3,"op":"COMMIT","viewNo":0}' to Node4 2017-10-11 17:06:47,688 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":1,"ppSeqNo":3,"op":"COMMIT","viewNo":0}' to Node4 2017-10-11 17:06:47,689 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'digest': 'e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c', 'ppSeqNo': 3, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': '4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7', 'instId': 0, 'ppTime': 1507741607, 'viewNo': 0} 2017-10-11 17:06:47,690 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (0, 3) 2017-10-11 17:06:47,690 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(0, 3) from Node3:0 2017-10-11 17:06:47,690 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'ppSeqNo': 3, 'viewNo': 0} commits where f is 1 2017-10-11 17:06:47,690 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(0, 3) 2017-10-11 17:06:47,690 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(0, 3) from Node4:0 2017-10-11 17:06:47,690 | TRACE | replica.py ( 925) | tryOrder | Node2:0 returning request to node 2017-10-11 17:06:47,691 | DEBUG | replica.py (1444) | doOrder | Node2:0 ordering COMMIT (0, 3) 2017-10-11 17:06:47,691 | DEBUG | replica.py ( 323) | last_ordered_3pc | Node2:0 set last ordered as (0, 3) 2017-10-11 17:06:47,691 | DEBUG | replica.py (2193) | send | Node2:0 sending Ordered 2017-10-11 17:06:47,691 | TRACE | replica.py (2194) | send | Node2:0 sending ORDERED{'ledgerId': 2, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741607555952)], 'ppSeqNo': 3, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': '4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7', 'instId': 0, 'ppTime': 1507741607, 'viewNo': 0} 2017-10-11 17:06:47,691 | INFO | replica.py (1483) | order_3pc_key | Node2:0 ordered batch request, view no 0, ppSeqNo 3, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1507741607555952)], discarded [] 2017-10-11 17:06:47,692 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(0, 3) 2017-10-11 17:06:47,692 | TRACE | message_processor.py ( 29) | discard | Node2:0 discarding message COMMIT{'instId': 0, 'ppSeqNo': 3, 'viewNo': 0} because already ordered 3 phase message 2017-10-11 17:06:47,692 | DEBUG | replica.py ( 894) | processCommit | Node2:1 received COMMIT(0, 3) from Node1:1 2017-10-11 17:06:47,692 | DEBUG | replica.py ( 929) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'ppSeqNo': 3, 'viewNo': 0} commits where f is 1 2017-10-11 17:06:47,692 | DEBUG | replica.py ( 905) | processCommit | Node2:1 processed incoming COMMIT(0, 3) 2017-10-11 17:06:47,692 | DEBUG | replica.py ( 917) | tryCommit | Node2:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'digest': 'e304c8fbb37d600e25530ff017ff312ad4eba587158b5c57a238d2a92ca8bb6c', 'ppSeqNo': 3, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741607, 'viewNo': 0} 2017-10-11 17:06:47,692 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (0, 3) 2017-10-11 17:06:47,692 | DEBUG | replica.py ( 894) | processCommit | Node2:1 received COMMIT(0, 3) from Node3:1 2017-10-11 17:06:47,692 | TRACE | replica.py ( 925) | tryOrder | Node2:1 returning request to node 2017-10-11 17:06:47,693 | DEBUG | replica.py (1444) | doOrder | Node2:1 ordering COMMIT (0, 3) 2017-10-11 17:06:47,693 | DEBUG | replica.py ( 323) | last_ordered_3pc | Node2:1 set last ordered as (0, 3) 2017-10-11 17:06:47,693 | DEBUG | replica.py (2193) | send | Node2:1 sending Ordered 2017-10-11 17:06:47,693 | TRACE | replica.py (2194) | send | Node2:1 sending ORDERED{'ledgerId': 2, 'reqIdr': [('V4SGRU86Z58d6TV7PBUe6f', 1507741607555952)], 'ppSeqNo': 3, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741607, 'viewNo': 0} 2017-10-11 17:06:47,693 | INFO | replica.py (1483) | order_3pc_key | Node2:1 ordered batch request, view no 0, ppSeqNo 3, ledger 2, state root None, txn root None, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1507741607555952)], discarded [] 2017-10-11 17:06:47,693 | DEBUG | replica.py ( 905) | processCommit | Node2:1 processed incoming COMMIT(0, 3) 2017-10-11 17:06:47,693 | TRACE | message_processor.py ( 29) | discard | Node2:1 discarding message COMMIT{'instId': 1, 'ppSeqNo': 3, 'viewNo': 0} because already ordered 3 phase message 2017-10-11 17:06:47,693 | DEBUG | node.py (1847) | processOrdered | Node2 executing Ordered batch 0 3 of 1 requests 2017-10-11 17:06:47,694 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-10-11 17:06:47,694 | INFO | upgrader.py ( 292) | handleUpgradeTxn | Node 'Node2' handles upgrade txn {'action': 'start', 'name': 'upgrade40', 'txnTime': 1507741607, 'reinstall': False, 'auditPath': ['AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ'], 'type': '109', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'justification': None, 'signature': '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i', 'force': False, 'seqNo': 2, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'timeout': 10, 'version': '1.1.40', 'rootHash': '4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7', 'reqId': 1507741607555952, 'schedule': {'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-10-11T17:20:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-10-11T17:10:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-10-11T17:15:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-10-11T17:25:00.258870+00:00'}} 2017-10-11 17:06:47,694 | INFO | upgrader.py ( 321) | handleUpgradeTxn | Node 'Node2' schedules upgrade to 1.1.40 2017-10-11 17:06:47,694 | INFO | upgrader.py ( 359) | _scheduleUpgrade | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb's upgrader processing upgrade for version 1.1.40 2017-10-11 17:06:47,695 | TRACE | has_action_queue.py ( 36) | _schedule | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb scheduling action partial(_callUpgradeAgent) with id 1 to run in 492 seconds 2017-10-11 17:06:47,695 | DEBUG | node.py (2386) | sendReplyToClient | Node2 sending reply for ('V4SGRU86Z58d6TV7PBUe6f', 1507741607555952) to client 2017-10-11 17:06:47,695 | TRACE | zstack.py ( 754) | transmitThroughListener | Node2C transmitting b'{"result":{"action":"start","name":"upgrade40","txnTime":1507741607,"reinstall":false,"auditPath":["AbMRpp8BeBz1f2CzFcoode4pTaGV2d4rv32XCRVMY4pZ"],"type":"109","identifier":"V4SGRU86Z58d6TV7PBUe6f","justification":null,"signature":"3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i","force":false,"seqNo":2,"sha256":"ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f","timeout":10,"version":"1.1.40","rootHash":"4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7","reqId":1507741607555952,"schedule":{"DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya":"2017-10-11T17:20:00.258870+00:00","Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv":"2017-10-11T17:10:00.258870+00:00","8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb":"2017-10-11T17:15:00.258870+00:00","4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA":"2017-10-11T17:25:00.258870+00:00"}},"op":"REPLY"}' to b'+>]}>OKfYQDWB?@taDN+?+{nc/ZoKRv9iH*/^F$a' through listener socket 2017-10-11 17:06:47,695 | INFO | node.py (2290) | executeBatch | Node2 committed batch request, view no 0, ppSeqNo 3, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7, requests: [('V4SGRU86Z58d6TV7PBUe6f', 1507741607555952)] 2017-10-11 17:06:47,696 | DEBUG | node.py (2305) | executeBatch | Node2 storing 3PC key (0, 3) for ledger 2 range (2, 2) 2017-10-11 17:06:47,696 | TRACE | node.py (1858) | processOrdered | Node2 got ordered requests from backup replica 1 2017-10-11 17:06:47,696 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:06:51,943 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:06:51,943 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:06:51,943 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:06:51,944 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:06:56,932 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 25 2017-10-11 17:06:56,932 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:06:56,933 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:06:56,933 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0018318735516387 is acceptable. 2017-10-11 17:06:56,933 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:06:56,934 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:06:56,934 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:06:56,934 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 26 to run in 10 seconds 2017-10-11 17:07:06,805 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 3 2017-10-11 17:07:06,805 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:07:06,805 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 4 to run in 60 seconds 2017-10-11 17:07:06,818 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 19 2017-10-11 17:07:06,818 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node2 checking its request amount 2017-10-11 17:07:06,819 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:07:06,819 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 27 to run in 60 seconds 2017-10-11 17:07:06,904 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action dump_json_file with id 21 2017-10-11 17:07:06,905 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action dump_json_file with id 28 to run in 60 seconds 2017-10-11 17:07:06,941 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 26 2017-10-11 17:07:06,941 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:07:06,941 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:07:06,941 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0018318735516387 is acceptable. 2017-10-11 17:07:06,941 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:07:06,942 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:07:06,942 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:07:06,942 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 29 to run in 10 seconds 2017-10-11 17:07:06,943 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:07:06,943 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:07:06,943 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:07:06,944 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:07:16,949 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 29 2017-10-11 17:07:16,949 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:07:16,949 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:07:16,950 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0018318735516387 is acceptable. 2017-10-11 17:07:16,950 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:07:16,950 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:07:16,950 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:07:16,950 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 30 to run in 10 seconds 2017-10-11 17:07:21,948 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:07:21,948 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:07:21,948 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:07:21,949 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:07:26,958 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 30 2017-10-11 17:07:26,959 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:07:26,960 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:07:26,960 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0018318735516387 is acceptable. 2017-10-11 17:07:26,963 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:07:26,963 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:07:26,963 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:07:26,963 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 31 to run in 10 seconds 2017-10-11 17:07:36,959 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:07:36,959 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:07:36,960 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:07:36,961 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:07:36,973 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 31 2017-10-11 17:07:36,974 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:07:36,974 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:07:36,975 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0018318735516387 is acceptable. 2017-10-11 17:07:36,975 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:07:36,976 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:07:36,976 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:07:36,976 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 32 to run in 10 seconds 2017-10-11 17:07:46,982 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 32 2017-10-11 17:07:46,984 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:07:46,984 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:07:46,984 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0018318735516387 is acceptable. 2017-10-11 17:07:46,984 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:07:46,985 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:07:46,985 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:07:46,985 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 33 to run in 10 seconds 2017-10-11 17:07:51,964 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:07:51,964 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:07:51,964 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:07:51,965 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:07:56,993 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 33 2017-10-11 17:07:56,993 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:07:56,994 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:07:56,994 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0018318735516387 is acceptable. 2017-10-11 17:07:56,994 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:07:56,994 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:07:56,994 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:07:56,994 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 34 to run in 10 seconds 2017-10-11 17:08:06,807 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 4 2017-10-11 17:08:06,808 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:08:06,808 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 5 to run in 60 seconds 2017-10-11 17:08:06,821 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 27 2017-10-11 17:08:06,821 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node2 checking its request amount 2017-10-11 17:08:06,821 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:08:06,821 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 35 to run in 60 seconds 2017-10-11 17:08:06,915 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action dump_json_file with id 28 2017-10-11 17:08:06,916 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action dump_json_file with id 36 to run in 60 seconds 2017-10-11 17:08:06,975 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:08:06,975 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:08:06,975 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:08:06,976 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:08:06,999 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 34 2017-10-11 17:08:06,999 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:08:06,999 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:08:06,999 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0018318735516387 is acceptable. 2017-10-11 17:08:06,999 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:08:07,000 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:08:07,000 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:08:07,000 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 37 to run in 10 seconds 2017-10-11 17:08:17,009 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 37 2017-10-11 17:08:17,010 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:08:17,010 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:08:17,011 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0018318735516387 is acceptable. 2017-10-11 17:08:17,011 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:08:17,011 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:08:17,011 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:08:17,011 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 38 to run in 10 seconds 2017-10-11 17:08:21,975 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:08:21,975 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:08:21,976 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:08:21,976 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:08:27,017 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 38 2017-10-11 17:08:27,017 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:08:27,017 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:08:27,017 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0018318735516387 is acceptable. 2017-10-11 17:08:27,018 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:08:27,018 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:08:27,018 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:08:27,018 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 39 to run in 10 seconds 2017-10-11 17:08:36,976 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:08:36,976 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:08:36,976 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:08:36,976 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:08:37,024 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 39 2017-10-11 17:08:37,025 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:08:37,025 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:08:37,025 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0018318735516387 is acceptable. 2017-10-11 17:08:37,025 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:08:37,025 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:08:37,025 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:08:37,025 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 40 to run in 10 seconds 2017-10-11 17:08:47,029 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 40 2017-10-11 17:08:47,029 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:08:47,029 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:08:47,029 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0018318735516387 is acceptable. 2017-10-11 17:08:47,030 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:08:47,030 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:08:47,030 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:08:47,030 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 41 to run in 10 seconds 2017-10-11 17:08:51,977 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:08:51,977 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:08:51,977 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:08:51,977 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:08:57,032 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 41 2017-10-11 17:08:57,032 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:08:57,032 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:08:57,032 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0018318735516387 is acceptable. 2017-10-11 17:08:57,032 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:08:57,033 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:08:57,034 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:08:57,034 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 42 to run in 10 seconds 2017-10-11 17:09:06,809 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 5 2017-10-11 17:09:06,810 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:09:06,810 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 6 to run in 60 seconds 2017-10-11 17:09:06,823 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 35 2017-10-11 17:09:06,824 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node2 checking its request amount 2017-10-11 17:09:06,824 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:09:06,824 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 43 to run in 60 seconds 2017-10-11 17:09:06,919 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action dump_json_file with id 36 2017-10-11 17:09:06,919 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action dump_json_file with id 44 to run in 60 seconds 2017-10-11 17:09:06,982 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:09:06,982 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:09:06,982 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:09:06,983 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:09:07,034 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 42 2017-10-11 17:09:07,034 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:09:07,034 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:09:07,034 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0018318735516387 is acceptable. 2017-10-11 17:09:07,035 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:09:07,035 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:09:07,035 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:09:07,035 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 45 to run in 10 seconds 2017-10-11 17:09:17,048 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 45 2017-10-11 17:09:17,048 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:09:17,048 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:09:17,048 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0018318735516387 is acceptable. 2017-10-11 17:09:17,048 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:09:17,049 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:09:17,049 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:09:17,049 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 46 to run in 10 seconds 2017-10-11 17:09:21,993 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:09:21,993 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:09:21,993 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:09:21,993 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:09:27,049 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 46 2017-10-11 17:09:27,049 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:09:27,050 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:09:27,050 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0018318735516387 is acceptable. 2017-10-11 17:09:27,050 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:09:27,050 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:09:27,051 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:09:27,051 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 47 to run in 10 seconds 2017-10-11 17:09:36,993 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:09:36,993 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:09:36,993 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:09:36,994 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:09:37,054 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 47 2017-10-11 17:09:37,054 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:09:37,054 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:09:37,054 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0018318735516387 is acceptable. 2017-10-11 17:09:37,054 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:09:37,055 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:09:37,055 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:09:37,055 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 48 to run in 10 seconds 2017-10-11 17:09:47,057 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 48 2017-10-11 17:09:47,057 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:09:47,057 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:09:47,057 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0018318735516387 is acceptable. 2017-10-11 17:09:47,057 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:09:47,057 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:09:47,057 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:09:47,057 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 49 to run in 10 seconds 2017-10-11 17:09:52,001 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:09:52,001 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:09:52,001 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:09:52,004 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:09:57,059 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 49 2017-10-11 17:09:57,059 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:09:57,060 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:09:57,060 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0018318735516387 is acceptable. 2017-10-11 17:09:57,060 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:09:57,060 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:09:57,061 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:09:57,061 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 50 to run in 10 seconds 2017-10-11 17:09:59,697 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:09:59,698 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': 'JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '3vKjs8t62JJURrrFFCfqaWwRQBsj563DHh7Yuy87BSbCdZWfzYUfPQVvSjpww2fMLwabneqkKQkAtKyh4Quvb9xL', 'reqId': 1507741799688892}, 'Node1') 2017-10-11 17:09:59,698 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox SafeRequest: {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': 'JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '3vKjs8t62JJURrrFFCfqaWwRQBsj563DHh7Yuy87BSbCdZWfzYUfPQVvSjpww2fMLwabneqkKQkAtKyh4Quvb9xL', 'reqId': 1507741799688892} 2017-10-11 17:09:59,699 | TRACE | propagator.py ( 165) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': 'JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '3vKjs8t62JJURrrFFCfqaWwRQBsj563DHh7Yuy87BSbCdZWfzYUfPQVvSjpww2fMLwabneqkKQkAtKyh4Quvb9xL', 'reqId': 1507741799688892} 2017-10-11 17:09:59,699 | INFO | propagator.py ( 148) | propagate | Node2 propagating request ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741799688892) from client Node1 2017-10-11 17:09:59,699 | DEBUG | node.py (2608) | send | Node2 sending message PROPAGATE{'senderClient': 'Node1', 'request': {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': 'JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '3vKjs8t62JJURrrFFCfqaWwRQBsj563DHh7Yuy87BSbCdZWfzYUfPQVvSjpww2fMLwabneqkKQkAtKyh4Quvb9xL', 'reqId': 1507741799688892}} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:09:59,699 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': 'JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '3vKjs8t62JJURrrFFCfqaWwRQBsj563DHh7Yuy87BSbCdZWfzYUfPQVvSjpww2fMLwabneqkKQkAtKyh4Quvb9xL', 'reqId': 1507741799688892} to its replicas since not finalised 2017-10-11 17:09:59,700 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"senderClient":"Node1","request":{"operation":{"type":"110","data":{"action":"in_progress","version":"1.1.40"},"signature":"JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw"},"identifier":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv","signature":"3vKjs8t62JJURrrFFCfqaWwRQBsj563DHh7Yuy87BSbCdZWfzYUfPQVvSjpww2fMLwabneqkKQkAtKyh4Quvb9xL","reqId":1507741799688892},"op":"PROPAGATE"}' to Node1 2017-10-11 17:09:59,700 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"senderClient":"Node1","request":{"operation":{"type":"110","data":{"action":"in_progress","version":"1.1.40"},"signature":"JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw"},"identifier":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv","signature":"3vKjs8t62JJURrrFFCfqaWwRQBsj563DHh7Yuy87BSbCdZWfzYUfPQVvSjpww2fMLwabneqkKQkAtKyh4Quvb9xL","reqId":1507741799688892},"op":"PROPAGATE"}' to Node1 2017-10-11 17:09:59,700 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"senderClient":"Node1","request":{"operation":{"type":"110","data":{"action":"in_progress","version":"1.1.40"},"signature":"JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw"},"identifier":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv","signature":"3vKjs8t62JJURrrFFCfqaWwRQBsj563DHh7Yuy87BSbCdZWfzYUfPQVvSjpww2fMLwabneqkKQkAtKyh4Quvb9xL","reqId":1507741799688892},"op":"PROPAGATE"}' to Node3 2017-10-11 17:09:59,700 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"senderClient":"Node1","request":{"operation":{"type":"110","data":{"action":"in_progress","version":"1.1.40"},"signature":"JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw"},"identifier":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv","signature":"3vKjs8t62JJURrrFFCfqaWwRQBsj563DHh7Yuy87BSbCdZWfzYUfPQVvSjpww2fMLwabneqkKQkAtKyh4Quvb9xL","reqId":1507741799688892},"op":"PROPAGATE"}' to Node3 2017-10-11 17:09:59,701 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"senderClient":"Node1","request":{"operation":{"type":"110","data":{"action":"in_progress","version":"1.1.40"},"signature":"JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw"},"identifier":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv","signature":"3vKjs8t62JJURrrFFCfqaWwRQBsj563DHh7Yuy87BSbCdZWfzYUfPQVvSjpww2fMLwabneqkKQkAtKyh4Quvb9xL","reqId":1507741799688892},"op":"PROPAGATE"}' to Node4 2017-10-11 17:09:59,701 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"senderClient":"Node1","request":{"operation":{"type":"110","data":{"action":"in_progress","version":"1.1.40"},"signature":"JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw"},"identifier":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv","signature":"3vKjs8t62JJURrrFFCfqaWwRQBsj563DHh7Yuy87BSbCdZWfzYUfPQVvSjpww2fMLwabneqkKQkAtKyh4Quvb9xL","reqId":1507741799688892},"op":"PROPAGATE"}' to Node4 2017-10-11 17:09:59,708 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:09:59,709 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv signature on propagate request 1507741799688892 2017-10-11 17:09:59,709 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PROPAGATE{'senderClient': 'Node1', 'request': {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': 'JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '3vKjs8t62JJURrrFFCfqaWwRQBsj563DHh7Yuy87BSbCdZWfzYUfPQVvSjpww2fMLwabneqkKQkAtKyh4Quvb9xL', 'reqId': 1507741799688892}} 2017-10-11 17:09:59,709 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'senderClient': 'Node1', 'request': {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': 'JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '3vKjs8t62JJURrrFFCfqaWwRQBsj563DHh7Yuy87BSbCdZWfzYUfPQVvSjpww2fMLwabneqkKQkAtKyh4Quvb9xL', 'reqId': 1507741799688892}, 'op': 'PROPAGATE'}, 'Node1') 2017-10-11 17:09:59,709 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': 'Node1', 'request': {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': 'JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '3vKjs8t62JJURrrFFCfqaWwRQBsj563DHh7Yuy87BSbCdZWfzYUfPQVvSjpww2fMLwabneqkKQkAtKyh4Quvb9xL', 'reqId': 1507741799688892}} 2017-10-11 17:09:59,709 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': 'Node1', 'request': {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': 'JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '3vKjs8t62JJURrrFFCfqaWwRQBsj563DHh7Yuy87BSbCdZWfzYUfPQVvSjpww2fMLwabneqkKQkAtKyh4Quvb9xL', 'reqId': 1507741799688892}} 2017-10-11 17:09:59,710 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': 'JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '3vKjs8t62JJURrrFFCfqaWwRQBsj563DHh7Yuy87BSbCdZWfzYUfPQVvSjpww2fMLwabneqkKQkAtKyh4Quvb9xL', 'reqId': 1507741799688892} 2017-10-11 17:09:59,710 | DEBUG | propagator.py ( 214) | forward | Node2 forwarding request ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741799688892) to 0 replicas 2017-10-11 17:09:59,711 | DEBUG | replica.py ( 667) | create3PCBatch | Node2:1 creating batch 4 for ledger 2 with state root None 2017-10-11 17:09:59,711 | DEBUG | replica.py ( 700) | create3PCBatch | Node2:1 created a PRE-PREPARE with 1 requests for ledger 2 2017-10-11 17:09:59,711 | DEBUG | replica.py (2193) | send | Node2:1 sending PrePrepare 2017-10-11 17:09:59,711 | TRACE | replica.py (2194) | send | Node2:1 sending PREPREPARE{'ledgerId': 2, 'reqIdr': [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741799688892)], 'ppSeqNo': 4, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741799, 'discarded': 1, 'digest': '71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12', 'viewNo': 0} 2017-10-11 17:09:59,711 | DEBUG | replica.py ( 614) | trackBatches | Node2:1 tracking batch for PREPREPARE{'ledgerId': 2, 'reqIdr': [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741799688892)], 'ppSeqNo': 4, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741799, 'discarded': 1, 'digest': '71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12', 'viewNo': 0} with state root None 2017-10-11 17:09:59,712 | DEBUG | node.py (2608) | send | Node2 sending message PREPREPARE{'ledgerId': 2, 'reqIdr': [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741799688892)], 'ppSeqNo': 4, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741799, 'discarded': 1, 'digest': '71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12', 'viewNo': 0} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:09:59,712 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:09:59,713 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv signature on propagate request 1507741799688892 2017-10-11 17:09:59,713 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PROPAGATE{'senderClient': 'Node1', 'request': {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': 'JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '3vKjs8t62JJURrrFFCfqaWwRQBsj563DHh7Yuy87BSbCdZWfzYUfPQVvSjpww2fMLwabneqkKQkAtKyh4Quvb9xL', 'reqId': 1507741799688892}} 2017-10-11 17:09:59,713 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'senderClient': 'Node1', 'request': {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': 'JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '3vKjs8t62JJURrrFFCfqaWwRQBsj563DHh7Yuy87BSbCdZWfzYUfPQVvSjpww2fMLwabneqkKQkAtKyh4Quvb9xL', 'reqId': 1507741799688892}, 'op': 'PROPAGATE'}, 'Node3') 2017-10-11 17:09:59,713 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': 'Node1', 'request': {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': 'JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '3vKjs8t62JJURrrFFCfqaWwRQBsj563DHh7Yuy87BSbCdZWfzYUfPQVvSjpww2fMLwabneqkKQkAtKyh4Quvb9xL', 'reqId': 1507741799688892}} 2017-10-11 17:09:59,713 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': 'Node1', 'request': {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': 'JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '3vKjs8t62JJURrrFFCfqaWwRQBsj563DHh7Yuy87BSbCdZWfzYUfPQVvSjpww2fMLwabneqkKQkAtKyh4Quvb9xL', 'reqId': 1507741799688892}} 2017-10-11 17:09:59,714 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': 'JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '3vKjs8t62JJURrrFFCfqaWwRQBsj563DHh7Yuy87BSbCdZWfzYUfPQVvSjpww2fMLwabneqkKQkAtKyh4Quvb9xL', 'reqId': 1507741799688892} 2017-10-11 17:09:59,714 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': 'JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '3vKjs8t62JJURrrFFCfqaWwRQBsj563DHh7Yuy87BSbCdZWfzYUfPQVvSjpww2fMLwabneqkKQkAtKyh4Quvb9xL', 'reqId': 1507741799688892} to its replicas since already forwarded 2017-10-11 17:09:59,715 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":null,"digest":"71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12","discarded":1,"ledgerId":2,"reqIdr":[["Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv",1507741799688892]],"ppSeqNo":4,"op":"PREPREPARE","instId":1,"ppTime":1507741799,"viewNo":0,"txnRootHash":null}' to Node1 2017-10-11 17:09:59,715 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":null,"digest":"71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12","discarded":1,"ledgerId":2,"reqIdr":[["Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv",1507741799688892]],"ppSeqNo":4,"op":"PREPREPARE","instId":1,"ppTime":1507741799,"viewNo":0,"txnRootHash":null}' to Node1 2017-10-11 17:09:59,715 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":null,"digest":"71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12","discarded":1,"ledgerId":2,"reqIdr":[["Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv",1507741799688892]],"ppSeqNo":4,"op":"PREPREPARE","instId":1,"ppTime":1507741799,"viewNo":0,"txnRootHash":null}' to Node3 2017-10-11 17:09:59,715 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":null,"digest":"71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12","discarded":1,"ledgerId":2,"reqIdr":[["Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv",1507741799688892]],"ppSeqNo":4,"op":"PREPREPARE","instId":1,"ppTime":1507741799,"viewNo":0,"txnRootHash":null}' to Node3 2017-10-11 17:09:59,716 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":null,"digest":"71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12","discarded":1,"ledgerId":2,"reqIdr":[["Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv",1507741799688892]],"ppSeqNo":4,"op":"PREPREPARE","instId":1,"ppTime":1507741799,"viewNo":0,"txnRootHash":null}' to Node4 2017-10-11 17:09:59,716 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":null,"digest":"71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12","discarded":1,"ledgerId":2,"reqIdr":[["Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv",1507741799688892]],"ppSeqNo":4,"op":"PREPREPARE","instId":1,"ppTime":1507741799,"viewNo":0,"txnRootHash":null}' to Node4 2017-10-11 17:09:59,717 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-10-11 17:09:59,717 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PREPREPARE{'ledgerId': 2, 'reqIdr': [['Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741799688892]], 'ppSeqNo': 4, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'instId': 0, 'ppTime': 1507741799, 'discarded': 1, 'digest': '71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12', 'viewNo': 0} 2017-10-11 17:09:59,718 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'ledgerId': 2, 'digest': '71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12', 'ppSeqNo': 4, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'op': 'PREPREPARE', 'txnRootHash': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'instId': 0, 'ppTime': 1507741799, 'discarded': 1, 'reqIdr': [['Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741799688892]], 'viewNo': 0}, 'Node1') 2017-10-11 17:09:59,718 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPREPARE{'ledgerId': 2, 'reqIdr': [['Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741799688892]], 'ppSeqNo': 4, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'instId': 0, 'ppTime': 1507741799, 'discarded': 1, 'digest': '71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12', 'viewNo': 0} 2017-10-11 17:09:59,719 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv signature on propagate request 1507741799688892 2017-10-11 17:09:59,719 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: PROPAGATE{'senderClient': 'Node1', 'request': {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': 'JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '3vKjs8t62JJURrrFFCfqaWwRQBsj563DHh7Yuy87BSbCdZWfzYUfPQVvSjpww2fMLwabneqkKQkAtKyh4Quvb9xL', 'reqId': 1507741799688892}} 2017-10-11 17:09:59,719 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'senderClient': 'Node1', 'request': {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': 'JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '3vKjs8t62JJURrrFFCfqaWwRQBsj563DHh7Yuy87BSbCdZWfzYUfPQVvSjpww2fMLwabneqkKQkAtKyh4Quvb9xL', 'reqId': 1507741799688892}, 'op': 'PROPAGATE'}, 'Node4') 2017-10-11 17:09:59,719 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': 'Node1', 'request': {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': 'JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '3vKjs8t62JJURrrFFCfqaWwRQBsj563DHh7Yuy87BSbCdZWfzYUfPQVvSjpww2fMLwabneqkKQkAtKyh4Quvb9xL', 'reqId': 1507741799688892}} 2017-10-11 17:09:59,719 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': 'Node1', 'request': {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': 'JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '3vKjs8t62JJURrrFFCfqaWwRQBsj563DHh7Yuy87BSbCdZWfzYUfPQVvSjpww2fMLwabneqkKQkAtKyh4Quvb9xL', 'reqId': 1507741799688892}} 2017-10-11 17:09:59,719 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': 'JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '3vKjs8t62JJURrrFFCfqaWwRQBsj563DHh7Yuy87BSbCdZWfzYUfPQVvSjpww2fMLwabneqkKQkAtKyh4Quvb9xL', 'reqId': 1507741799688892} 2017-10-11 17:09:59,719 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': 'JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '3vKjs8t62JJURrrFFCfqaWwRQBsj563DHh7Yuy87BSbCdZWfzYUfPQVvSjpww2fMLwabneqkKQkAtKyh4Quvb9xL', 'reqId': 1507741799688892} to its replicas since already forwarded 2017-10-11 17:09:59,720 | DEBUG | replica.py ( 823) | processPrePrepare | Node2:0 received PRE-PREPARE(0, 4) from Node1:0 at 2270885.178074619 2017-10-11 17:09:59,721 | DEBUG | replica.py (1034) | validate_pre_prepare | Node2:0 state root before processing PREPREPARE{'ledgerId': 2, 'reqIdr': [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741799688892)], 'ppSeqNo': 4, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'instId': 0, 'ppTime': 1507741799, 'discarded': 1, 'digest': '71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12', 'viewNo': 0} is b'\xbc q\xa4\xde\x84o(W\x02D\x7f%\x89\xdd\x166x\xe0\x97*\x8a\x1b\r(\xb0N\xd5\xc0\x94T\x7f', 4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7 2017-10-11 17:09:59,721 | DEBUG | replica.py ( 934) | doPrepare | Node2:0 Sending PREPARE(0, 4) at 2270885.179085368 2017-10-11 17:09:59,722 | DEBUG | replica.py (2193) | send | Node2:0 sending Prepare 2017-10-11 17:09:59,722 | TRACE | replica.py (2194) | send | Node2:0 sending PREPARE{'digest': '71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12', 'ppSeqNo': 4, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'instId': 0, 'ppTime': 1507741799, 'viewNo': 0} 2017-10-11 17:09:59,722 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': '71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12', 'ppSeqNo': 4, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'instId': 0, 'ppTime': 1507741799, 'viewNo': 0} 2017-10-11 17:09:59,722 | DEBUG | replica.py ( 614) | trackBatches | Node2:0 tracking batch for PREPREPARE{'ledgerId': 2, 'reqIdr': [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741799688892)], 'ppSeqNo': 4, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'instId': 0, 'ppTime': 1507741799, 'discarded': 1, 'digest': '71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12', 'viewNo': 0} with state root b'\xbc q\xa4\xde\x84o(W\x02D\x7f%\x89\xdd\x166x\xe0\x97*\x8a\x1b\r(\xb0N\xd5\xc0\x94T\x7f' 2017-10-11 17:09:59,722 | DEBUG | replica.py ( 842) | processPrePrepare | Node2:0 processed incoming PRE-PREPARE(0, 4) 2017-10-11 17:09:59,722 | DEBUG | node.py (2608) | send | Node2 sending message PREPARE{'digest': '71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12', 'ppSeqNo': 4, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'instId': 0, 'ppTime': 1507741799, 'viewNo': 0} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:09:59,723 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","viewNo":0,"digest":"71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12","ppSeqNo":4,"op":"PREPARE","instId":0,"ppTime":1507741799,"txnRootHash":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu"}' to Node1 2017-10-11 17:09:59,723 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","viewNo":0,"digest":"71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12","ppSeqNo":4,"op":"PREPARE","instId":0,"ppTime":1507741799,"txnRootHash":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu"}' to Node1 2017-10-11 17:09:59,724 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","viewNo":0,"digest":"71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12","ppSeqNo":4,"op":"PREPARE","instId":0,"ppTime":1507741799,"txnRootHash":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu"}' to Node3 2017-10-11 17:09:59,724 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","viewNo":0,"digest":"71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12","ppSeqNo":4,"op":"PREPARE","instId":0,"ppTime":1507741799,"txnRootHash":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu"}' to Node3 2017-10-11 17:09:59,724 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","viewNo":0,"digest":"71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12","ppSeqNo":4,"op":"PREPARE","instId":0,"ppTime":1507741799,"txnRootHash":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu"}' to Node4 2017-10-11 17:09:59,724 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","viewNo":0,"digest":"71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12","ppSeqNo":4,"op":"PREPARE","instId":0,"ppTime":1507741799,"txnRootHash":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu"}' to Node4 2017-10-11 17:09:59,736 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-10-11 17:09:59,737 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: BATCH{'messages': ['{"txnRootHash":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu","ppSeqNo":4,"digest":"71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12","op":"PREPARE","viewNo":0,"ppTime":1507741799,"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","instId":0}', '{"txnRootHash":null,"ppSeqNo":4,"digest":"71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12","op":"PREPARE","viewNo":0,"ppTime":1507741799,"stateRootHash":null,"instId":1}'], 'signature': None} 2017-10-11 17:09:59,737 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"txnRootHash":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu","ppSeqNo":4,"digest":"71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12","op":"PREPARE","viewNo":0,"ppTime":1507741799,"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","instId":0}', '{"txnRootHash":null,"ppSeqNo":4,"digest":"71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12","op":"PREPARE","viewNo":0,"ppTime":1507741799,"stateRootHash":null,"instId":1}'], 'signature': None, 'op': 'BATCH'}, 'Node4') 2017-10-11 17:09:59,737 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"txnRootHash":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu","ppSeqNo":4,"digest":"71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12","op":"PREPARE","viewNo":0,"ppTime":1507741799,"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","instId":0}', '{"txnRootHash":null,"ppSeqNo":4,"digest":"71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12","op":"PREPARE","viewNo":0,"ppTime":1507741799,"stateRootHash":null,"instId":1}'], 'signature': None} 2017-10-11 17:09:59,738 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'digest': '71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12', 'ppSeqNo': 4, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'instId': 0, 'ppTime': 1507741799, 'viewNo': 0} 2017-10-11 17:09:59,738 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'ppSeqNo': 4, 'digest': '71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12', 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'op': 'PREPARE', 'instId': 0, 'ppTime': 1507741799, 'viewNo': 0}, 'Node4') 2017-10-11 17:09:59,738 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'digest': '71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12', 'ppSeqNo': 4, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'instId': 0, 'ppTime': 1507741799, 'viewNo': 0} 2017-10-11 17:09:59,738 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'digest': '71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12', 'ppSeqNo': 4, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741799, 'viewNo': 0} 2017-10-11 17:09:59,738 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'ppSeqNo': 4, 'digest': '71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12', 'stateRootHash': None, 'txnRootHash': None, 'op': 'PREPARE', 'instId': 1, 'ppTime': 1507741799, 'viewNo': 0}, 'Node4') 2017-10-11 17:09:59,738 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'digest': '71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12', 'ppSeqNo': 4, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741799, 'viewNo': 0} 2017-10-11 17:09:59,739 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 0} 2017-10-11 17:09:59,739 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'ppSeqNo': 4, 'op': 'COMMIT', 'viewNo': 0}, 'Node4') 2017-10-11 17:09:59,739 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 0} 2017-10-11 17:09:59,740 | DEBUG | replica.py ( 953) | doCommit | Node2:0 Sending COMMIT(0, 4) at 2270885.197844172 2017-10-11 17:09:59,740 | DEBUG | replica.py (2193) | send | Node2:0 sending Commit 2017-10-11 17:09:59,740 | TRACE | replica.py (2194) | send | Node2:0 sending COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 0} 2017-10-11 17:09:59,741 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 0} commits where f is 1 2017-10-11 17:09:59,741 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (0, 4) 2017-10-11 17:09:59,742 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(0, 4) from Node4:0 2017-10-11 17:09:59,742 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 0} commits where f is 1 2017-10-11 17:09:59,742 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(0, 4) 2017-10-11 17:09:59,742 | DEBUG | replica.py ( 917) | tryCommit | Node2:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': '71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12', 'ppSeqNo': 4, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741799, 'viewNo': 0} 2017-10-11 17:09:59,743 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (0, 4) 2017-10-11 17:09:59,744 | DEBUG | node.py (2608) | send | Node2 sending message COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 0} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:09:59,744 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 3 messages through listener 2017-10-11 17:09:59,745 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: BATCH{'messages': ['{"ppSeqNo":4,"viewNo":0,"instId":0,"op":"PREPARE","stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","ppTime":1507741799,"digest":"71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12","txnRootHash":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu"}', '{"viewNo":0,"instId":0,"op":"COMMIT","ppSeqNo":4}', '{"ppSeqNo":4,"viewNo":0,"instId":1,"op":"PREPARE","stateRootHash":null,"ppTime":1507741799,"digest":"71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12","txnRootHash":null}'], 'signature': None} 2017-10-11 17:09:59,745 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"ppSeqNo":4,"viewNo":0,"instId":0,"op":"PREPARE","stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","ppTime":1507741799,"digest":"71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12","txnRootHash":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu"}', '{"viewNo":0,"instId":0,"op":"COMMIT","ppSeqNo":4}', '{"ppSeqNo":4,"viewNo":0,"instId":1,"op":"PREPARE","stateRootHash":null,"ppTime":1507741799,"digest":"71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12","txnRootHash":null}'], 'signature': None, 'op': 'BATCH'}, 'Node3') 2017-10-11 17:09:59,745 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"ppSeqNo":4,"viewNo":0,"instId":0,"op":"PREPARE","stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","ppTime":1507741799,"digest":"71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12","txnRootHash":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu"}', '{"viewNo":0,"instId":0,"op":"COMMIT","ppSeqNo":4}', '{"ppSeqNo":4,"viewNo":0,"instId":1,"op":"PREPARE","stateRootHash":null,"ppTime":1507741799,"digest":"71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12","txnRootHash":null}'], 'signature': None} 2017-10-11 17:09:59,745 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'digest': '71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12', 'ppSeqNo': 4, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'instId': 0, 'ppTime': 1507741799, 'viewNo': 0} 2017-10-11 17:09:59,745 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'digest': '71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12', 'ppSeqNo': 4, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'op': 'PREPARE', 'txnRootHash': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'instId': 0, 'ppTime': 1507741799, 'viewNo': 0}, 'Node3') 2017-10-11 17:09:59,745 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'digest': '71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12', 'ppSeqNo': 4, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'instId': 0, 'ppTime': 1507741799, 'viewNo': 0} 2017-10-11 17:09:59,745 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 0} 2017-10-11 17:09:59,746 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'ppSeqNo': 4, 'op': 'COMMIT', 'viewNo': 0}, 'Node3') 2017-10-11 17:09:59,746 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 0} 2017-10-11 17:09:59,746 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'digest': '71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12', 'ppSeqNo': 4, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741799, 'viewNo': 0} 2017-10-11 17:09:59,746 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'digest': '71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12', 'ppSeqNo': 4, 'stateRootHash': None, 'op': 'PREPARE', 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741799, 'viewNo': 0}, 'Node3') 2017-10-11 17:09:59,746 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'digest': '71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12', 'ppSeqNo': 4, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741799, 'viewNo': 0} 2017-10-11 17:09:59,746 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PREPARE{'digest': '71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12', 'ppSeqNo': 4, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741799, 'viewNo': 0} 2017-10-11 17:09:59,746 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'digest': '71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12', 'ppSeqNo': 4, 'stateRootHash': None, 'op': 'PREPARE', 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741799, 'viewNo': 0}, 'Node1') 2017-10-11 17:09:59,746 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'digest': '71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12', 'ppSeqNo': 4, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741799, 'viewNo': 0} 2017-10-11 17:09:59,747 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 1, 'ppSeqNo': 4, 'viewNo': 0} 2017-10-11 17:09:59,748 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'ppSeqNo': 4, 'op': 'COMMIT', 'viewNo': 0}, 'Node3') 2017-10-11 17:09:59,748 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'ppSeqNo': 4, 'viewNo': 0} 2017-10-11 17:09:59,748 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":0,"ppSeqNo":4,"op":"COMMIT","viewNo":0}' to Node1 2017-10-11 17:09:59,749 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":0,"ppSeqNo":4,"op":"COMMIT","viewNo":0}' to Node1 2017-10-11 17:09:59,749 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":0,"ppSeqNo":4,"op":"COMMIT","viewNo":0}' to Node3 2017-10-11 17:09:59,749 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":0,"ppSeqNo":4,"op":"COMMIT","viewNo":0}' to Node3 2017-10-11 17:09:59,749 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":0,"ppSeqNo":4,"op":"COMMIT","viewNo":0}' to Node4 2017-10-11 17:09:59,750 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":0,"ppSeqNo":4,"op":"COMMIT","viewNo":0}' to Node4 2017-10-11 17:09:59,751 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since has already sent COMMIT for PREPARE{'digest': '71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12', 'ppSeqNo': 4, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'instId': 0, 'ppTime': 1507741799, 'viewNo': 0} 2017-10-11 17:09:59,751 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (0, 4) 2017-10-11 17:09:59,751 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(0, 4) from Node3:0 2017-10-11 17:09:59,751 | TRACE | replica.py ( 925) | tryOrder | Node2:0 returning request to node 2017-10-11 17:09:59,752 | DEBUG | replica.py (1444) | doOrder | Node2:0 ordering COMMIT (0, 4) 2017-10-11 17:09:59,752 | DEBUG | replica.py ( 323) | last_ordered_3pc | Node2:0 set last ordered as (0, 4) 2017-10-11 17:09:59,752 | DEBUG | replica.py (2193) | send | Node2:0 sending Ordered 2017-10-11 17:09:59,752 | TRACE | replica.py (2194) | send | Node2:0 sending ORDERED{'ledgerId': 2, 'reqIdr': [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741799688892)], 'ppSeqNo': 4, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'instId': 0, 'ppTime': 1507741799, 'viewNo': 0} 2017-10-11 17:09:59,752 | INFO | replica.py (1483) | order_3pc_key | Node2:0 ordered batch request, view no 0, ppSeqNo 4, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu, requests ordered [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741799688892)], discarded [] 2017-10-11 17:09:59,753 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(0, 4) 2017-10-11 17:09:59,753 | DEBUG | replica.py ( 953) | doCommit | Node2:1 Sending COMMIT(0, 4) at 2270885.210624406 2017-10-11 17:09:59,753 | DEBUG | replica.py (2193) | send | Node2:1 sending Commit 2017-10-11 17:09:59,753 | TRACE | replica.py (2194) | send | Node2:1 sending COMMIT{'instId': 1, 'ppSeqNo': 4, 'viewNo': 0} 2017-10-11 17:09:59,753 | DEBUG | replica.py ( 929) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'ppSeqNo': 4, 'viewNo': 0} commits where f is 1 2017-10-11 17:09:59,753 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (0, 4) 2017-10-11 17:09:59,753 | DEBUG | replica.py ( 917) | tryCommit | Node2:1 cannot send COMMIT since has already sent COMMIT for PREPARE{'digest': '71de6d0ae16dd33ae43d9d7ac08e235fc29742f10e0473400f30659958134f12', 'ppSeqNo': 4, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741799, 'viewNo': 0} 2017-10-11 17:09:59,753 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (0, 4) 2017-10-11 17:09:59,754 | DEBUG | replica.py ( 894) | processCommit | Node2:1 received COMMIT(0, 4) from Node3:1 2017-10-11 17:09:59,754 | DEBUG | replica.py ( 929) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'ppSeqNo': 4, 'viewNo': 0} commits where f is 1 2017-10-11 17:09:59,754 | DEBUG | replica.py ( 905) | processCommit | Node2:1 processed incoming COMMIT(0, 4) 2017-10-11 17:09:59,754 | DEBUG | node.py (1847) | processOrdered | Node2 executing Ordered batch 0 4 of 1 requests 2017-10-11 17:09:59,754 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-10-11 17:09:59,755 | DEBUG | node.py (2386) | sendReplyToClient | Node2 sending reply for ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741799688892) to client 2017-10-11 17:09:59,755 | DEBUG | zstack.py ( 744) | transmitThroughListener | Node2C not sending message {'result': {'auditPath': ['4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7'], 'seqNo': 3, 'signature': 'JJhLp5nsb5W7rNHn9RbBUQPsEcgyFynvBs5tRt62GLsoVt4wZZDRtTKM8kX2Mm8d39Xa5L5QVKsjVmmz6ngyYGw', 'type': '110', 'txnTime': 1507741799, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'reqId': 1507741799688892, 'rootHash': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu'}, 'op': 'REPLY'} to b'Node1' 2017-10-11 17:09:59,755 | DEBUG | zstack.py ( 745) | transmitThroughListener | This is a temporary workaround for not being able to disconnect a ROUTER's remote 2017-10-11 17:09:59,755 | INFO | node.py (2290) | executeBatch | Node2 committed batch request, view no 0, ppSeqNo 4, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu, requests: [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741799688892)] 2017-10-11 17:09:59,755 | DEBUG | node.py (2305) | executeBatch | Node2 storing 3PC key (0, 4) for ledger 2 range (3, 3) 2017-10-11 17:09:59,755 | DEBUG | node.py (2608) | send | Node2 sending message COMMIT{'instId': 1, 'ppSeqNo': 4, 'viewNo': 0} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:09:59,756 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":1,"ppSeqNo":4,"op":"COMMIT","viewNo":0}' to Node1 2017-10-11 17:09:59,756 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":1,"ppSeqNo":4,"op":"COMMIT","viewNo":0}' to Node1 2017-10-11 17:09:59,757 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":1,"ppSeqNo":4,"op":"COMMIT","viewNo":0}' to Node3 2017-10-11 17:09:59,757 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":1,"ppSeqNo":4,"op":"COMMIT","viewNo":0}' to Node3 2017-10-11 17:09:59,757 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"instId":1,"ppSeqNo":4,"op":"COMMIT","viewNo":0}' to Node4 2017-10-11 17:09:59,757 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"instId":1,"ppSeqNo":4,"op":"COMMIT","viewNo":0}' to Node4 2017-10-11 17:09:59,758 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:09:59,758 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 1, 'ppSeqNo': 4, 'viewNo': 0} 2017-10-11 17:09:59,759 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'ppSeqNo': 4, 'op': 'COMMIT', 'viewNo': 0}, 'Node4') 2017-10-11 17:09:59,759 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'ppSeqNo': 4, 'viewNo': 0} 2017-10-11 17:09:59,759 | DEBUG | replica.py ( 894) | processCommit | Node2:1 received COMMIT(0, 4) from Node4:1 2017-10-11 17:09:59,759 | TRACE | replica.py ( 925) | tryOrder | Node2:1 returning request to node 2017-10-11 17:09:59,760 | DEBUG | replica.py (1444) | doOrder | Node2:1 ordering COMMIT (0, 4) 2017-10-11 17:09:59,760 | DEBUG | replica.py ( 323) | last_ordered_3pc | Node2:1 set last ordered as (0, 4) 2017-10-11 17:09:59,760 | DEBUG | replica.py (2193) | send | Node2:1 sending Ordered 2017-10-11 17:09:59,760 | TRACE | replica.py (2194) | send | Node2:1 sending ORDERED{'ledgerId': 2, 'reqIdr': [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741799688892)], 'ppSeqNo': 4, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741799, 'viewNo': 0} 2017-10-11 17:09:59,760 | INFO | replica.py (1483) | order_3pc_key | Node2:1 ordered batch request, view no 0, ppSeqNo 4, ledger 2, state root None, txn root None, requests ordered [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741799688892)], discarded [] 2017-10-11 17:09:59,760 | DEBUG | replica.py ( 905) | processCommit | Node2:1 processed incoming COMMIT(0, 4) 2017-10-11 17:09:59,760 | TRACE | node.py (1858) | processOrdered | Node2 got ordered requests from backup replica 1 2017-10-11 17:09:59,761 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:09:59,773 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:09:59,774 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: BATCH{'messages': ['{"ppSeqNo":4,"op":"COMMIT","instId":0,"viewNo":0}', '{"ppSeqNo":4,"op":"COMMIT","instId":1,"viewNo":0}'], 'signature': None} 2017-10-11 17:09:59,774 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"ppSeqNo":4,"op":"COMMIT","instId":0,"viewNo":0}', '{"ppSeqNo":4,"op":"COMMIT","instId":1,"viewNo":0}'], 'op': 'BATCH', 'signature': None}, 'Node1') 2017-10-11 17:09:59,774 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"ppSeqNo":4,"op":"COMMIT","instId":0,"viewNo":0}', '{"ppSeqNo":4,"op":"COMMIT","instId":1,"viewNo":0}'], 'signature': None} 2017-10-11 17:09:59,774 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 0} 2017-10-11 17:09:59,774 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'ppSeqNo': 4, 'op': 'COMMIT', 'viewNo': 0}, 'Node1') 2017-10-11 17:09:59,774 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 0} 2017-10-11 17:09:59,775 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 1, 'ppSeqNo': 4, 'viewNo': 0} 2017-10-11 17:09:59,775 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'ppSeqNo': 4, 'op': 'COMMIT', 'viewNo': 0}, 'Node1') 2017-10-11 17:09:59,775 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'ppSeqNo': 4, 'viewNo': 0} 2017-10-11 17:09:59,776 | TRACE | message_processor.py ( 29) | discard | Node2:0 discarding message COMMIT{'instId': 0, 'ppSeqNo': 4, 'viewNo': 0} because already ordered 3 phase message 2017-10-11 17:09:59,776 | TRACE | message_processor.py ( 29) | discard | Node2:1 discarding message COMMIT{'instId': 1, 'ppSeqNo': 4, 'viewNo': 0} because already ordered 3 phase message 2017-10-11 17:10:06,810 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 6 2017-10-11 17:10:06,811 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:10:06,811 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 7 to run in 60 seconds 2017-10-11 17:10:06,836 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 43 2017-10-11 17:10:06,836 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node2 checking its request amount 2017-10-11 17:10:06,836 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:10:06,836 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 51 to run in 60 seconds 2017-10-11 17:10:06,929 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action dump_json_file with id 44 2017-10-11 17:10:06,930 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action dump_json_file with id 52 to run in 60 seconds 2017-10-11 17:10:07,012 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:10:07,012 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:10:07,012 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:10:07,013 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:10:07,071 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 50 2017-10-11 17:10:07,071 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:10:07,071 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:10:07,071 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.018988576482066 is acceptable. 2017-10-11 17:10:07,071 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:10:07,071 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:10:07,071 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:10:07,071 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 53 to run in 10 seconds 2017-10-11 17:10:17,079 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 53 2017-10-11 17:10:17,080 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:10:17,080 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:10:17,080 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.018988576482066 is acceptable. 2017-10-11 17:10:17,080 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:10:17,080 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:10:17,080 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:10:17,080 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 54 to run in 10 seconds 2017-10-11 17:10:17,349 | TRACE | remote.py ( 119) | hasLostConnection | Remote Node1:HA(host='10.0.0.2', port=9701) has monitor events: [512, 4] 2017-10-11 17:10:17,349 | DEBUG | remote.py ( 123) | hasLostConnection | Node1:HA(host='10.0.0.2', port=9701) found disconnected event on monitor 2017-10-11 17:10:17,350 | DEBUG | keep_in_touch.py ( 68) | conns | Node2's connections changed from {'Node1', 'Node3', 'Node4'} to {'Node3', 'Node4'} 2017-10-11 17:10:17,350 | INFO | keep_in_touch.py ( 93) | _connsChanged | CONNECTION: Node2 disconnected from Node1 2017-10-11 17:10:17,350 | DEBUG | motor.py ( 34) | set_status | Node2 changing status from started to started_hungry 2017-10-11 17:10:17,350 | INFO | node.py ( 823) | onConnsChanged | Node2 lost connection to primary of master 2017-10-11 17:10:17,350 | DEBUG | node.py (2121) | lost_master_primary | Node2 scheduling a view change in 2 sec 2017-10-11 17:10:17,350 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action propose_view_change with id 55 to run in 2 seconds 2017-10-11 17:10:17,350 | DEBUG | node.py ( 941) | checkInstances | Node2 choosing to start election on the basis of count 3 and nodes {'Node3', 'Node4'} 2017-10-11 17:10:19,359 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action propose_view_change with id 55 2017-10-11 17:10:19,359 | DEBUG | throttler.py ( 31) | acquire | now: 2270904.817010284, len(actionsLog): 0 2017-10-11 17:10:19,359 | DEBUG | throttler.py ( 34) | acquire | after trim, len(actionsLog): 0 2017-10-11 17:10:19,359 | DEBUG | throttler.py ( 39) | acquire | len(actionsLog) was 0, after append, len(actionsLog): 1 2017-10-11 17:10:19,359 | INFO | node.py (2048) | sendInstanceChange | VIEW CHANGE: Node2 sending an instance change with view_no 1 since Primary of master protocol instance disconnected 2017-10-11 17:10:19,360 | INFO | node.py (2051) | sendInstanceChange | MONITORING: Node2 metrics for monitor: Node2 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [2270532.253484148, 2270532.253832242] ordered request counts: {0: 4, 1: 4} ordered request durations: {0: 0.299711134750396, 1: 0.3054022225551307} master request latencies: {} client avg request latencies: [{'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': (1, 0.04541354300454259), 'V4SGRU86Z58d6TV7PBUe6f': (3, 0.08476586391528447)}, {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': (1, 0.050638789776712656), 'V4SGRU86Z58d6TV7PBUe6f': (3, 0.08492114425947268)}] throughput: {0: 13.346184162731427, 1: 13.097481631057635} master throughput: 13.346184162731427 total requests: 4 avg backup throughput: 13.097481631057635 master throughput ratio: 1.018988576482066 2017-10-11 17:10:19,360 | DEBUG | node.py (2608) | send | Node2 sending message INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:10:19,360 | DEBUG | node.py (1958) | do_view_change_if_possible | Node2 has no quorum for view 1 2017-10-11 17:10:19,360 | INFO | node.py (2108) | propose_view_change | Node2 sent view change since was disconnected from primary for too long 2017-10-11 17:10:19,360 | DEBUG | node.py (1958) | do_view_change_if_possible | Node2 has no quorum for view 1 2017-10-11 17:10:19,360 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"reason":26,"op":"INSTANCE_CHANGE","viewNo":1}' to Node1 2017-10-11 17:10:19,360 | DEBUG | zstack.py ( 728) | transmit | Remote Node1 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:10:19,360 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"reason":26,"op":"INSTANCE_CHANGE","viewNo":1}' to Node1 2017-10-11 17:10:19,360 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"reason":26,"op":"INSTANCE_CHANGE","viewNo":1}' to Node3 2017-10-11 17:10:19,361 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"reason":26,"op":"INSTANCE_CHANGE","viewNo":1}' to Node3 2017-10-11 17:10:19,361 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"reason":26,"op":"INSTANCE_CHANGE","viewNo":1}' to Node4 2017-10-11 17:10:19,361 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"reason":26,"op":"INSTANCE_CHANGE","viewNo":1}' to Node4 2017-10-11 17:10:19,361 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:10:19,362 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} 2017-10-11 17:10:19,362 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'reason': 26, 'op': 'INSTANCE_CHANGE', 'viewNo': 1}, 'Node3') 2017-10-11 17:10:19,362 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} 2017-10-11 17:10:19,362 | DEBUG | node.py (1915) | processInstanceChange | Node2 received instance change request: INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} from Node3 2017-10-11 17:10:19,362 | INFO | node.py (1954) | do_view_change_if_possible | VIEW CHANGE: Node2 initiating a view change to 1 from 0 2017-10-11 17:10:19,362 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action _check_view_change_completed with id 56 to run in 60 seconds 2017-10-11 17:10:19,362 | DEBUG | replica.py ( 455) | on_view_change_start | Node2:0 setting last prepared for master to (0, 4) 2017-10-11 17:10:19,362 | DEBUG | node.py (2146) | startViewChange | Node2 resetting monitor stats after view change 2017-10-11 17:10:19,363 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-10-11 17:10:19,363 | DEBUG | replica.py ( 408) | primaryName | Node2:0 setting primaryName for view no 1 to: None 2017-10-11 17:10:19,363 | DEBUG | replica.py ( 408) | primaryName | Node2:1 setting primaryName for view no 1 to: None 2017-10-11 17:10:19,363 | INFO | node.py (2159) | startViewChange | VIEW CHANGE: Node2 changed to view 1, will start catchup now 2017-10-11 17:10:19,363 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node2:0 going to remove 0 Ordered messages from outbox 2017-10-11 17:10:19,363 | DEBUG | node.py (1888) | force_process_ordered | Node2 processed 0 Ordered batches for instance 0 before starting catch up 2017-10-11 17:10:19,364 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node2:1 going to remove 0 Ordered messages from outbox 2017-10-11 17:10:19,364 | DEBUG | node.py (1888) | force_process_ordered | Node2 processed 0 Ordered batches for instance 1 before starting catch up 2017-10-11 17:10:19,364 | DEBUG | node.py (2192) | start_catchup | Node2 reverted 0 batches before starting catch up 2017-10-11 17:10:19,364 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node1'] 2017-10-11 17:10:19,364 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node1 for ledger status of ledger 0 2017-10-11 17:10:19,364 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:10:19,364 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node2 for ledger status of ledger 0 2017-10-11 17:10:19,365 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node3'] 2017-10-11 17:10:19,365 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node3 for ledger status of ledger 0 2017-10-11 17:10:19,365 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node4'] 2017-10-11 17:10:19,365 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node4 for ledger status of ledger 0 2017-10-11 17:10:19,365 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-10-11 17:10:19,365 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:10:19,365 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:10:19,365 | DEBUG | node.py (1946) | processInstanceChange | Node2 received instance change message INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} but did not find the master to be slow or has already sent an instance change message 2017-10-11 17:10:19,365 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-10-11 17:10:19,366 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":0}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":0}}']) 2017-10-11 17:10:19,366 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node1: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:10:19,366 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null,"op":"BATCH"}' to Node1 2017-10-11 17:10:19,366 | DEBUG | zstack.py ( 728) | transmit | Remote Node1 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:10:19,366 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-10-11 17:10:19,366 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":0}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":0}}']) 2017-10-11 17:10:19,366 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node3: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:10:19,366 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null,"op":"BATCH"}' to Node3 2017-10-11 17:10:19,367 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node4 into one transmission 2017-10-11 17:10:19,367 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":0}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":0}}']) 2017-10-11 17:10:19,367 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node4: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:10:19,367 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null,"op":"BATCH"}' to Node4 2017-10-11 17:10:19,368 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-10-11 17:10:19,368 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} 2017-10-11 17:10:19,368 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'reason': 26, 'op': 'INSTANCE_CHANGE', 'viewNo': 1}, 'Node4') 2017-10-11 17:10:19,368 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} 2017-10-11 17:10:19,368 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"op":"MESSAGE_REQUEST"}'], 'signature': None} 2017-10-11 17:10:19,370 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"op":"MESSAGE_REQUEST"}'], 'signature': None, 'op': 'BATCH'}, 'Node3') 2017-10-11 17:10:19,370 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"op":"MESSAGE_REQUEST"}'], 'signature': None} 2017-10-11 17:10:19,370 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-10-11 17:10:19,370 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 0}}, 'Node3') 2017-10-11 17:10:19,370 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-10-11 17:10:19,371 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-10-11 17:10:19,371 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 0}}, 'Node3') 2017-10-11 17:10:19,371 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-10-11 17:10:19,371 | DEBUG | node.py (1915) | processInstanceChange | Node2 received instance change request: INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} from Node4 2017-10-11 17:10:19,371 | INFO | message_processor.py ( 29) | discard | Node2 discarding message INSTANCE_CHANGE{'reason': 26, 'viewNo': 1} because Received instance change request with view no 1 which is not more than its view no 1 2017-10-11 17:10:19,371 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node3'] 2017-10-11 17:10:19,372 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node3'] 2017-10-11 17:10:19,372 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-10-11 17:10:19,372 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":0}}', b'{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":0}}']) 2017-10-11 17:10:19,372 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node3: b'{"messages":["{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:10:19,372 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null,"op":"BATCH"}' to Node3 2017-10-11 17:10:19,373 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:10:19,373 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"op":"MESSAGE_REQUEST"}'], 'signature': None} 2017-10-11 17:10:19,374 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"op":"MESSAGE_REQUEST"}'], 'signature': None, 'op': 'BATCH'}, 'Node4') 2017-10-11 17:10:19,374 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"op":"MESSAGE_REQUEST"}'], 'signature': None} 2017-10-11 17:10:19,374 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-10-11 17:10:19,374 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 0}}, 'Node4') 2017-10-11 17:10:19,374 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-10-11 17:10:19,374 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-10-11 17:10:19,374 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 0}}, 'Node4') 2017-10-11 17:10:19,374 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-10-11 17:10:19,375 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node4'] 2017-10-11 17:10:19,375 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} to 1 recipients: ['Node4'] 2017-10-11 17:10:19,375 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node4 into one transmission 2017-10-11 17:10:19,376 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":0}}', b'{"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":0}}']) 2017-10-11 17:10:19,376 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node4: b'{"messages":["{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:10:19,376 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0}}","{\\"msg\\":{\\"ledgerId\\":0,\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ppSeqNo\\":null,\\"txnSeqNo\\":4,\\"viewNo\\":null},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":0}}"],"signature":null,"op":"BATCH"}' to Node4 2017-10-11 17:10:19,376 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:10:19,377 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}'], 'signature': None} 2017-10-11 17:10:19,377 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}'], 'signature': None, 'op': 'BATCH'}, 'Node3') 2017-10-11 17:10:19,377 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null}}'], 'signature': None} 2017-10-11 17:10:19,377 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-10-11 17:10:19,377 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 0}}, 'Node3') 2017-10-11 17:10:19,377 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-10-11 17:10:19,378 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-10-11 17:10:19,378 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 0}}, 'Node3') 2017-10-11 17:10:19,378 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-10-11 17:10:19,378 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4} from Node3 2017-10-11 17:10:19,378 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 17:10:19,378 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 17:10:19,379 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4} from Node3 2017-10-11 17:10:19,379 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 17:10:19,379 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 17:10:19,393 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:10:19,393 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"params":{"ledgerId":0},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"params":{"ledgerId":0},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-10-11 17:10:19,393 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"params":{"ledgerId":0},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"params":{"ledgerId":0},"op":"MESSAGE_RESPONSE"}'], 'signature': None, 'op': 'BATCH'}, 'Node4') 2017-10-11 17:10:19,393 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"params":{"ledgerId":0},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"params":{"ledgerId":0},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-10-11 17:10:19,394 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-10-11 17:10:19,394 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 0}}, 'Node4') 2017-10-11 17:10:19,394 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-10-11 17:10:19,394 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-10-11 17:10:19,394 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 0}}, 'Node4') 2017-10-11 17:10:19,394 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-10-11 17:10:19,395 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4} from Node4 2017-10-11 17:10:19,395 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 17:10:19,395 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 17:10:19,395 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node3', 'Node4'} that its ledger of type 0 is latest 2017-10-11 17:10:19,395 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node2 found from ledger status LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4} that it does not need catchup 2017-10-11 17:10:19,395 | DEBUG | node.py (1479) | preLedgerCatchUp | Node2 going to process any ordered requests before starting catchup. 2017-10-11 17:10:19,396 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node2:0 going to remove 0 Ordered messages from outbox 2017-10-11 17:10:19,396 | DEBUG | node.py (1888) | force_process_ordered | Node2 processed 0 Ordered batches for instance 0 before starting catch up 2017-10-11 17:10:19,396 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node2:1 going to remove 0 Ordered messages from outbox 2017-10-11 17:10:19,396 | DEBUG | node.py (1888) | force_process_ordered | Node2 processed 0 Ordered batches for instance 1 before starting catch up 2017-10-11 17:10:19,396 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-10-11 17:10:19,396 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-10-11 17:10:19,396 | INFO | node.py (1489) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 0 2017-10-11 17:10:19,396 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 17:10:19,396 | DEBUG | node.py ( 941) | checkInstances | Node2 choosing to start election on the basis of count 3 and nodes {'Node3', 'Node4'} 2017-10-11 17:10:19,396 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node1'] 2017-10-11 17:10:19,397 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node1 for ledger status of ledger 2 2017-10-11 17:10:19,397 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:10:19,397 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node2 for ledger status of ledger 2 2017-10-11 17:10:19,397 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-10-11 17:10:19,397 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node3 for ledger status of ledger 2 2017-10-11 17:10:19,397 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-10-11 17:10:19,397 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node4 for ledger status of ledger 2 2017-10-11 17:10:19,397 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node2 going to process 0 stashed ledger statuses for ledger 2 2017-10-11 17:10:19,398 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 0, caught up 0 in total 2017-10-11 17:10:19,398 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4} from Node4 2017-10-11 17:10:19,398 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 17:10:19,398 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 17:10:19,398 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-10-11 17:10:19,399 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}']) 2017-10-11 17:10:19,399 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node1: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:10:19,399 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null,"op":"BATCH"}' to Node1 2017-10-11 17:10:19,399 | DEBUG | zstack.py ( 728) | transmit | Remote Node1 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:10:19,399 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-10-11 17:10:19,399 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}']) 2017-10-11 17:10:19,399 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node3: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:10:19,400 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null,"op":"BATCH"}' to Node3 2017-10-11 17:10:19,400 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node4 into one transmission 2017-10-11 17:10:19,400 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":2}}']) 2017-10-11 17:10:19,400 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node4: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:10:19,400 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null,"op":"BATCH"}' to Node4 2017-10-11 17:10:19,401 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-10-11 17:10:19,402 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}'], 'signature': None} 2017-10-11 17:10:19,402 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}'], 'signature': None, 'op': 'BATCH'}, 'Node3') 2017-10-11 17:10:19,402 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}'], 'signature': None} 2017-10-11 17:10:19,402 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:10:19,402 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}}, 'Node3') 2017-10-11 17:10:19,402 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:10:19,402 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:10:19,404 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}}, 'Node3') 2017-10-11 17:10:19,404 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:10:19,404 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}'], 'signature': None} 2017-10-11 17:10:19,404 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}'], 'signature': None, 'op': 'BATCH'}, 'Node4') 2017-10-11 17:10:19,404 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}'], 'signature': None} 2017-10-11 17:10:19,405 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:10:19,405 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}}, 'Node4') 2017-10-11 17:10:19,405 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:10:19,405 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:10:19,405 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}}, 'Node4') 2017-10-11 17:10:19,406 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:10:19,406 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': 0, 'ppSeqNo': 4, 'txnSeqNo': 3}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-10-11 17:10:19,406 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': 0, 'ppSeqNo': 4, 'txnSeqNo': 3}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node3'] 2017-10-11 17:10:19,407 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': 0, 'ppSeqNo': 4, 'txnSeqNo': 3}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-10-11 17:10:19,407 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': 0, 'ppSeqNo': 4, 'txnSeqNo': 3}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node4'] 2017-10-11 17:10:19,407 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-10-11 17:10:19,407 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":2,"merkleRoot":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu","ppSeqNo":4,"txnSeqNo":3,"viewNo":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":2,"merkleRoot":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu","ppSeqNo":4,"txnSeqNo":3,"viewNo":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}']) 2017-10-11 17:10:19,408 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node3: b'{"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu\\",\\"ppSeqNo\\":4,\\"txnSeqNo\\":3,\\"viewNo\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu\\",\\"ppSeqNo\\":4,\\"txnSeqNo\\":3,\\"viewNo\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:10:19,408 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu\\",\\"ppSeqNo\\":4,\\"txnSeqNo\\":3,\\"viewNo\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu\\",\\"ppSeqNo\\":4,\\"txnSeqNo\\":3,\\"viewNo\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null,"op":"BATCH"}' to Node3 2017-10-11 17:10:19,408 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node4 into one transmission 2017-10-11 17:10:19,408 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":2,"merkleRoot":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu","ppSeqNo":4,"txnSeqNo":3,"viewNo":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":2,"merkleRoot":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu","ppSeqNo":4,"txnSeqNo":3,"viewNo":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}']) 2017-10-11 17:10:19,408 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node4: b'{"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu\\",\\"ppSeqNo\\":4,\\"txnSeqNo\\":3,\\"viewNo\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu\\",\\"ppSeqNo\\":4,\\"txnSeqNo\\":3,\\"viewNo\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:10:19,408 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu\\",\\"ppSeqNo\\":4,\\"txnSeqNo\\":3,\\"viewNo\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu\\",\\"ppSeqNo\\":4,\\"txnSeqNo\\":3,\\"viewNo\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}"],"signature":null,"op":"BATCH"}' to Node4 2017-10-11 17:10:19,409 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:10:19,409 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu","ppSeqNo":4,"txnSeqNo":3,"viewNo":0},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu","ppSeqNo":4,"txnSeqNo":3,"viewNo":0},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-10-11 17:10:19,409 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu","ppSeqNo":4,"txnSeqNo":3,"viewNo":0},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu","ppSeqNo":4,"txnSeqNo":3,"viewNo":0},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}'], 'signature': None, 'op': 'BATCH'}, 'Node4') 2017-10-11 17:10:19,409 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu","ppSeqNo":4,"txnSeqNo":3,"viewNo":0},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":2,"merkleRoot":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu","ppSeqNo":4,"txnSeqNo":3,"viewNo":0},"params":{"ledgerId":2},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-10-11 17:10:19,410 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_RESPONSE{'msg': {'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': 0, 'ppSeqNo': 4, 'txnSeqNo': 3}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:10:19,410 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': 0, 'ppSeqNo': 4, 'txnSeqNo': 3}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}}, 'Node4') 2017-10-11 17:10:19,410 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': 0, 'ppSeqNo': 4, 'txnSeqNo': 3}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:10:19,410 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_RESPONSE{'msg': {'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': 0, 'ppSeqNo': 4, 'txnSeqNo': 3}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:10:19,410 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': 0, 'ppSeqNo': 4, 'txnSeqNo': 3}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}}, 'Node4') 2017-10-11 17:10:19,410 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': 0, 'ppSeqNo': 4, 'txnSeqNo': 3}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:10:19,411 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': 0, 'ppSeqNo': 4, 'txnSeqNo': 3} from Node4 2017-10-11 17:10:19,411 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 3 with 3 2017-10-11 17:10:19,411 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 3 with 3 2017-10-11 17:10:19,411 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': 0, 'ppSeqNo': 4, 'txnSeqNo': 3} from Node4 2017-10-11 17:10:19,411 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 3 with 3 2017-10-11 17:10:19,412 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 3 with 3 2017-10-11 17:10:19,423 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:10:19,424 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu","ppSeqNo":4,"txnSeqNo":3,"viewNo":0}}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu","ppSeqNo":4,"txnSeqNo":3,"viewNo":0}}'], 'signature': None} 2017-10-11 17:10:19,424 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu","ppSeqNo":4,"txnSeqNo":3,"viewNo":0}}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu","ppSeqNo":4,"txnSeqNo":3,"viewNo":0}}'], 'signature': None, 'op': 'BATCH'}, 'Node3') 2017-10-11 17:10:19,424 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu","ppSeqNo":4,"txnSeqNo":3,"viewNo":0}}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":2,"merkleRoot":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu","ppSeqNo":4,"txnSeqNo":3,"viewNo":0}}'], 'signature': None} 2017-10-11 17:10:19,424 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg': {'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': 0, 'ppSeqNo': 4, 'txnSeqNo': 3}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:10:19,425 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': 0, 'ppSeqNo': 4, 'txnSeqNo': 3}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}}, 'Node3') 2017-10-11 17:10:19,425 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': 0, 'ppSeqNo': 4, 'txnSeqNo': 3}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:10:19,425 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg': {'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': 0, 'ppSeqNo': 4, 'txnSeqNo': 3}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:10:19,425 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': 0, 'ppSeqNo': 4, 'txnSeqNo': 3}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}}, 'Node3') 2017-10-11 17:10:19,425 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': 0, 'ppSeqNo': 4, 'txnSeqNo': 3}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:10:19,425 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': 0, 'ppSeqNo': 4, 'txnSeqNo': 3} from Node3 2017-10-11 17:10:19,425 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 3 with 3 2017-10-11 17:10:19,427 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 3 with 3 2017-10-11 17:10:19,427 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node3', 'Node4'} that its ledger of type 2 is latest 2017-10-11 17:10:19,427 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node2 found from ledger status LEDGER_STATUS{'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': 0, 'ppSeqNo': 4, 'txnSeqNo': 3} that it does not need catchup 2017-10-11 17:10:19,427 | DEBUG | node.py (1479) | preLedgerCatchUp | Node2 going to process any ordered requests before starting catchup. 2017-10-11 17:10:19,427 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node2:0 going to remove 0 Ordered messages from outbox 2017-10-11 17:10:19,427 | DEBUG | node.py (1888) | force_process_ordered | Node2 processed 0 Ordered batches for instance 0 before starting catch up 2017-10-11 17:10:19,427 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node2:1 going to remove 0 Ordered messages from outbox 2017-10-11 17:10:19,427 | DEBUG | node.py (1888) | force_process_ordered | Node2 processed 0 Ordered batches for instance 1 before starting catch up 2017-10-11 17:10:19,428 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-10-11 17:10:19,428 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-10-11 17:10:19,428 | INFO | node.py (1489) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 2 2017-10-11 17:10:19,428 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 3 with 3 2017-10-11 17:10:19,428 | DEBUG | pool_config.py ( 35) | processLedger | processing config ledger for any POOL_CONFIGs 2017-10-11 17:10:19,428 | DEBUG | upgrader.py ( 208) | processLedger | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb processing config ledger for any upgrades 2017-10-11 17:10:19,428 | INFO | upgrader.py ( 214) | processLedger | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb found upgrade START txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade40'), ('reinstall', False), ('reqId', 1507741607555952), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-10-11T17:25:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-10-11T17:15:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-10-11T17:20:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-10-11T17:10:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i'), ('timeout', 10), ('txnTime', 1507741607), ('type', '109'), ('version', '1.1.40'), ('seqNo', 2)]) 2017-10-11 17:10:19,428 | INFO | upgrader.py ( 292) | handleUpgradeTxn | Node 'Node2' handles upgrade txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade40'), ('reinstall', False), ('reqId', 1507741607555952), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-10-11T17:25:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-10-11T17:15:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-10-11T17:20:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-10-11T17:10:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', '3ACoow9Gjp7Ta2XUW2CBYqytPQ7LThnAp9b8cwdCvd3bFHPWry15xbHSeGLjUWMfRRfPm4X98q9ccGtqCutqj1i'), ('timeout', 10), ('txnTime', 1507741607), ('type', '109'), ('version', '1.1.40'), ('seqNo', 2)]) 2017-10-11 17:10:19,429 | INFO | upgrader.py ( 321) | handleUpgradeTxn | Node 'Node2' schedules upgrade to 1.1.40 2017-10-11 17:10:19,429 | INFO | upgrader.py ( 326) | handleUpgradeTxn | Node 'Node2' cancels previous upgrade and schedules a new one to 1.1.40 2017-10-11 17:10:19,429 | INFO | upgrader.py ( 390) | _cancelScheduledUpgrade | Cancelling upgrade of node 'Node2' to version 1.1.40 due to some reason 2017-10-11 17:10:19,429 | INFO | upgrader.py ( 359) | _scheduleUpgrade | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb's upgrader processing upgrade for version 1.1.40 2017-10-11 17:10:19,429 | TRACE | has_action_queue.py ( 36) | _schedule | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb scheduling action partial(_callUpgradeAgent) with id 2 to run in 280 seconds 2017-10-11 17:10:19,429 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node1'] 2017-10-11 17:10:19,430 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node1 for ledger status of ledger 1 2017-10-11 17:10:19,430 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:10:19,430 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node2 for ledger status of ledger 1 2017-10-11 17:10:19,430 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-10-11 17:10:19,430 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node3 for ledger status of ledger 1 2017-10-11 17:10:19,430 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-10-11 17:10:19,430 | DEBUG | node.py ( 855) | _ask_for_ledger_status | Node2 asking Node4 for ledger status of ledger 1 2017-10-11 17:10:19,430 | DEBUG | ledger_manager.py (1027) | processStashedLedgerStatuses | Node2 going to process 0 stashed ledger statuses for ledger 1 2017-10-11 17:10:19,431 | INFO | upgrader.py ( 150) | should_notify_about_upgrade_result | Node's 'Node2' last upgrade txn is None 2017-10-11 17:10:19,431 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 2, caught up 0 in total 2017-10-11 17:10:19,431 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': 0, 'ppSeqNo': 4, 'txnSeqNo': 3} from Node3 2017-10-11 17:10:19,431 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 3 with 3 2017-10-11 17:10:19,432 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 3 with 3 2017-10-11 17:10:19,432 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-10-11 17:10:19,432 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}']) 2017-10-11 17:10:19,432 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node1: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:10:19,432 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null,"op":"BATCH"}' to Node1 2017-10-11 17:10:19,433 | DEBUG | zstack.py ( 728) | transmit | Remote Node1 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:10:19,433 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-10-11 17:10:19,433 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}']) 2017-10-11 17:10:19,433 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node3: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:10:19,433 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null,"op":"BATCH"}' to Node3 2017-10-11 17:10:19,433 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node4 into one transmission 2017-10-11 17:10:19,433 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}', b'{"msg_type":"LEDGER_STATUS","op":"MESSAGE_REQUEST","params":{"ledgerId":1}}']) 2017-10-11 17:10:19,434 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node4: b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:10:19,434 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_REQUEST\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null,"op":"BATCH"}' to Node4 2017-10-11 17:10:19,435 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:10:19,435 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}'], 'signature': None} 2017-10-11 17:10:19,439 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}'], 'signature': None, 'op': 'BATCH'}, 'Node3') 2017-10-11 17:10:19,439 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}'], 'signature': None} 2017-10-11 17:10:19,439 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:10:19,439 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}}, 'Node3') 2017-10-11 17:10:19,440 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:10:19,440 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:10:19,440 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}}, 'Node3') 2017-10-11 17:10:19,440 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:10:19,440 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'ledgerId': 1, 'viewNo': 0, 'ppSeqNo': 2, 'txnSeqNo': 13}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-10-11 17:10:19,441 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'ledgerId': 1, 'viewNo': 0, 'ppSeqNo': 2, 'txnSeqNo': 13}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node3'] 2017-10-11 17:10:19,441 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-10-11 17:10:19,441 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":1,"merkleRoot":"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn","ppSeqNo":2,"txnSeqNo":13,"viewNo":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', b'{"msg":{"ledgerId":1,"merkleRoot":"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn","ppSeqNo":2,"txnSeqNo":13,"viewNo":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}']) 2017-10-11 17:10:19,441 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node3: b'{"messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn\\",\\"ppSeqNo\\":2,\\"txnSeqNo\\":13,\\"viewNo\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn\\",\\"ppSeqNo\\":2,\\"txnSeqNo\\":13,\\"viewNo\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:10:19,442 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn\\",\\"ppSeqNo\\":2,\\"txnSeqNo\\":13,\\"viewNo\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn\\",\\"ppSeqNo\\":2,\\"txnSeqNo\\":13,\\"viewNo\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null,"op":"BATCH"}' to Node3 2017-10-11 17:10:19,442 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-10-11 17:10:19,443 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}'], 'signature': None} 2017-10-11 17:10:19,443 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}'], 'signature': None, 'op': 'BATCH'}, 'Node4') 2017-10-11 17:10:19,443 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}'], 'signature': None} 2017-10-11 17:10:19,443 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:10:19,443 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}}, 'Node4') 2017-10-11 17:10:19,443 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:10:19,444 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:10:19,444 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}}, 'Node4') 2017-10-11 17:10:19,444 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:10:19,444 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn","ppSeqNo":2,"txnSeqNo":13,"viewNo":0},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn","ppSeqNo":2,"txnSeqNo":13,"viewNo":0},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-10-11 17:10:19,444 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn","ppSeqNo":2,"txnSeqNo":13,"viewNo":0},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn","ppSeqNo":2,"txnSeqNo":13,"viewNo":0},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}'], 'signature': None, 'op': 'BATCH'}, 'Node4') 2017-10-11 17:10:19,444 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn","ppSeqNo":2,"txnSeqNo":13,"viewNo":0},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","msg":{"ledgerId":1,"merkleRoot":"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn","ppSeqNo":2,"txnSeqNo":13,"viewNo":0},"params":{"ledgerId":1},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-10-11 17:10:19,444 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'ledgerId': 1, 'viewNo': 0, 'ppSeqNo': 2, 'txnSeqNo': 13}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:10:19,444 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'ledgerId': 1, 'viewNo': 0, 'ppSeqNo': 2, 'txnSeqNo': 13}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}}, 'Node4') 2017-10-11 17:10:19,444 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'ledgerId': 1, 'viewNo': 0, 'ppSeqNo': 2, 'txnSeqNo': 13}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:10:19,445 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'ledgerId': 1, 'viewNo': 0, 'ppSeqNo': 2, 'txnSeqNo': 13}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:10:19,445 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'ledgerId': 1, 'viewNo': 0, 'ppSeqNo': 2, 'txnSeqNo': 13}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}}, 'Node4') 2017-10-11 17:10:19,445 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'ledgerId': 1, 'viewNo': 0, 'ppSeqNo': 2, 'txnSeqNo': 13}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:10:19,445 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'ledgerId': 1, 'viewNo': 0, 'ppSeqNo': 2, 'txnSeqNo': 13}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-10-11 17:10:19,445 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'ledgerId': 1, 'viewNo': 0, 'ppSeqNo': 2, 'txnSeqNo': 13}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node4'] 2017-10-11 17:10:19,446 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'ledgerId': 1, 'viewNo': 0, 'ppSeqNo': 2, 'txnSeqNo': 13} from Node4 2017-10-11 17:10:19,446 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 13 with 13 2017-10-11 17:10:19,446 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 13 with 13 2017-10-11 17:10:19,446 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'ledgerId': 1, 'viewNo': 0, 'ppSeqNo': 2, 'txnSeqNo': 13} from Node4 2017-10-11 17:10:19,446 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 13 with 13 2017-10-11 17:10:19,446 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 13 with 13 2017-10-11 17:10:19,447 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node4 into one transmission 2017-10-11 17:10:19,447 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":1,"merkleRoot":"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn","ppSeqNo":2,"txnSeqNo":13,"viewNo":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', b'{"msg":{"ledgerId":1,"merkleRoot":"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn","ppSeqNo":2,"txnSeqNo":13,"viewNo":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}']) 2017-10-11 17:10:19,447 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node4: b'{"messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn\\",\\"ppSeqNo\\":2,\\"txnSeqNo\\":13,\\"viewNo\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn\\",\\"ppSeqNo\\":2,\\"txnSeqNo\\":13,\\"viewNo\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:10:19,447 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn\\",\\"ppSeqNo\\":2,\\"txnSeqNo\\":13,\\"viewNo\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn\\",\\"ppSeqNo\\":2,\\"txnSeqNo\\":13,\\"viewNo\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}"],"signature":null,"op":"BATCH"}' to Node4 2017-10-11 17:10:19,448 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:10:19,448 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn","ppSeqNo":2,"txnSeqNo":13,"viewNo":0}}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn","ppSeqNo":2,"txnSeqNo":13,"viewNo":0}}'], 'signature': None} 2017-10-11 17:10:19,448 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn","ppSeqNo":2,"txnSeqNo":13,"viewNo":0}}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn","ppSeqNo":2,"txnSeqNo":13,"viewNo":0}}'], 'signature': None, 'op': 'BATCH'}, 'Node3') 2017-10-11 17:10:19,449 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn","ppSeqNo":2,"txnSeqNo":13,"viewNo":0}}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_RESPONSE","msg":{"ledgerId":1,"merkleRoot":"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn","ppSeqNo":2,"txnSeqNo":13,"viewNo":0}}'], 'signature': None} 2017-10-11 17:10:19,449 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'ledgerId': 1, 'viewNo': 0, 'ppSeqNo': 2, 'txnSeqNo': 13}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:10:19,449 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'ledgerId': 1, 'viewNo': 0, 'ppSeqNo': 2, 'txnSeqNo': 13}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}}, 'Node3') 2017-10-11 17:10:19,449 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'ledgerId': 1, 'viewNo': 0, 'ppSeqNo': 2, 'txnSeqNo': 13}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:10:19,449 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'ledgerId': 1, 'viewNo': 0, 'ppSeqNo': 2, 'txnSeqNo': 13}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:10:19,450 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'ledgerId': 1, 'viewNo': 0, 'ppSeqNo': 2, 'txnSeqNo': 13}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}}, 'Node3') 2017-10-11 17:10:19,450 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'ledgerId': 1, 'viewNo': 0, 'ppSeqNo': 2, 'txnSeqNo': 13}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:10:19,450 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'ledgerId': 1, 'viewNo': 0, 'ppSeqNo': 2, 'txnSeqNo': 13} from Node3 2017-10-11 17:10:19,450 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 13 with 13 2017-10-11 17:10:19,450 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 13 with 13 2017-10-11 17:10:19,450 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node3', 'Node4'} that its ledger of type 1 is latest 2017-10-11 17:10:19,450 | DEBUG | ledger_manager.py ( 315) | processLedgerStatus | Node2 found from ledger status LEDGER_STATUS{'merkleRoot': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'ledgerId': 1, 'viewNo': 0, 'ppSeqNo': 2, 'txnSeqNo': 13} that it does not need catchup 2017-10-11 17:10:19,450 | DEBUG | node.py (1479) | preLedgerCatchUp | Node2 going to process any ordered requests before starting catchup. 2017-10-11 17:10:19,450 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node2:0 going to remove 0 Ordered messages from outbox 2017-10-11 17:10:19,451 | DEBUG | node.py (1888) | force_process_ordered | Node2 processed 0 Ordered batches for instance 0 before starting catch up 2017-10-11 17:10:19,451 | DEBUG | replica.py (2259) | _remove_ordered_from_queue | Node2:1 going to remove 0 Ordered messages from outbox 2017-10-11 17:10:19,451 | DEBUG | node.py (1888) | force_process_ordered | Node2 processed 0 Ordered batches for instance 1 before starting catch up 2017-10-11 17:10:19,451 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-10-11 17:10:19,451 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-10-11 17:10:19,451 | INFO | node.py (1489) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 1 2017-10-11 17:10:19,451 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 13 with 13 2017-10-11 17:10:19,451 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 1, caught up 0 in total 2017-10-11 17:10:19,451 | DEBUG | node.py (1609) | num_txns_caught_up_in_last_catchup | Node2 caught up to 0 txns in the last catchup 2017-10-11 17:10:19,452 | DEBUG | node.py (2469) | processStashedOrderedReqs | Node2 processed 0 stashed ordered requests 2017-10-11 17:10:19,452 | DEBUG | monitor.py ( 192) | reset | Node2's Monitor being reset 2017-10-11 17:10:19,452 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node2 needs 3 ViewChangeDone messages 2017-10-11 17:10:19,452 | DEBUG | node.py (1568) | caught_up_for_current_view | Node2 does not have view change quorum for view 1 2017-10-11 17:10:19,452 | DEBUG | node.py (1552) | is_catchup_needed | Node2 is not caught up for the current view 1 2017-10-11 17:10:19,452 | DEBUG | node.py (1609) | num_txns_caught_up_in_last_catchup | Node2 caught up to 0 txns in the last catchup 2017-10-11 17:10:19,452 | DEBUG | node.py (1557) | is_catchup_needed | Node2 ordered till last prepared certificate 2017-10-11 17:10:19,452 | INFO | node.py (1537) | allLedgersCaughtUp | CATCH-UP: Node2 does not need any more catchups 2017-10-11 17:10:19,453 | DEBUG | primary_decider.py ( 134) | send | Node2's elector sending VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 13, 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn'), (2, 3, '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu')], 'viewNo': 1} 2017-10-11 17:10:19,453 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node2 needs 2 ViewChangeDone messages 2017-10-11 17:10:19,453 | DEBUG | primary_selector.py ( 266) | _startSelection | Node2 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-10-11 17:10:19,453 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'ledgerId': 1, 'viewNo': 0, 'ppSeqNo': 2, 'txnSeqNo': 13} from Node3 2017-10-11 17:10:19,453 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 13 with 13 2017-10-11 17:10:19,453 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 13 with 13 2017-10-11 17:10:19,454 | DEBUG | node.py (2608) | send | Node2 sending message VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 13, 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn'), (2, 3, '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu')], 'viewNo': 1} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:10:19,454 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"name":"Node2","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn"],[2,3,"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu"]],"op":"VIEW_CHANGE_DONE","viewNo":1}' to Node1 2017-10-11 17:10:19,454 | DEBUG | zstack.py ( 728) | transmit | Remote Node1 is not connected - message will not be sent immediately.If this problem does not resolve itself - check your firewall settings 2017-10-11 17:10:19,454 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"name":"Node2","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn"],[2,3,"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu"]],"op":"VIEW_CHANGE_DONE","viewNo":1}' to Node1 2017-10-11 17:10:19,454 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"name":"Node2","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn"],[2,3,"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu"]],"op":"VIEW_CHANGE_DONE","viewNo":1}' to Node3 2017-10-11 17:10:19,455 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"name":"Node2","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn"],[2,3,"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu"]],"op":"VIEW_CHANGE_DONE","viewNo":1}' to Node3 2017-10-11 17:10:19,455 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"name":"Node2","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn"],[2,3,"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu"]],"op":"VIEW_CHANGE_DONE","viewNo":1}' to Node4 2017-10-11 17:10:19,455 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"name":"Node2","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn"],[2,3,"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu"]],"op":"VIEW_CHANGE_DONE","viewNo":1}' to Node4 2017-10-11 17:10:19,456 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:10:19,456 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn'], [2, 3, '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu']], 'viewNo': 1} 2017-10-11 17:10:19,456 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'name': 'Node2', 'op': 'VIEW_CHANGE_DONE', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn'], [2, 3, '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu']], 'viewNo': 1}, 'Node3') 2017-10-11 17:10:19,456 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn'], [2, 3, '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu']], 'viewNo': 1} 2017-10-11 17:10:19,457 | DEBUG | node.py (1205) | sendToElector | Node2 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn'], [2, 3, '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu']], 'viewNo': 1}, 'Node3') 2017-10-11 17:10:19,457 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node2's primary selector started processing of ViewChangeDone msg from Node3 : VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn'], [2, 3, '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu']], 'viewNo': 1} 2017-10-11 17:10:19,457 | DEBUG | primary_selector.py ( 190) | _hasViewChangeQuorum | Node2 needs 1 ViewChangeDone messages 2017-10-11 17:10:19,457 | DEBUG | primary_selector.py ( 266) | _startSelection | Node2 cannot start primary selection found failure in primary verification. This can happen due to lack of appropriate ViewChangeDone messages 2017-10-11 17:10:19,469 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:10:19,470 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn'], [2, 3, '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu']], 'viewNo': 1} 2017-10-11 17:10:19,470 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'name': 'Node2', 'op': 'VIEW_CHANGE_DONE', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn'], [2, 3, '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu']], 'viewNo': 1}, 'Node4') 2017-10-11 17:10:19,471 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn'], [2, 3, '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu']], 'viewNo': 1} 2017-10-11 17:10:19,471 | DEBUG | node.py (1205) | sendToElector | Node2 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn'], [2, 3, '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu']], 'viewNo': 1}, 'Node4') 2017-10-11 17:10:19,471 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node2's primary selector started processing of ViewChangeDone msg from Node4 : VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn'], [2, 3, '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu']], 'viewNo': 1} 2017-10-11 17:10:19,471 | DEBUG | primary_selector.py ( 196) | _hasViewChangeQuorum | Node2 got view change quorum (3 >= 3) 2017-10-11 17:10:19,471 | DEBUG | primary_selector.py ( 215) | has_view_change_from_primary | Node2 received ViewChangeDone from primary Node2 2017-10-11 17:10:19,471 | DEBUG | primary_selector.py ( 238) | has_sufficient_same_view_change_done_messages | Node2 found acceptable primary Node2 and ledger info ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 13, 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn'), (2, 3, '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu')) 2017-10-11 17:10:19,472 | DEBUG | primary_selector.py ( 281) | _startSelection | Node2 starting selection 2017-10-11 17:10:19,472 | DISPLAY | primary_selector.py ( 291) | _startSelection | PRIMARY SELECTION: Node2:0 selected primary Node2:0 for instance 0 (view 1) 2017-10-11 17:10:19,472 | INFO | node.py ( 510) | start_participating | Node2 started participating 2017-10-11 17:10:19,472 | DEBUG | replica.py ( 408) | primaryName | Node2:0 setting primaryName for view no 1 to: Node2:0 2017-10-11 17:10:19,472 | DEBUG | replica.py (1694) | _gc | Node2:0 cleaning up till (0, 4) 2017-10-11 17:10:19,472 | DEBUG | replica.py (1709) | _gc | Node2:0 found 4 3-phase keys to clean 2017-10-11 17:10:19,472 | DEBUG | replica.py (1711) | _gc | Node2:0 found 4 request keys to clean 2017-10-11 17:10:19,472 | DEBUG | replica.py ( 313) | h | Node2:0 set watermarks as 0 300 2017-10-11 17:10:19,473 | DISPLAY | primary_selector.py ( 315) | _startSelection | VIEW CHANGE: Node2:0 declares view change 1 as completed for instance 0, new primary is Node2:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 13, 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn'), (2, 3, '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu')] 2017-10-11 17:10:19,473 | DISPLAY | primary_selector.py ( 291) | _startSelection | PRIMARY SELECTION: Node2:1 selected primary Node3:1 for instance 1 (view 1) 2017-10-11 17:10:19,473 | DEBUG | replica.py ( 408) | primaryName | Node2:1 setting primaryName for view no 1 to: Node3:1 2017-10-11 17:10:19,473 | DEBUG | replica.py (1694) | _gc | Node2:1 cleaning up till (0, 4) 2017-10-11 17:10:19,473 | DEBUG | replica.py (1709) | _gc | Node2:1 found 4 3-phase keys to clean 2017-10-11 17:10:19,473 | DEBUG | replica.py (1711) | _gc | Node2:1 found 4 request keys to clean 2017-10-11 17:10:19,473 | DEBUG | replica.py (1733) | _gc | Node2:1 clearing request ('V4SGRU86Z58d6TV7PBUe6f', 1507741607555952) from previous checkpoints 2017-10-11 17:10:19,473 | DEBUG | replica.py (1733) | _gc | Node2:1 clearing request ('V4SGRU86Z58d6TV7PBUe6f', 1507741503995435) from previous checkpoints 2017-10-11 17:10:19,474 | DEBUG | replica.py (1733) | _gc | Node2:1 clearing request ('V4SGRU86Z58d6TV7PBUe6f', 1507741500981867) from previous checkpoints 2017-10-11 17:10:19,474 | DEBUG | replica.py (1733) | _gc | Node2:1 clearing request ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741799688892) from previous checkpoints 2017-10-11 17:10:19,474 | DEBUG | replica.py ( 313) | h | Node2:1 set watermarks as 0 300 2017-10-11 17:10:19,474 | DEBUG | replica.py ( 514) | _setup_for_non_master | Node2:1 Setting last ordered for non-master as (0, 4) 2017-10-11 17:10:19,474 | DEBUG | replica.py ( 323) | last_ordered_3pc | Node2:1 set last ordered as (1, 0) 2017-10-11 17:10:19,474 | DISPLAY | primary_selector.py ( 315) | _startSelection | VIEW CHANGE: Node2:1 declares view change 1 as completed for instance 1, new primary is Node3:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 13, 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn'), (2, 3, '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu')] 2017-10-11 17:10:22,018 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:10:22,019 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:10:22,019 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:10:22,019 | DEBUG | zstack.py ( 645) | sendPingPong | Node2 pinged Node1 2017-10-11 17:10:22,019 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:10:22,031 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'pi' to Node1 2017-10-11 17:10:22,031 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'pi' to Node1 2017-10-11 17:10:27,091 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 54 2017-10-11 17:10:27,091 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:10:27,091 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:10:27,091 | DEBUG | monitor.py ( 335) | isMasterThroughputTooLow | Node2 master throughput is not measurable. 2017-10-11 17:10:27,091 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:10:27,091 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:10:27,092 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:10:27,092 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 57 to run in 10 seconds 2017-10-11 17:10:28,322 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:10:28,323 | DEBUG | zstack.py ( 663) | handlePingPong | Node2 got ping from Node1 2017-10-11 17:10:28,323 | DEBUG | zstack.py ( 645) | sendPingPong | Node2 ponged Node1 2017-10-11 17:10:28,323 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'po' to Node1 2017-10-11 17:10:28,323 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'po' to Node1 2017-10-11 17:10:28,426 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-10-11 17:10:28,427 | DEBUG | zstack.py ( 669) | handlePingPong | Node2 got pong from Node1 2017-10-11 17:10:28,427 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu","ppSeqNo":null,"txnSeqNo":3,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu","ppSeqNo":null,"txnSeqNo":3,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr","ppSeqNo":null,"txnSeqNo":13,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr","ppSeqNo":null,"txnSeqNo":13,"viewNo":null},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-10-11 17:10:28,427 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu","ppSeqNo":null,"txnSeqNo":3,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu","ppSeqNo":null,"txnSeqNo":3,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr","ppSeqNo":null,"txnSeqNo":13,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr","ppSeqNo":null,"txnSeqNo":13,"viewNo":null},"op":"MESSAGE_RESPONSE"}'], 'op': 'BATCH', 'signature': None}, 'Node1') 2017-10-11 17:10:28,428 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":0},"msg":{"ledgerId":0,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ppSeqNo":null,"txnSeqNo":4,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu","ppSeqNo":null,"txnSeqNo":3,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"msg":{"ledgerId":2,"merkleRoot":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu","ppSeqNo":null,"txnSeqNo":3,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr","ppSeqNo":null,"txnSeqNo":13,"viewNo":null},"op":"MESSAGE_RESPONSE"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"msg":{"ledgerId":1,"merkleRoot":"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr","ppSeqNo":null,"txnSeqNo":13,"viewNo":null},"op":"MESSAGE_RESPONSE"}'], 'signature': None} 2017-10-11 17:10:28,428 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-10-11 17:10:28,428 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 0}}, 'Node1') 2017-10-11 17:10:28,428 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-10-11 17:10:28,428 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-10-11 17:10:28,428 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 0}}, 'Node1') 2017-10-11 17:10:28,429 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 0}} 2017-10-11 17:10:28,429 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg': {'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 3}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:10:28,429 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 3}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}}, 'Node1') 2017-10-11 17:10:28,429 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 3}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:10:28,429 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg': {'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 3}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:10:28,429 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 3}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 2}}, 'Node1') 2017-10-11 17:10:28,430 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 3}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:10:28,430 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 13}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:10:28,430 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 13}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}}, 'Node1') 2017-10-11 17:10:28,430 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 13}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:10:28,431 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_RESPONSE{'msg': {'merkleRoot': 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 13}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:10:28,431 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg': {'merkleRoot': 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 13}, 'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_RESPONSE', 'params': {'ledgerId': 1}}, 'Node1') 2017-10-11 17:10:28,432 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_RESPONSE{'msg': {'merkleRoot': 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 13}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:10:28,432 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: BATCH{'messages': ['{"viewNo":1,"primary":[],"op":"CURRENT_STATE"}', '{"viewNo":null,"txnSeqNo":4,"ppSeqNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ledgerId":0,"op":"LEDGER_STATUS"}'], 'signature': None} 2017-10-11 17:10:28,432 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"viewNo":1,"primary":[],"op":"CURRENT_STATE"}', '{"viewNo":null,"txnSeqNo":4,"ppSeqNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ledgerId":0,"op":"LEDGER_STATUS"}'], 'op': 'BATCH', 'signature': None}, 'Node1') 2017-10-11 17:10:28,432 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"viewNo":1,"primary":[],"op":"CURRENT_STATE"}', '{"viewNo":null,"txnSeqNo":4,"ppSeqNo":null,"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ledgerId":0,"op":"LEDGER_STATUS"}'], 'signature': None} 2017-10-11 17:10:28,432 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: CURRENT_STATE{'primary': [], 'viewNo': 1} 2017-10-11 17:10:28,433 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'primary': [], 'op': 'CURRENT_STATE', 'viewNo': 1}, 'Node1') 2017-10-11 17:10:28,433 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox CURRENT_STATE{'primary': [], 'viewNo': 1} 2017-10-11 17:10:28,433 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4} 2017-10-11 17:10:28,433 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'txnSeqNo': 4, 'ppSeqNo': None, 'op': 'LEDGER_STATUS', 'ledgerId': 0, 'viewNo': None}, 'Node1') 2017-10-11 17:10:28,433 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4} 2017-10-11 17:10:28,434 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4} from Node1 2017-10-11 17:10:28,434 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 17:10:28,434 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 17:10:28,434 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node1', 'Node4'} that its ledger of type 0 is latest 2017-10-11 17:10:28,434 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4} from Node1 2017-10-11 17:10:28,435 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 17:10:28,435 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 17:10:28,435 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node1', 'Node4'} that its ledger of type 0 is latest 2017-10-11 17:10:28,435 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 3} from Node1 2017-10-11 17:10:28,435 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 3 with 3 2017-10-11 17:10:28,435 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 3 with 3 2017-10-11 17:10:28,436 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node1', 'Node3'} that its ledger of type 2 is latest 2017-10-11 17:10:28,436 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 3} from Node1 2017-10-11 17:10:28,436 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 3 with 3 2017-10-11 17:10:28,436 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 2 of size 3 with 3 2017-10-11 17:10:28,436 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node1', 'Node3'} that its ledger of type 2 is latest 2017-10-11 17:10:28,437 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 13} from Node1 2017-10-11 17:10:28,437 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 13 with 13 2017-10-11 17:10:28,437 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 13 with 13 2017-10-11 17:10:28,437 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node1', 'Node3'} that its ledger of type 1 is latest 2017-10-11 17:10:28,437 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr', 'ledgerId': 1, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 13} from Node1 2017-10-11 17:10:28,438 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 13 with 13 2017-10-11 17:10:28,438 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 1 of size 13 with 13 2017-10-11 17:10:28,438 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node1', 'Node3'} that its ledger of type 1 is latest 2017-10-11 17:10:28,439 | DEBUG | node.py ( 907) | process_current_state_message | Node2 processing current state CURRENT_STATE{'primary': [], 'viewNo': 1} from Node1 2017-10-11 17:10:28,440 | DEBUG | ledger_manager.py ( 246) | processLedgerStatus | Node2 received ledger status: LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4} from Node1 2017-10-11 17:10:28,441 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 17:10:28,441 | DEBUG | ledger_manager.py ( 984) | _compareLedger | Node2 comparing its ledger 0 of size 4 with 4 2017-10-11 17:10:28,441 | DEBUG | ledger_manager.py ( 312) | processLedgerStatus | Node2 found out from {'Node1', 'Node4'} that its ledger of type 0 is latest 2017-10-11 17:10:28,443 | TRACE | remote.py ( 119) | hasLostConnection | Remote Node1:HA(host='10.0.0.2', port=9701) has monitor events: [2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 128, 4, 2, 1] 2017-10-11 17:10:28,443 | DEBUG | remote.py ( 123) | hasLostConnection | Node1:HA(host='10.0.0.2', port=9701) found disconnected event on monitor 2017-10-11 17:10:28,443 | DEBUG | keep_in_touch.py ( 68) | conns | Node2's connections changed from {'Node3', 'Node4'} to {'Node1', 'Node3', 'Node4'} 2017-10-11 17:10:28,443 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node2 now connected to Node1 2017-10-11 17:10:28,443 | DEBUG | motor.py ( 34) | set_status | Node2 changing status from started_hungry to started 2017-10-11 17:10:28,444 | DEBUG | node.py ( 941) | checkInstances | Node2 choosing to start election on the basis of count 4 and nodes {'Node1', 'Node3', 'Node4'} 2017-10-11 17:10:28,444 | DEBUG | node.py ( 902) | send_current_state_to_lagging_node | Node2 sending current state CURRENT_STATE{'primary': [VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 13, 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn'), (2, 3, '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu')), 'viewNo': 1}], 'viewNo': 1} to lagged node Node1 2017-10-11 17:10:28,444 | DEBUG | node.py (2608) | send | Node2 sending message CURRENT_STATE{'primary': [VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': ((0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 13, 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn'), (2, 3, '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu')), 'viewNo': 1}], 'viewNo': 1} to 1 recipients: ['Node1'] 2017-10-11 17:10:28,444 | DEBUG | node.py (2608) | send | Node2 sending message LEDGER_STATUS{'merkleRoot': '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3', 'ledgerId': 0, 'viewNo': None, 'ppSeqNo': None, 'txnSeqNo': 4} to 1 recipients: ['Node1'] 2017-10-11 17:10:28,445 | DEBUG | node.py (2608) | send | Node2 sending message LEDGER_STATUS{'merkleRoot': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'ledgerId': 1, 'viewNo': 0, 'ppSeqNo': 2, 'txnSeqNo': 13} to 1 recipients: ['Node1'] 2017-10-11 17:10:28,445 | DEBUG | node.py (2608) | send | Node2 sending message LEDGER_STATUS{'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': 0, 'ppSeqNo': 4, 'txnSeqNo': 3} to 1 recipients: ['Node1'] 2017-10-11 17:10:28,445 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 4 msgs to Node1 into one transmission 2017-10-11 17:10:28,446 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"primary":[{"ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn"],[2,3,"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu"]],"name":"Node2","viewNo":1}],"op":"CURRENT_STATE","viewNo":1}', b'{"merkleRoot":"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3","ledgerId":0,"ppSeqNo":null,"op":"LEDGER_STATUS","txnSeqNo":4,"viewNo":null}', b'{"merkleRoot":"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn","ledgerId":1,"ppSeqNo":2,"op":"LEDGER_STATUS","txnSeqNo":13,"viewNo":0}', b'{"merkleRoot":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu","ledgerId":2,"ppSeqNo":4,"op":"LEDGER_STATUS","txnSeqNo":3,"viewNo":0}']) 2017-10-11 17:10:28,446 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node1: b'{"messages":["{\\"primary\\":[{\\"ledgerInfo\\":[[0,4,\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\"],[1,13,\\"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn\\"],[2,3,\\"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu\\"]],\\"name\\":\\"Node2\\",\\"viewNo\\":1}],\\"op\\":\\"CURRENT_STATE\\",\\"viewNo\\":1}","{\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ledgerId\\":0,\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"txnSeqNo\\":4,\\"viewNo\\":null}","{\\"merkleRoot\\":\\"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn\\",\\"ledgerId\\":1,\\"ppSeqNo\\":2,\\"op\\":\\"LEDGER_STATUS\\",\\"txnSeqNo\\":13,\\"viewNo\\":0}","{\\"merkleRoot\\":\\"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu\\",\\"ledgerId\\":2,\\"ppSeqNo\\":4,\\"op\\":\\"LEDGER_STATUS\\",\\"txnSeqNo\\":3,\\"viewNo\\":0}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:10:28,446 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"primary\\":[{\\"ledgerInfo\\":[[0,4,\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\"],[1,13,\\"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn\\"],[2,3,\\"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu\\"]],\\"name\\":\\"Node2\\",\\"viewNo\\":1}],\\"op\\":\\"CURRENT_STATE\\",\\"viewNo\\":1}","{\\"merkleRoot\\":\\"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3\\",\\"ledgerId\\":0,\\"ppSeqNo\\":null,\\"op\\":\\"LEDGER_STATUS\\",\\"txnSeqNo\\":4,\\"viewNo\\":null}","{\\"merkleRoot\\":\\"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn\\",\\"ledgerId\\":1,\\"ppSeqNo\\":2,\\"op\\":\\"LEDGER_STATUS\\",\\"txnSeqNo\\":13,\\"viewNo\\":0}","{\\"merkleRoot\\":\\"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu\\",\\"ledgerId\\":2,\\"ppSeqNo\\":4,\\"op\\":\\"LEDGER_STATUS\\",\\"txnSeqNo\\":3,\\"viewNo\\":0}"],"signature":null,"op":"BATCH"}' to Node1 2017-10-11 17:10:28,486 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:10:28,486 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"reqId":1507741828468646,"signature":"4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm","operation":{"type":"110","data":{"action":"complete","version":"1.1.40"},"signature":"4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ"},"identifier":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv"}', '{"viewNo":1,"name":"Node2","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr"],[2,3,"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu"]],"op":"VIEW_CHANGE_DONE"}'], 'signature': None} 2017-10-11 17:10:28,486 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"reqId":1507741828468646,"signature":"4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm","operation":{"type":"110","data":{"action":"complete","version":"1.1.40"},"signature":"4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ"},"identifier":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv"}', '{"viewNo":1,"name":"Node2","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr"],[2,3,"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu"]],"op":"VIEW_CHANGE_DONE"}'], 'op': 'BATCH', 'signature': None}, 'Node1') 2017-10-11 17:10:28,487 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":2},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"msg_type":"LEDGER_STATUS","params":{"ledgerId":1},"op":"MESSAGE_REQUEST"}', '{"reqId":1507741828468646,"signature":"4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm","operation":{"type":"110","data":{"action":"complete","version":"1.1.40"},"signature":"4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ"},"identifier":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv"}', '{"viewNo":1,"name":"Node2","ledgerInfo":[[0,4,"5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3"],[1,13,"B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr"],[2,3,"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu"]],"op":"VIEW_CHANGE_DONE"}'], 'signature': None} 2017-10-11 17:10:28,487 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:10:28,487 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}}, 'Node1') 2017-10-11 17:10:28,487 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:10:28,492 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:10:28,493 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 2}}, 'Node1') 2017-10-11 17:10:28,493 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} 2017-10-11 17:10:28,494 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:10:28,494 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}}, 'Node1') 2017-10-11 17:10:28,494 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:10:28,495 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:10:28,495 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'msg_type': 'LEDGER_STATUS', 'op': 'MESSAGE_REQUEST', 'params': {'ledgerId': 1}}, 'Node1') 2017-10-11 17:10:28,495 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox MESSAGE_REQUEST{'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} 2017-10-11 17:10:28,495 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'operation': {'type': '110', 'data': {'action': 'complete', 'version': '1.1.40'}, 'signature': '4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm', 'reqId': 1507741828468646}, 'Node1') 2017-10-11 17:10:28,495 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox SafeRequest: {'operation': {'type': '110', 'data': {'action': 'complete', 'version': '1.1.40'}, 'signature': '4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm', 'reqId': 1507741828468646} 2017-10-11 17:10:28,496 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 3, '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu']], 'viewNo': 1} 2017-10-11 17:10:28,496 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'name': 'Node2', 'op': 'VIEW_CHANGE_DONE', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 3, '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu']], 'viewNo': 1}, 'Node1') 2017-10-11 17:10:28,496 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 3, '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu']], 'viewNo': 1} 2017-10-11 17:10:28,496 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': 0, 'ppSeqNo': 4, 'txnSeqNo': 3}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node1'] 2017-10-11 17:10:28,496 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu', 'ledgerId': 2, 'viewNo': 0, 'ppSeqNo': 4, 'txnSeqNo': 3}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 2}} to 1 recipients: ['Node1'] 2017-10-11 17:10:28,497 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'ledgerId': 1, 'viewNo': 0, 'ppSeqNo': 2, 'txnSeqNo': 13}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node1'] 2017-10-11 17:10:28,497 | DEBUG | node.py (2608) | send | Node2 sending message MESSAGE_RESPONSE{'msg': LEDGER_STATUS{'merkleRoot': 'DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn', 'ledgerId': 1, 'viewNo': 0, 'ppSeqNo': 2, 'txnSeqNo': 13}, 'msg_type': 'LEDGER_STATUS', 'params': {'ledgerId': 1}} to 1 recipients: ['Node1'] 2017-10-11 17:10:28,498 | TRACE | propagator.py ( 165) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'type': '110', 'data': {'action': 'complete', 'version': '1.1.40'}, 'signature': '4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm', 'reqId': 1507741828468646} 2017-10-11 17:10:28,498 | INFO | propagator.py ( 148) | propagate | Node2 propagating request ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741828468646) from client Node1 2017-10-11 17:10:28,498 | DEBUG | node.py (2608) | send | Node2 sending message PROPAGATE{'senderClient': 'Node1', 'request': {'operation': {'type': '110', 'data': {'action': 'complete', 'version': '1.1.40'}, 'signature': '4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm', 'reqId': 1507741828468646}} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:10:28,498 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'type': '110', 'data': {'action': 'complete', 'version': '1.1.40'}, 'signature': '4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm', 'reqId': 1507741828468646} to its replicas since not finalised 2017-10-11 17:10:28,499 | DEBUG | node.py (1205) | sendToElector | Node2 sending message to elector: (VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 3, '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu']], 'viewNo': 1}, 'Node1') 2017-10-11 17:10:28,499 | DEBUG | primary_selector.py ( 106) | _processViewChangeDoneMessage | Node2's primary selector started processing of ViewChangeDone msg from Node1 : VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 3, '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu']], 'viewNo': 1} 2017-10-11 17:10:28,499 | DEBUG | message_processor.py ( 29) | discard | Node2 discarding message VIEW_CHANGE_DONE{'name': 'Node2', 'ledgerInfo': [[0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'], [1, 13, 'B2WPnH16xNgc2or78LRvBb8FmmgB9gx15yB3ww4u36Mr'], [2, 3, '7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu']], 'viewNo': 1} because it already decided primary which is Node2:0 2017-10-11 17:10:28,499 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 5 msgs to Node1 into one transmission 2017-10-11 17:10:28,499 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"msg":{"ledgerId":2,"merkleRoot":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu","ppSeqNo":4,"txnSeqNo":3,"viewNo":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":2,"merkleRoot":"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu","ppSeqNo":4,"txnSeqNo":3,"viewNo":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":2}}', b'{"msg":{"ledgerId":1,"merkleRoot":"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn","ppSeqNo":2,"txnSeqNo":13,"viewNo":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', b'{"msg":{"ledgerId":1,"merkleRoot":"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn","ppSeqNo":2,"txnSeqNo":13,"viewNo":0},"msg_type":"LEDGER_STATUS","op":"MESSAGE_RESPONSE","params":{"ledgerId":1}}', b'{"senderClient":"Node1","request":{"operation":{"type":"110","data":{"action":"complete","version":"1.1.40"},"signature":"4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ"},"identifier":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv","signature":"4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm","reqId":1507741828468646},"op":"PROPAGATE"}']) 2017-10-11 17:10:28,500 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node1: b'{"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu\\",\\"ppSeqNo\\":4,\\"txnSeqNo\\":3,\\"viewNo\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu\\",\\"ppSeqNo\\":4,\\"txnSeqNo\\":3,\\"viewNo\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn\\",\\"ppSeqNo\\":2,\\"txnSeqNo\\":13,\\"viewNo\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn\\",\\"ppSeqNo\\":2,\\"txnSeqNo\\":13,\\"viewNo\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"senderClient\\":\\"Node1\\",\\"request\\":{\\"operation\\":{\\"type\\":\\"110\\",\\"data\\":{\\"action\\":\\"complete\\",\\"version\\":\\"1.1.40\\"},\\"signature\\":\\"4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ\\"},\\"identifier\\":\\"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv\\",\\"signature\\":\\"4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm\\",\\"reqId\\":1507741828468646},\\"op\\":\\"PROPAGATE\\"}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:10:28,500 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu\\",\\"ppSeqNo\\":4,\\"txnSeqNo\\":3,\\"viewNo\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":2,\\"merkleRoot\\":\\"7243ccvhfvHyHtPipBrTmwRrpp4LdNT2V6zeEuFUZsXu\\",\\"ppSeqNo\\":4,\\"txnSeqNo\\":3,\\"viewNo\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":2}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn\\",\\"ppSeqNo\\":2,\\"txnSeqNo\\":13,\\"viewNo\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"msg\\":{\\"ledgerId\\":1,\\"merkleRoot\\":\\"DQdu5vrv4a9kqbELDnF7SiXYcktCHTAQV786DE6hKrxn\\",\\"ppSeqNo\\":2,\\"txnSeqNo\\":13,\\"viewNo\\":0},\\"msg_type\\":\\"LEDGER_STATUS\\",\\"op\\":\\"MESSAGE_RESPONSE\\",\\"params\\":{\\"ledgerId\\":1}}","{\\"senderClient\\":\\"Node1\\",\\"request\\":{\\"operation\\":{\\"type\\":\\"110\\",\\"data\\":{\\"action\\":\\"complete\\",\\"version\\":\\"1.1.40\\"},\\"signature\\":\\"4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ\\"},\\"identifier\\":\\"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv\\",\\"signature\\":\\"4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm\\",\\"reqId\\":1507741828468646},\\"op\\":\\"PROPAGATE\\"}"],"signature":null,"op":"BATCH"}' to Node1 2017-10-11 17:10:28,501 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"senderClient":"Node1","request":{"operation":{"type":"110","data":{"action":"complete","version":"1.1.40"},"signature":"4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ"},"identifier":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv","signature":"4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm","reqId":1507741828468646},"op":"PROPAGATE"}' to Node3 2017-10-11 17:10:28,501 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"senderClient":"Node1","request":{"operation":{"type":"110","data":{"action":"complete","version":"1.1.40"},"signature":"4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ"},"identifier":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv","signature":"4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm","reqId":1507741828468646},"op":"PROPAGATE"}' to Node3 2017-10-11 17:10:28,501 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"senderClient":"Node1","request":{"operation":{"type":"110","data":{"action":"complete","version":"1.1.40"},"signature":"4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ"},"identifier":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv","signature":"4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm","reqId":1507741828468646},"op":"PROPAGATE"}' to Node4 2017-10-11 17:10:28,501 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"senderClient":"Node1","request":{"operation":{"type":"110","data":{"action":"complete","version":"1.1.40"},"signature":"4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ"},"identifier":"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv","signature":"4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm","reqId":1507741828468646},"op":"PROPAGATE"}' to Node4 2017-10-11 17:10:28,502 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-10-11 17:10:28,503 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv signature on propagate request 1507741828468646 2017-10-11 17:10:28,503 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PROPAGATE{'senderClient': 'Node1', 'request': {'operation': {'type': '110', 'data': {'action': 'complete', 'version': '1.1.40'}, 'signature': '4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm', 'reqId': 1507741828468646}} 2017-10-11 17:10:28,503 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'senderClient': 'Node1', 'request': {'operation': {'type': '110', 'data': {'action': 'complete', 'version': '1.1.40'}, 'signature': '4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm', 'reqId': 1507741828468646}, 'op': 'PROPAGATE'}, 'Node3') 2017-10-11 17:10:28,503 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': 'Node1', 'request': {'operation': {'type': '110', 'data': {'action': 'complete', 'version': '1.1.40'}, 'signature': '4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm', 'reqId': 1507741828468646}} 2017-10-11 17:10:28,504 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv signature on propagate request 1507741828468646 2017-10-11 17:10:28,504 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: PROPAGATE{'senderClient': 'Node1', 'request': {'operation': {'type': '110', 'data': {'action': 'complete', 'version': '1.1.40'}, 'signature': '4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm', 'reqId': 1507741828468646}} 2017-10-11 17:10:28,504 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'senderClient': 'Node1', 'request': {'operation': {'type': '110', 'data': {'action': 'complete', 'version': '1.1.40'}, 'signature': '4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm', 'reqId': 1507741828468646}, 'op': 'PROPAGATE'}, 'Node4') 2017-10-11 17:10:28,504 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': 'Node1', 'request': {'operation': {'type': '110', 'data': {'action': 'complete', 'version': '1.1.40'}, 'signature': '4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm', 'reqId': 1507741828468646}} 2017-10-11 17:10:28,504 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': 'Node1', 'request': {'operation': {'type': '110', 'data': {'action': 'complete', 'version': '1.1.40'}, 'signature': '4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm', 'reqId': 1507741828468646}} 2017-10-11 17:10:28,506 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'type': '110', 'data': {'action': 'complete', 'version': '1.1.40'}, 'signature': '4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm', 'reqId': 1507741828468646} 2017-10-11 17:10:28,507 | DEBUG | propagator.py ( 214) | forward | Node2 forwarding request ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741828468646) to 0 replicas 2017-10-11 17:10:28,507 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': 'Node1', 'request': {'operation': {'type': '110', 'data': {'action': 'complete', 'version': '1.1.40'}, 'signature': '4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm', 'reqId': 1507741828468646}} 2017-10-11 17:10:28,508 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'type': '110', 'data': {'action': 'complete', 'version': '1.1.40'}, 'signature': '4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm', 'reqId': 1507741828468646} 2017-10-11 17:10:28,508 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'type': '110', 'data': {'action': 'complete', 'version': '1.1.40'}, 'signature': '4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm', 'reqId': 1507741828468646} to its replicas since already forwarded 2017-10-11 17:10:28,509 | DEBUG | replica.py ( 667) | create3PCBatch | Node2:0 creating batch 1 for ledger 2 with state root b'\xbc q\xa4\xde\x84o(W\x02D\x7f%\x89\xdd\x166x\xe0\x97*\x8a\x1b\r(\xb0N\xd5\xc0\x94T\x7f' 2017-10-11 17:10:28,509 | DEBUG | replica.py ( 700) | create3PCBatch | Node2:0 created a PRE-PREPARE with 1 requests for ledger 2 2017-10-11 17:10:28,509 | DEBUG | replica.py (2193) | send | Node2:0 sending PrePrepare 2017-10-11 17:10:28,510 | TRACE | replica.py (2194) | send | Node2:0 sending PREPREPARE{'ledgerId': 2, 'reqIdr': [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741828468646)], 'ppSeqNo': 1, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': 'Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw', 'instId': 0, 'ppTime': 1507741828, 'discarded': 1, 'digest': '9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171', 'viewNo': 1} 2017-10-11 17:10:28,510 | DEBUG | replica.py ( 614) | trackBatches | Node2:0 tracking batch for PREPREPARE{'ledgerId': 2, 'reqIdr': [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741828468646)], 'ppSeqNo': 1, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': 'Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw', 'instId': 0, 'ppTime': 1507741828, 'discarded': 1, 'digest': '9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171', 'viewNo': 1} with state root b'\xbc q\xa4\xde\x84o(W\x02D\x7f%\x89\xdd\x166x\xe0\x97*\x8a\x1b\r(\xb0N\xd5\xc0\x94T\x7f' 2017-10-11 17:10:28,510 | DEBUG | node.py (2608) | send | Node2 sending message PREPREPARE{'ledgerId': 2, 'reqIdr': [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741828468646)], 'ppSeqNo': 1, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': 'Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw', 'instId': 0, 'ppTime': 1507741828, 'discarded': 1, 'digest': '9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171', 'viewNo': 1} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:10:28,511 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","digest":"9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171","discarded":1,"ledgerId":2,"reqIdr":[["Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv",1507741828468646]],"ppSeqNo":1,"op":"PREPREPARE","instId":0,"ppTime":1507741828,"viewNo":1,"txnRootHash":"Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw"}' to Node1 2017-10-11 17:10:28,511 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","digest":"9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171","discarded":1,"ledgerId":2,"reqIdr":[["Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv",1507741828468646]],"ppSeqNo":1,"op":"PREPREPARE","instId":0,"ppTime":1507741828,"viewNo":1,"txnRootHash":"Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw"}' to Node1 2017-10-11 17:10:28,511 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","digest":"9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171","discarded":1,"ledgerId":2,"reqIdr":[["Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv",1507741828468646]],"ppSeqNo":1,"op":"PREPREPARE","instId":0,"ppTime":1507741828,"viewNo":1,"txnRootHash":"Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw"}' to Node3 2017-10-11 17:10:28,512 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","digest":"9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171","discarded":1,"ledgerId":2,"reqIdr":[["Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv",1507741828468646]],"ppSeqNo":1,"op":"PREPREPARE","instId":0,"ppTime":1507741828,"viewNo":1,"txnRootHash":"Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw"}' to Node3 2017-10-11 17:10:28,512 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","digest":"9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171","discarded":1,"ledgerId":2,"reqIdr":[["Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv",1507741828468646]],"ppSeqNo":1,"op":"PREPREPARE","instId":0,"ppTime":1507741828,"viewNo":1,"txnRootHash":"Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw"}' to Node4 2017-10-11 17:10:28,512 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","digest":"9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171","discarded":1,"ledgerId":2,"reqIdr":[["Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv",1507741828468646]],"ppSeqNo":1,"op":"PREPREPARE","instId":0,"ppTime":1507741828,"viewNo":1,"txnRootHash":"Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw"}' to Node4 2017-10-11 17:10:28,513 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:10:28,514 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv signature on propagate request 1507741828468646 2017-10-11 17:10:28,514 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PROPAGATE{'senderClient': 'Node1', 'request': {'operation': {'type': '110', 'data': {'action': 'complete', 'version': '1.1.40'}, 'signature': '4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm', 'reqId': 1507741828468646}} 2017-10-11 17:10:28,514 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'senderClient': 'Node1', 'request': {'operation': {'type': '110', 'data': {'action': 'complete', 'version': '1.1.40'}, 'signature': '4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm', 'reqId': 1507741828468646}, 'op': 'PROPAGATE'}, 'Node1') 2017-10-11 17:10:28,514 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': 'Node1', 'request': {'operation': {'type': '110', 'data': {'action': 'complete', 'version': '1.1.40'}, 'signature': '4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm', 'reqId': 1507741828468646}} 2017-10-11 17:10:28,514 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': 'Node1', 'request': {'operation': {'type': '110', 'data': {'action': 'complete', 'version': '1.1.40'}, 'signature': '4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm', 'reqId': 1507741828468646}} 2017-10-11 17:10:28,515 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'type': '110', 'data': {'action': 'complete', 'version': '1.1.40'}, 'signature': '4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm', 'reqId': 1507741828468646} 2017-10-11 17:10:28,515 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'type': '110', 'data': {'action': 'complete', 'version': '1.1.40'}, 'signature': '4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ'}, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'signature': '4ed6Uw3oKPy8UVB8K3Pk9TiLjkjeREjycjhCnefmMzdjV8A6fZnsP91D5RWoFiq3iz3if4FjosezWWk11JPJstgm', 'reqId': 1507741828468646} to its replicas since already forwarded 2017-10-11 17:10:28,516 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:10:28,517 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PREPREPARE{'ledgerId': 2, 'reqIdr': [['Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741828468646]], 'ppSeqNo': 1, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741828, 'discarded': 1, 'digest': '9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171', 'viewNo': 1} 2017-10-11 17:10:28,517 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'ledgerId': 2, 'reqIdr': [['Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741828468646]], 'stateRootHash': None, 'ppSeqNo': 1, 'op': 'PREPREPARE', 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741828, 'discarded': 1, 'digest': '9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171', 'viewNo': 1}, 'Node3') 2017-10-11 17:10:28,517 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPREPARE{'ledgerId': 2, 'reqIdr': [['Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741828468646]], 'ppSeqNo': 1, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741828, 'discarded': 1, 'digest': '9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171', 'viewNo': 1} 2017-10-11 17:10:28,518 | DEBUG | replica.py ( 823) | processPrePrepare | Node2:1 received PRE-PREPARE(1, 1) from Node3:1 at 2270913.975498228 2017-10-11 17:10:28,518 | DEBUG | replica.py ( 934) | doPrepare | Node2:1 Sending PREPARE(1, 1) at 2270913.975795908 2017-10-11 17:10:28,518 | DEBUG | replica.py (2193) | send | Node2:1 sending Prepare 2017-10-11 17:10:28,518 | TRACE | replica.py (2194) | send | Node2:1 sending PREPARE{'digest': '9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171', 'ppSeqNo': 1, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741828, 'viewNo': 1} 2017-10-11 17:10:28,518 | DEBUG | replica.py ( 917) | tryCommit | Node2:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': '9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171', 'ppSeqNo': 1, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741828, 'viewNo': 1} 2017-10-11 17:10:28,519 | DEBUG | replica.py ( 614) | trackBatches | Node2:1 tracking batch for PREPREPARE{'ledgerId': 2, 'reqIdr': [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741828468646)], 'ppSeqNo': 1, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741828, 'discarded': 1, 'digest': '9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171', 'viewNo': 1} with state root None 2017-10-11 17:10:28,519 | DEBUG | replica.py ( 842) | processPrePrepare | Node2:1 processed incoming PRE-PREPARE(1, 1) 2017-10-11 17:10:28,519 | DEBUG | node.py (2608) | send | Node2 sending message PREPARE{'digest': '9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171', 'ppSeqNo': 1, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741828, 'viewNo': 1} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:10:28,520 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":null,"viewNo":1,"digest":"9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171","ppSeqNo":1,"op":"PREPARE","instId":1,"ppTime":1507741828,"txnRootHash":null}' to Node1 2017-10-11 17:10:28,520 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":null,"viewNo":1,"digest":"9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171","ppSeqNo":1,"op":"PREPARE","instId":1,"ppTime":1507741828,"txnRootHash":null}' to Node1 2017-10-11 17:10:28,520 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":null,"viewNo":1,"digest":"9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171","ppSeqNo":1,"op":"PREPARE","instId":1,"ppTime":1507741828,"txnRootHash":null}' to Node3 2017-10-11 17:10:28,521 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":null,"viewNo":1,"digest":"9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171","ppSeqNo":1,"op":"PREPARE","instId":1,"ppTime":1507741828,"txnRootHash":null}' to Node3 2017-10-11 17:10:28,521 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":null,"viewNo":1,"digest":"9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171","ppSeqNo":1,"op":"PREPARE","instId":1,"ppTime":1507741828,"txnRootHash":null}' to Node4 2017-10-11 17:10:28,522 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":null,"viewNo":1,"digest":"9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171","ppSeqNo":1,"op":"PREPARE","instId":1,"ppTime":1507741828,"txnRootHash":null}' to Node4 2017-10-11 17:10:28,524 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:10:28,524 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'digest': '9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171', 'ppSeqNo': 1, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': 'Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw', 'instId': 0, 'ppTime': 1507741828, 'viewNo': 1} 2017-10-11 17:10:28,525 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'digest': '9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171', 'ppSeqNo': 1, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'op': 'PREPARE', 'txnRootHash': 'Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw', 'instId': 0, 'ppTime': 1507741828, 'viewNo': 1}, 'Node3') 2017-10-11 17:10:28,525 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'digest': '9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171', 'ppSeqNo': 1, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': 'Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw', 'instId': 0, 'ppTime': 1507741828, 'viewNo': 1} 2017-10-11 17:10:28,525 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': '9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171', 'ppSeqNo': 1, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': 'Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw', 'instId': 0, 'ppTime': 1507741828, 'viewNo': 1} 2017-10-11 17:10:28,526 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (1, 1) 2017-10-11 17:10:28,539 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-10-11 17:10:28,541 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: BATCH{'messages': ['{"txnRootHash":"Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw","ppSeqNo":1,"digest":"9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171","op":"PREPARE","viewNo":1,"ppTime":1507741828,"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","instId":0}', '{"op":"COMMIT","instId":0,"viewNo":1,"ppSeqNo":1}', '{"txnRootHash":null,"ppSeqNo":1,"digest":"9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171","op":"PREPARE","viewNo":1,"ppTime":1507741828,"stateRootHash":null,"instId":1}'], 'signature': None} 2017-10-11 17:10:28,541 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"txnRootHash":"Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw","ppSeqNo":1,"digest":"9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171","op":"PREPARE","viewNo":1,"ppTime":1507741828,"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","instId":0}', '{"op":"COMMIT","instId":0,"viewNo":1,"ppSeqNo":1}', '{"txnRootHash":null,"ppSeqNo":1,"digest":"9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171","op":"PREPARE","viewNo":1,"ppTime":1507741828,"stateRootHash":null,"instId":1}'], 'signature': None, 'op': 'BATCH'}, 'Node4') 2017-10-11 17:10:28,541 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"txnRootHash":"Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw","ppSeqNo":1,"digest":"9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171","op":"PREPARE","viewNo":1,"ppTime":1507741828,"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","instId":0}', '{"op":"COMMIT","instId":0,"viewNo":1,"ppSeqNo":1}', '{"txnRootHash":null,"ppSeqNo":1,"digest":"9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171","op":"PREPARE","viewNo":1,"ppTime":1507741828,"stateRootHash":null,"instId":1}'], 'signature': None} 2017-10-11 17:10:28,541 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'digest': '9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171', 'ppSeqNo': 1, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': 'Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw', 'instId': 0, 'ppTime': 1507741828, 'viewNo': 1} 2017-10-11 17:10:28,541 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'ppSeqNo': 1, 'digest': '9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171', 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': 'Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw', 'op': 'PREPARE', 'instId': 0, 'ppTime': 1507741828, 'viewNo': 1}, 'Node4') 2017-10-11 17:10:28,541 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'digest': '9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171', 'ppSeqNo': 1, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': 'Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw', 'instId': 0, 'ppTime': 1507741828, 'viewNo': 1} 2017-10-11 17:10:28,542 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 1} 2017-10-11 17:10:28,542 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'ppSeqNo': 1, 'op': 'COMMIT', 'viewNo': 1}, 'Node4') 2017-10-11 17:10:28,542 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 1} 2017-10-11 17:10:28,542 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'digest': '9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171', 'ppSeqNo': 1, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741828, 'viewNo': 1} 2017-10-11 17:10:28,542 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'ppSeqNo': 1, 'digest': '9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171', 'stateRootHash': None, 'txnRootHash': None, 'op': 'PREPARE', 'instId': 1, 'ppTime': 1507741828, 'viewNo': 1}, 'Node4') 2017-10-11 17:10:28,542 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'digest': '9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171', 'ppSeqNo': 1, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741828, 'viewNo': 1} 2017-10-11 17:10:28,542 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: COMMIT{'instId': 1, 'ppSeqNo': 1, 'viewNo': 1} 2017-10-11 17:10:28,542 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'ppSeqNo': 1, 'op': 'COMMIT', 'viewNo': 1}, 'Node4') 2017-10-11 17:10:28,543 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'ppSeqNo': 1, 'viewNo': 1} 2017-10-11 17:10:28,547 | DEBUG | replica.py ( 953) | doCommit | Node2:0 Sending COMMIT(1, 1) at 2270914.005346477 2017-10-11 17:10:28,548 | DEBUG | replica.py (2193) | send | Node2:0 sending Commit 2017-10-11 17:10:28,548 | TRACE | replica.py (2194) | send | Node2:0 sending COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 1} 2017-10-11 17:10:28,548 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 1} commits where f is 1 2017-10-11 17:10:28,548 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (1, 1) 2017-10-11 17:10:28,548 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(1, 1) from Node4:0 2017-10-11 17:10:28,548 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 1} commits where f is 1 2017-10-11 17:10:28,548 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(1, 1) 2017-10-11 17:10:28,548 | DEBUG | replica.py ( 953) | doCommit | Node2:1 Sending COMMIT(1, 1) at 2270914.006357532 2017-10-11 17:10:28,549 | DEBUG | replica.py (2193) | send | Node2:1 sending Commit 2017-10-11 17:10:28,549 | TRACE | replica.py (2194) | send | Node2:1 sending COMMIT{'instId': 1, 'ppSeqNo': 1, 'viewNo': 1} 2017-10-11 17:10:28,549 | DEBUG | replica.py ( 929) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'ppSeqNo': 1, 'viewNo': 1} commits where f is 1 2017-10-11 17:10:28,549 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (1, 1) 2017-10-11 17:10:28,549 | DEBUG | replica.py ( 894) | processCommit | Node2:1 received COMMIT(1, 1) from Node4:1 2017-10-11 17:10:28,549 | DEBUG | replica.py ( 929) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'ppSeqNo': 1, 'viewNo': 1} commits where f is 1 2017-10-11 17:10:28,549 | DEBUG | replica.py ( 905) | processCommit | Node2:1 processed incoming COMMIT(1, 1) 2017-10-11 17:10:28,549 | DEBUG | node.py (2608) | send | Node2 sending message COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 1} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:10:28,549 | DEBUG | node.py (2608) | send | Node2 sending message COMMIT{'instId': 1, 'ppSeqNo': 1, 'viewNo': 1} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:10:28,550 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:10:28,551 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: BATCH{'messages': ['{"viewNo":1,"instId":0,"op":"COMMIT","ppSeqNo":1}', '{"viewNo":1,"instId":1,"op":"COMMIT","ppSeqNo":1}'], 'signature': None} 2017-10-11 17:10:28,551 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"viewNo":1,"instId":0,"op":"COMMIT","ppSeqNo":1}', '{"viewNo":1,"instId":1,"op":"COMMIT","ppSeqNo":1}'], 'signature': None, 'op': 'BATCH'}, 'Node3') 2017-10-11 17:10:28,551 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"viewNo":1,"instId":0,"op":"COMMIT","ppSeqNo":1}', '{"viewNo":1,"instId":1,"op":"COMMIT","ppSeqNo":1}'], 'signature': None} 2017-10-11 17:10:28,551 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 1} 2017-10-11 17:10:28,551 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'ppSeqNo': 1, 'op': 'COMMIT', 'viewNo': 1}, 'Node3') 2017-10-11 17:10:28,551 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 1} 2017-10-11 17:10:28,552 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 1, 'ppSeqNo': 1, 'viewNo': 1} 2017-10-11 17:10:28,552 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'ppSeqNo': 1, 'op': 'COMMIT', 'viewNo': 1}, 'Node3') 2017-10-11 17:10:28,552 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'ppSeqNo': 1, 'viewNo': 1} 2017-10-11 17:10:28,552 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-10-11 17:10:28,552 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"instId":0,"ppSeqNo":1,"op":"COMMIT","viewNo":1}', b'{"instId":1,"ppSeqNo":1,"op":"COMMIT","viewNo":1}']) 2017-10-11 17:10:28,553 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node1: b'{"messages":["{\\"instId\\":0,\\"ppSeqNo\\":1,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1}","{\\"instId\\":1,\\"ppSeqNo\\":1,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:10:28,553 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"instId\\":0,\\"ppSeqNo\\":1,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1}","{\\"instId\\":1,\\"ppSeqNo\\":1,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1}"],"signature":null,"op":"BATCH"}' to Node1 2017-10-11 17:10:28,553 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-10-11 17:10:28,553 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"instId":0,"ppSeqNo":1,"op":"COMMIT","viewNo":1}', b'{"instId":1,"ppSeqNo":1,"op":"COMMIT","viewNo":1}']) 2017-10-11 17:10:28,553 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node3: b'{"messages":["{\\"instId\\":0,\\"ppSeqNo\\":1,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1}","{\\"instId\\":1,\\"ppSeqNo\\":1,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:10:28,553 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"instId\\":0,\\"ppSeqNo\\":1,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1}","{\\"instId\\":1,\\"ppSeqNo\\":1,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1}"],"signature":null,"op":"BATCH"}' to Node3 2017-10-11 17:10:28,553 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node4 into one transmission 2017-10-11 17:10:28,553 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"instId":0,"ppSeqNo":1,"op":"COMMIT","viewNo":1}', b'{"instId":1,"ppSeqNo":1,"op":"COMMIT","viewNo":1}']) 2017-10-11 17:10:28,554 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node4: b'{"messages":["{\\"instId\\":0,\\"ppSeqNo\\":1,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1}","{\\"instId\\":1,\\"ppSeqNo\\":1,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:10:28,554 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"instId\\":0,\\"ppSeqNo\\":1,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1}","{\\"instId\\":1,\\"ppSeqNo\\":1,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1}"],"signature":null,"op":"BATCH"}' to Node4 2017-10-11 17:10:28,554 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(1, 1) from Node3:0 2017-10-11 17:10:28,555 | TRACE | replica.py ( 925) | tryOrder | Node2:0 returning request to node 2017-10-11 17:10:28,555 | DEBUG | replica.py (1444) | doOrder | Node2:0 ordering COMMIT (1, 1) 2017-10-11 17:10:28,555 | DEBUG | replica.py ( 323) | last_ordered_3pc | Node2:0 set last ordered as (1, 1) 2017-10-11 17:10:28,555 | DEBUG | replica.py (2193) | send | Node2:0 sending Ordered 2017-10-11 17:10:28,555 | TRACE | replica.py (2194) | send | Node2:0 sending ORDERED{'ledgerId': 2, 'reqIdr': [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741828468646)], 'ppSeqNo': 1, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': 'Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw', 'instId': 0, 'ppTime': 1507741828, 'viewNo': 1} 2017-10-11 17:10:28,556 | INFO | replica.py (1483) | order_3pc_key | Node2:0 ordered batch request, view no 1, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw, requests ordered [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741828468646)], discarded [] 2017-10-11 17:10:28,556 | DEBUG | replica.py (1556) | _newCheckpointState | Node2:0 adding new checkpoint state for (1, 100) 2017-10-11 17:10:28,556 | DEBUG | replica.py ( 905) | processCommit | Node2:0 processed incoming COMMIT(1, 1) 2017-10-11 17:10:28,556 | DEBUG | replica.py ( 894) | processCommit | Node2:1 received COMMIT(1, 1) from Node3:1 2017-10-11 17:10:28,556 | TRACE | replica.py ( 925) | tryOrder | Node2:1 returning request to node 2017-10-11 17:10:28,556 | DEBUG | replica.py (1444) | doOrder | Node2:1 ordering COMMIT (1, 1) 2017-10-11 17:10:28,556 | DEBUG | replica.py ( 323) | last_ordered_3pc | Node2:1 set last ordered as (1, 1) 2017-10-11 17:10:28,557 | DEBUG | replica.py (2193) | send | Node2:1 sending Ordered 2017-10-11 17:10:28,557 | TRACE | replica.py (2194) | send | Node2:1 sending ORDERED{'ledgerId': 2, 'reqIdr': [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741828468646)], 'ppSeqNo': 1, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741828, 'viewNo': 1} 2017-10-11 17:10:28,557 | INFO | replica.py (1483) | order_3pc_key | Node2:1 ordered batch request, view no 1, ppSeqNo 1, ledger 2, state root None, txn root None, requests ordered [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741828468646)], discarded [] 2017-10-11 17:10:28,557 | DEBUG | replica.py (1556) | _newCheckpointState | Node2:1 adding new checkpoint state for (1, 100) 2017-10-11 17:10:28,557 | DEBUG | replica.py ( 905) | processCommit | Node2:1 processed incoming COMMIT(1, 1) 2017-10-11 17:10:28,557 | DEBUG | node.py (1847) | processOrdered | Node2 executing Ordered batch 1 1 of 1 requests 2017-10-11 17:10:28,558 | DEBUG | ledger.py ( 52) | commitTxns | Committed 1 txns, 0 are uncommitted 2017-10-11 17:10:28,558 | DEBUG | node.py (2386) | sendReplyToClient | Node2 sending reply for ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741828468646) to client 2017-10-11 17:10:28,558 | DEBUG | zstack.py ( 744) | transmitThroughListener | Node2C not sending message {'result': {'auditPath': ['FFusKBPJVV3mVuKVMNsZ1KcJ77B3EPoyMZoXiPdLTKdv', '4EqhrnCKJRxQSAQ8cjfpZUr71oCijptQWfbLq59MwGE7'], 'seqNo': 4, 'signature': '4KPg3heM8TgwKUPvegXDuLkE8UYiUNP162dfCtncKFwgmuQo4hgCgV3ptu8KWVNFQd68w9mJkhQFcdTY9YgqProZ', 'type': '110', 'txnTime': 1507741828, 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'data': {'action': 'complete', 'version': '1.1.40'}, 'reqId': 1507741828468646, 'rootHash': 'Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw'}, 'op': 'REPLY'} to b'Node1' 2017-10-11 17:10:28,558 | DEBUG | zstack.py ( 745) | transmitThroughListener | This is a temporary workaround for not being able to disconnect a ROUTER's remote 2017-10-11 17:10:28,558 | INFO | node.py (2290) | executeBatch | Node2 committed batch request, view no 1, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw, requests: [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1507741828468646)] 2017-10-11 17:10:28,559 | DEBUG | node.py (2305) | executeBatch | Node2 storing 3PC key (1, 1) for ledger 2 range (4, 4) 2017-10-11 17:10:28,559 | TRACE | node.py (1858) | processOrdered | Node2 got ordered requests from backup replica 1 2017-10-11 17:10:28,559 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:10:28,559 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:10:28,560 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PREPARE{'digest': '9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171', 'ppSeqNo': 1, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': 'Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw', 'instId': 0, 'ppTime': 1507741828, 'viewNo': 1} 2017-10-11 17:10:28,560 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'digest': '9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171', 'ppSeqNo': 1, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'op': 'PREPARE', 'txnRootHash': 'Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw', 'instId': 0, 'ppTime': 1507741828, 'viewNo': 1}, 'Node1') 2017-10-11 17:10:28,560 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'digest': '9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171', 'ppSeqNo': 1, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': 'Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw', 'instId': 0, 'ppTime': 1507741828, 'viewNo': 1} 2017-10-11 17:10:28,561 | TRACE | message_processor.py ( 29) | discard | Node2:0 discarding message PREPARE{'digest': '9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171', 'ppSeqNo': 1, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': 'Ee5vA4vbk6gKvwoVh471Z3NsPFyKiYXbCjf3sFHi2nZw', 'instId': 0, 'ppTime': 1507741828, 'viewNo': 1} because already ordered 3 phase message 2017-10-11 17:10:28,573 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:10:28,574 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: BATCH{'messages': ['{"viewNo":1,"ppSeqNo":1,"instId":0,"op":"COMMIT"}', '{"viewNo":1,"digest":"9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171","ppTime":1507741828,"instId":1,"ppSeqNo":1,"txnRootHash":null,"stateRootHash":null,"op":"PREPARE"}', '{"viewNo":1,"ppSeqNo":1,"instId":1,"op":"COMMIT"}'], 'signature': None} 2017-10-11 17:10:28,574 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"viewNo":1,"ppSeqNo":1,"instId":0,"op":"COMMIT"}', '{"viewNo":1,"digest":"9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171","ppTime":1507741828,"instId":1,"ppSeqNo":1,"txnRootHash":null,"stateRootHash":null,"op":"PREPARE"}', '{"viewNo":1,"ppSeqNo":1,"instId":1,"op":"COMMIT"}'], 'op': 'BATCH', 'signature': None}, 'Node1') 2017-10-11 17:10:28,574 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"viewNo":1,"ppSeqNo":1,"instId":0,"op":"COMMIT"}', '{"viewNo":1,"digest":"9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171","ppTime":1507741828,"instId":1,"ppSeqNo":1,"txnRootHash":null,"stateRootHash":null,"op":"PREPARE"}', '{"viewNo":1,"ppSeqNo":1,"instId":1,"op":"COMMIT"}'], 'signature': None} 2017-10-11 17:10:28,574 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 1} 2017-10-11 17:10:28,575 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'ppSeqNo': 1, 'op': 'COMMIT', 'viewNo': 1}, 'Node1') 2017-10-11 17:10:28,575 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 1} 2017-10-11 17:10:28,575 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PREPARE{'digest': '9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171', 'ppSeqNo': 1, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741828, 'viewNo': 1} 2017-10-11 17:10:28,575 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'digest': '9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171', 'ppSeqNo': 1, 'stateRootHash': None, 'op': 'PREPARE', 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741828, 'viewNo': 1}, 'Node1') 2017-10-11 17:10:28,575 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'digest': '9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171', 'ppSeqNo': 1, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741828, 'viewNo': 1} 2017-10-11 17:10:28,575 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 1, 'ppSeqNo': 1, 'viewNo': 1} 2017-10-11 17:10:28,575 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 1, 'ppSeqNo': 1, 'op': 'COMMIT', 'viewNo': 1}, 'Node1') 2017-10-11 17:10:28,575 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 1, 'ppSeqNo': 1, 'viewNo': 1} 2017-10-11 17:10:28,576 | TRACE | message_processor.py ( 29) | discard | Node2:0 discarding message COMMIT{'instId': 0, 'ppSeqNo': 1, 'viewNo': 1} because already ordered 3 phase message 2017-10-11 17:10:28,576 | TRACE | message_processor.py ( 29) | discard | Node2:1 discarding message PREPARE{'digest': '9d617548384ccf6c1f17e5d7d5bd4d2d1b03325dafd8844bea4c566a86ba0171', 'ppSeqNo': 1, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507741828, 'viewNo': 1} because already ordered 3 phase message 2017-10-11 17:10:28,576 | TRACE | message_processor.py ( 29) | discard | Node2:1 discarding message COMMIT{'instId': 1, 'ppSeqNo': 1, 'viewNo': 1} because already ordered 3 phase message 2017-10-11 17:10:37,020 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:10:37,020 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:10:37,020 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:10:37,020 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:10:37,103 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 57 2017-10-11 17:10:37,103 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:10:37,103 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:10:37,103 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0040439605668454 is acceptable. 2017-10-11 17:10:37,103 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:10:37,104 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:10:37,104 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:10:37,104 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 58 to run in 10 seconds 2017-10-11 17:10:47,111 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 58 2017-10-11 17:10:47,111 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:10:47,112 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:10:47,112 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0040439605668454 is acceptable. 2017-10-11 17:10:47,112 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:10:47,112 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:10:47,113 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:10:47,113 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 59 to run in 10 seconds 2017-10-11 17:10:52,030 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:10:52,030 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:10:52,030 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:10:52,030 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:10:57,119 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 59 2017-10-11 17:10:57,120 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:10:57,120 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:10:57,120 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0040439605668454 is acceptable. 2017-10-11 17:10:57,120 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:10:57,120 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:10:57,120 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:10:57,120 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 60 to run in 10 seconds 2017-10-11 17:11:06,811 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 7 2017-10-11 17:11:06,811 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:11:06,811 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 8 to run in 60 seconds 2017-10-11 17:11:06,846 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 51 2017-10-11 17:11:06,846 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node2 checking its request amount 2017-10-11 17:11:06,847 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:11:06,847 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 61 to run in 60 seconds 2017-10-11 17:11:06,940 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action dump_json_file with id 52 2017-10-11 17:11:06,941 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action dump_json_file with id 62 to run in 60 seconds 2017-10-11 17:11:07,032 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:11:07,032 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:11:07,032 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:11:07,032 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:11:07,124 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 60 2017-10-11 17:11:07,124 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:11:07,124 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:11:07,125 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0040439605668454 is acceptable. 2017-10-11 17:11:07,125 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:11:07,125 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:11:07,125 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:11:07,125 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 63 to run in 10 seconds 2017-10-11 17:11:17,127 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 63 2017-10-11 17:11:17,127 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:11:17,127 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:11:17,127 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0040439605668454 is acceptable. 2017-10-11 17:11:17,127 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:11:17,127 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:11:17,127 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:11:17,127 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 64 to run in 10 seconds 2017-10-11 17:11:19,367 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action _check_view_change_completed with id 56 2017-10-11 17:11:19,367 | DEBUG | node.py (1011) | _check_view_change_completed | Node2 running the scheduled check for view change completion 2017-10-11 17:11:19,367 | DEBUG | node.py (1013) | _check_view_change_completed | Node2 already completion view change 2017-10-11 17:11:22,041 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:11:22,041 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:11:22,041 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:11:22,042 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:11:27,134 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 64 2017-10-11 17:11:27,135 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:11:27,135 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:11:27,135 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0040439605668454 is acceptable. 2017-10-11 17:11:27,136 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:11:27,136 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:11:27,136 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:11:27,136 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 65 to run in 10 seconds 2017-10-11 17:11:37,051 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:11:37,051 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:11:37,051 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:11:37,052 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:11:37,145 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 65 2017-10-11 17:11:37,145 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:11:37,145 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:11:37,145 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0040439605668454 is acceptable. 2017-10-11 17:11:37,145 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:11:37,145 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:11:37,145 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:11:37,146 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 66 to run in 10 seconds 2017-10-11 17:11:47,149 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 66 2017-10-11 17:11:47,149 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:11:47,149 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:11:47,149 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0040439605668454 is acceptable. 2017-10-11 17:11:47,150 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:11:47,150 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:11:47,150 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:11:47,150 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 67 to run in 10 seconds 2017-10-11 17:11:52,055 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:11:52,055 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:11:52,055 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:11:52,055 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:11:57,156 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 67 2017-10-11 17:11:57,156 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:11:57,156 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:11:57,156 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0040439605668454 is acceptable. 2017-10-11 17:11:57,156 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:11:57,157 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:11:57,157 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:11:57,157 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 68 to run in 10 seconds 2017-10-11 17:12:06,814 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 8 2017-10-11 17:12:06,814 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:12:06,814 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 9 to run in 60 seconds 2017-10-11 17:12:06,849 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 61 2017-10-11 17:12:06,850 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node2 checking its request amount 2017-10-11 17:12:06,850 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:12:06,850 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 69 to run in 60 seconds 2017-10-11 17:12:06,942 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action dump_json_file with id 62 2017-10-11 17:12:06,943 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action dump_json_file with id 70 to run in 60 seconds 2017-10-11 17:12:07,059 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:12:07,059 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:12:07,059 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:12:07,060 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:12:07,162 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 68 2017-10-11 17:12:07,162 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:12:07,162 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:12:07,162 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0040439605668454 is acceptable. 2017-10-11 17:12:07,162 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:12:07,162 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:12:07,162 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:12:07,162 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 71 to run in 10 seconds 2017-10-11 17:12:17,169 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 71 2017-10-11 17:12:17,169 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:12:17,169 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:12:17,169 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0040439605668454 is acceptable. 2017-10-11 17:12:17,169 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:12:17,169 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:12:17,169 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:12:17,170 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 72 to run in 10 seconds 2017-10-11 17:12:22,060 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:12:22,061 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:12:22,061 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:12:22,061 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:12:27,178 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 72 2017-10-11 17:12:27,179 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:12:27,179 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:12:27,179 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0040439605668454 is acceptable. 2017-10-11 17:12:27,179 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:12:27,179 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:12:27,179 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:12:27,179 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 73 to run in 10 seconds 2017-10-11 17:12:37,064 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:12:37,064 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:12:37,064 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:12:37,065 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:12:37,185 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 73 2017-10-11 17:12:37,186 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:12:37,186 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:12:37,186 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0040439605668454 is acceptable. 2017-10-11 17:12:37,186 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:12:37,186 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:12:37,186 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:12:37,186 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 74 to run in 10 seconds 2017-10-11 17:12:47,189 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 74 2017-10-11 17:12:47,189 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:12:47,189 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:12:47,190 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0040439605668454 is acceptable. 2017-10-11 17:12:47,190 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:12:47,190 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:12:47,190 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:12:47,190 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 75 to run in 10 seconds 2017-10-11 17:12:52,064 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:12:52,064 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:12:52,065 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:12:52,065 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:12:57,200 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 75 2017-10-11 17:12:57,200 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:12:57,200 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:12:57,200 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0040439605668454 is acceptable. 2017-10-11 17:12:57,200 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:12:57,200 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:12:57,201 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:12:57,201 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 76 to run in 10 seconds 2017-10-11 17:13:06,824 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 9 2017-10-11 17:13:06,824 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:13:06,824 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 10 to run in 60 seconds 2017-10-11 17:13:06,861 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 69 2017-10-11 17:13:06,861 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node2 checking its request amount 2017-10-11 17:13:06,861 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:13:06,861 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 77 to run in 60 seconds 2017-10-11 17:13:06,945 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action dump_json_file with id 70 2017-10-11 17:13:06,946 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action dump_json_file with id 78 to run in 60 seconds 2017-10-11 17:13:07,075 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:13:07,075 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:13:07,075 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:13:07,076 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:13:07,202 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 76 2017-10-11 17:13:07,203 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:13:07,203 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:13:07,203 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0040439605668454 is acceptable. 2017-10-11 17:13:07,203 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:13:07,203 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:13:07,203 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:13:07,203 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 79 to run in 10 seconds 2017-10-11 17:13:17,205 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 79 2017-10-11 17:13:17,205 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:13:17,205 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:13:17,205 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0040439605668454 is acceptable. 2017-10-11 17:13:17,205 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:13:17,206 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:13:17,206 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:13:17,206 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 80 to run in 10 seconds 2017-10-11 17:13:22,086 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:13:22,086 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:13:22,086 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:13:22,086 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:13:27,208 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 80 2017-10-11 17:13:27,208 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:13:27,208 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:13:27,208 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0040439605668454 is acceptable. 2017-10-11 17:13:27,208 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:13:27,208 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:13:27,208 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:13:27,208 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 81 to run in 10 seconds 2017-10-11 17:13:37,090 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:13:37,090 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:13:37,090 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:13:37,091 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:13:37,209 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 81 2017-10-11 17:13:37,209 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:13:37,209 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:13:37,209 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0040439605668454 is acceptable. 2017-10-11 17:13:37,209 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:13:37,210 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:13:37,210 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:13:37,210 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 82 to run in 10 seconds 2017-10-11 17:13:47,221 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 82 2017-10-11 17:13:47,221 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:13:47,221 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:13:47,221 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0040439605668454 is acceptable. 2017-10-11 17:13:47,221 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:13:47,222 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:13:47,222 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:13:47,222 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 83 to run in 10 seconds 2017-10-11 17:13:52,093 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:13:52,093 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:13:52,093 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:13:52,094 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:13:57,229 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 83 2017-10-11 17:13:57,230 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:13:57,230 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:13:57,230 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0040439605668454 is acceptable. 2017-10-11 17:13:57,230 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:13:57,230 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:13:57,230 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:13:57,230 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 84 to run in 10 seconds 2017-10-11 17:14:06,829 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 10 2017-10-11 17:14:06,829 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a ClusterThroughputSuspiciousSpike spike 2017-10-11 17:14:06,829 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 11 to run in 60 seconds 2017-10-11 17:14:06,868 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkNodeRequestSpike with id 77 2017-10-11 17:14:06,868 | DEBUG | node.py (1999) | checkNodeRequestSpike | Node2 checking its request amount 2017-10-11 17:14:06,869 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:14:06,869 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkNodeRequestSpike with id 85 to run in 60 seconds 2017-10-11 17:14:06,947 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action dump_json_file with id 78 2017-10-11 17:14:06,948 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action dump_json_file with id 86 to run in 60 seconds 2017-10-11 17:14:07,103 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:14:07,104 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:14:07,104 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:14:07,106 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:14:07,233 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 84 2017-10-11 17:14:07,234 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:14:07,234 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:14:07,234 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0040439605668454 is acceptable. 2017-10-11 17:14:07,234 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:14:07,234 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:14:07,235 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:14:07,235 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 87 to run in 10 seconds 2017-10-11 17:14:17,247 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 87 2017-10-11 17:14:17,247 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:14:17,247 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:14:17,247 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0040439605668454 is acceptable. 2017-10-11 17:14:17,247 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:14:17,247 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:14:17,247 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:14:17,248 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 88 to run in 10 seconds 2017-10-11 17:14:22,115 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:14:22,115 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:14:22,116 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:14:22,116 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:14:27,248 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 88 2017-10-11 17:14:27,249 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:14:27,249 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:14:27,249 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0040439605668454 is acceptable. 2017-10-11 17:14:27,249 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:14:27,249 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:14:27,249 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:14:27,249 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 89 to run in 10 seconds 2017-10-11 17:14:37,117 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:14:37,117 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:14:37,117 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:14:37,118 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:14:37,260 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 89 2017-10-11 17:14:37,260 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:14:37,260 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:14:37,260 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0040439605668454 is acceptable. 2017-10-11 17:14:37,260 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:14:37,260 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:14:37,260 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:14:37,261 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 90 to run in 10 seconds 2017-10-11 17:14:47,271 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 90 2017-10-11 17:14:47,271 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:14:47,271 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:14:47,271 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0040439605668454 is acceptable. 2017-10-11 17:14:47,271 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:14:47,271 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:14:47,271 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:14:47,271 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 91 to run in 10 seconds 2017-10-11 17:14:52,127 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node1 HA(host='10.0.0.2', port=9701) 2017-10-11 17:14:52,128 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node3 HA(host='10.0.0.4', port=9705) 2017-10-11 17:14:52,128 | DEBUG | kit_zstack.py ( 67) | reconcileNodeReg | Node2 matched remote Node4 HA(host='10.0.0.5', port=9707) 2017-10-11 17:14:52,128 | DEBUG | kit_zstack.py ( 50) | maintainConnections | Node2 next check for retries in 15.00 seconds 2017-10-11 17:14:57,272 | TRACE | has_action_queue.py ( 66) | _serviceActions | Node2 running action checkPerformance with id 91 2017-10-11 17:14:57,272 | TRACE | node.py (1978) | checkPerformance | Node2 checking its performance 2017-10-11 17:14:57,272 | DEBUG | notifier_plugin_manager.py ( 74) | sendMessageUponSuspiciousSpike | Not enough data to detect a NodeRequestSuspiciousSpike spike 2017-10-11 17:14:57,272 | TRACE | monitor.py ( 344) | isMasterThroughputTooLow | Node2 master throughput ratio 1.0040439605668454 is acceptable. 2017-10-11 17:14:57,273 | TRACE | monitor.py ( 361) | isMasterReqLatencyTooHigh | Node2 found master's latency to be lower than the threshold for all requests. 2017-10-11 17:14:57,273 | TRACE | monitor.py ( 391) | isMasterAvgReqLatencyTooHigh | Node2 found difference between master and backups avg latencies to be acceptable 2017-10-11 17:14:57,273 | DEBUG | node.py (1995) | checkPerformance | Node2's master has higher performance than backups 2017-10-11 17:14:57,273 | TRACE | has_action_queue.py ( 36) | _schedule | Node2 scheduling action checkPerformance with id 92 to run in 10 seconds 2017-10-11 17:14:59,431 | TRACE | has_action_queue.py ( 66) | _serviceActions | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb running action partial(_callUpgradeAgent) with id 2 2017-10-11 17:14:59,431 | INFO | upgrader.py ( 421) | _callUpgradeAgent | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb's upgrader calling agent for upgrade 2017-10-11 17:14:59,431 | INFO | node.py ( 254) | notify_upgrade_start | Node2 is about to be upgraded, sending NODE_UPGRADE 2017-10-11 17:14:59,432 | DEBUG | node.py (2608) | send | Node2 sending message Request: {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': '3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS'}, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '5hEHFtUkz1JBGstkZSUKhaeUqJbnjBSLimd7yJWuPbHdpc5iZ8yqXL3TSU5obg9MbeNt2WmnaJyZFQDx5AY3JHVB', 'reqId': 1507742099432084} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:14:59,433 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"operation":{"type":"110","data":{"action":"in_progress","version":"1.1.40"},"signature":"3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS"},"identifier":"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb","signature":"5hEHFtUkz1JBGstkZSUKhaeUqJbnjBSLimd7yJWuPbHdpc5iZ8yqXL3TSU5obg9MbeNt2WmnaJyZFQDx5AY3JHVB","reqId":1507742099432084}' to Node1 2017-10-11 17:14:59,433 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"operation":{"type":"110","data":{"action":"in_progress","version":"1.1.40"},"signature":"3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS"},"identifier":"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb","signature":"5hEHFtUkz1JBGstkZSUKhaeUqJbnjBSLimd7yJWuPbHdpc5iZ8yqXL3TSU5obg9MbeNt2WmnaJyZFQDx5AY3JHVB","reqId":1507742099432084}' to Node1 2017-10-11 17:14:59,433 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"operation":{"type":"110","data":{"action":"in_progress","version":"1.1.40"},"signature":"3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS"},"identifier":"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb","signature":"5hEHFtUkz1JBGstkZSUKhaeUqJbnjBSLimd7yJWuPbHdpc5iZ8yqXL3TSU5obg9MbeNt2WmnaJyZFQDx5AY3JHVB","reqId":1507742099432084}' to Node3 2017-10-11 17:14:59,433 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"operation":{"type":"110","data":{"action":"in_progress","version":"1.1.40"},"signature":"3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS"},"identifier":"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb","signature":"5hEHFtUkz1JBGstkZSUKhaeUqJbnjBSLimd7yJWuPbHdpc5iZ8yqXL3TSU5obg9MbeNt2WmnaJyZFQDx5AY3JHVB","reqId":1507742099432084}' to Node3 2017-10-11 17:14:59,433 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"operation":{"type":"110","data":{"action":"in_progress","version":"1.1.40"},"signature":"3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS"},"identifier":"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb","signature":"5hEHFtUkz1JBGstkZSUKhaeUqJbnjBSLimd7yJWuPbHdpc5iZ8yqXL3TSU5obg9MbeNt2WmnaJyZFQDx5AY3JHVB","reqId":1507742099432084}' to Node4 2017-10-11 17:14:59,433 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"operation":{"type":"110","data":{"action":"in_progress","version":"1.1.40"},"signature":"3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS"},"identifier":"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb","signature":"5hEHFtUkz1JBGstkZSUKhaeUqJbnjBSLimd7yJWuPbHdpc5iZ8yqXL3TSU5obg9MbeNt2WmnaJyZFQDx5AY3JHVB","reqId":1507742099432084}' to Node4 2017-10-11 17:14:59,434 | INFO | upgrader.py ( 433) | _sendUpdateRequest | Sending message to control tool: {"version": "1.1.40"} 2017-10-11 17:14:59,435 | INFO | upgrader.py ( 452) | _sendUpdateRequest | Waiting 10 minutes for upgrade to be performed 2017-10-11 17:14:59,435 | TRACE | has_action_queue.py ( 36) | _schedule | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb scheduling action partial(_declareTimeoutExceeded) with id 3 to run in 600 seconds 2017-10-11 17:14:59,447 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 3 messages through listener 2017-10-11 17:14:59,448 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb signature on propagate request 1507742099432084 2017-10-11 17:14:59,449 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PROPAGATE{'senderClient': 'Node2', 'request': {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': '3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS'}, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '5hEHFtUkz1JBGstkZSUKhaeUqJbnjBSLimd7yJWuPbHdpc5iZ8yqXL3TSU5obg9MbeNt2WmnaJyZFQDx5AY3JHVB', 'reqId': 1507742099432084}} 2017-10-11 17:14:59,449 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'senderClient': 'Node2', 'request': {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': '3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS'}, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '5hEHFtUkz1JBGstkZSUKhaeUqJbnjBSLimd7yJWuPbHdpc5iZ8yqXL3TSU5obg9MbeNt2WmnaJyZFQDx5AY3JHVB', 'reqId': 1507742099432084}, 'op': 'PROPAGATE'}, 'Node3') 2017-10-11 17:14:59,449 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': 'Node2', 'request': {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': '3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS'}, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '5hEHFtUkz1JBGstkZSUKhaeUqJbnjBSLimd7yJWuPbHdpc5iZ8yqXL3TSU5obg9MbeNt2WmnaJyZFQDx5AY3JHVB', 'reqId': 1507742099432084}} 2017-10-11 17:14:59,450 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb signature on propagate request 1507742099432084 2017-10-11 17:14:59,450 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PROPAGATE{'senderClient': 'Node2', 'request': {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': '3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS'}, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '5hEHFtUkz1JBGstkZSUKhaeUqJbnjBSLimd7yJWuPbHdpc5iZ8yqXL3TSU5obg9MbeNt2WmnaJyZFQDx5AY3JHVB', 'reqId': 1507742099432084}} 2017-10-11 17:14:59,451 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'senderClient': 'Node2', 'request': {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': '3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS'}, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '5hEHFtUkz1JBGstkZSUKhaeUqJbnjBSLimd7yJWuPbHdpc5iZ8yqXL3TSU5obg9MbeNt2WmnaJyZFQDx5AY3JHVB', 'reqId': 1507742099432084}, 'op': 'PROPAGATE'}, 'Node1') 2017-10-11 17:14:59,451 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': 'Node2', 'request': {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': '3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS'}, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '5hEHFtUkz1JBGstkZSUKhaeUqJbnjBSLimd7yJWuPbHdpc5iZ8yqXL3TSU5obg9MbeNt2WmnaJyZFQDx5AY3JHVB', 'reqId': 1507742099432084}} 2017-10-11 17:14:59,451 | DEBUG | node.py (2236) | verifySignature | Node2 authenticated 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb signature on propagate request 1507742099432084 2017-10-11 17:14:59,451 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: PROPAGATE{'senderClient': 'Node2', 'request': {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': '3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS'}, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '5hEHFtUkz1JBGstkZSUKhaeUqJbnjBSLimd7yJWuPbHdpc5iZ8yqXL3TSU5obg9MbeNt2WmnaJyZFQDx5AY3JHVB', 'reqId': 1507742099432084}} 2017-10-11 17:14:59,452 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'senderClient': 'Node2', 'request': {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': '3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS'}, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '5hEHFtUkz1JBGstkZSUKhaeUqJbnjBSLimd7yJWuPbHdpc5iZ8yqXL3TSU5obg9MbeNt2WmnaJyZFQDx5AY3JHVB', 'reqId': 1507742099432084}, 'op': 'PROPAGATE'}, 'Node4') 2017-10-11 17:14:59,452 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PROPAGATE{'senderClient': 'Node2', 'request': {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': '3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS'}, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '5hEHFtUkz1JBGstkZSUKhaeUqJbnjBSLimd7yJWuPbHdpc5iZ8yqXL3TSU5obg9MbeNt2WmnaJyZFQDx5AY3JHVB', 'reqId': 1507742099432084}} 2017-10-11 17:14:59,452 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': 'Node2', 'request': {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': '3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS'}, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '5hEHFtUkz1JBGstkZSUKhaeUqJbnjBSLimd7yJWuPbHdpc5iZ8yqXL3TSU5obg9MbeNt2WmnaJyZFQDx5AY3JHVB', 'reqId': 1507742099432084}} 2017-10-11 17:14:59,452 | TRACE | propagator.py ( 165) | createPropagate | Creating PROPAGATE for REQUEST SafeRequest: {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': '3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS'}, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '5hEHFtUkz1JBGstkZSUKhaeUqJbnjBSLimd7yJWuPbHdpc5iZ8yqXL3TSU5obg9MbeNt2WmnaJyZFQDx5AY3JHVB', 'reqId': 1507742099432084} 2017-10-11 17:14:59,452 | INFO | propagator.py ( 148) | propagate | Node2 propagating request ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1507742099432084) from client Node2 2017-10-11 17:14:59,452 | DEBUG | node.py (2608) | send | Node2 sending message PROPAGATE{'senderClient': 'Node2', 'request': {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': '3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS'}, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '5hEHFtUkz1JBGstkZSUKhaeUqJbnjBSLimd7yJWuPbHdpc5iZ8yqXL3TSU5obg9MbeNt2WmnaJyZFQDx5AY3JHVB', 'reqId': 1507742099432084}} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:14:59,454 | DEBUG | propagator.py ( 214) | forward | Node2 forwarding request ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1507742099432084) to 0 replicas 2017-10-11 17:14:59,454 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': 'Node2', 'request': {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': '3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS'}, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '5hEHFtUkz1JBGstkZSUKhaeUqJbnjBSLimd7yJWuPbHdpc5iZ8yqXL3TSU5obg9MbeNt2WmnaJyZFQDx5AY3JHVB', 'reqId': 1507742099432084}} 2017-10-11 17:14:59,455 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': '3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS'}, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '5hEHFtUkz1JBGstkZSUKhaeUqJbnjBSLimd7yJWuPbHdpc5iZ8yqXL3TSU5obg9MbeNt2WmnaJyZFQDx5AY3JHVB', 'reqId': 1507742099432084} 2017-10-11 17:14:59,455 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': '3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS'}, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '5hEHFtUkz1JBGstkZSUKhaeUqJbnjBSLimd7yJWuPbHdpc5iZ8yqXL3TSU5obg9MbeNt2WmnaJyZFQDx5AY3JHVB', 'reqId': 1507742099432084} to its replicas since already forwarded 2017-10-11 17:14:59,455 | DEBUG | node.py (1757) | processPropagate | Node Node2 received propagated request: PROPAGATE{'senderClient': 'Node2', 'request': {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': '3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS'}, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '5hEHFtUkz1JBGstkZSUKhaeUqJbnjBSLimd7yJWuPbHdpc5iZ8yqXL3TSU5obg9MbeNt2WmnaJyZFQDx5AY3JHVB', 'reqId': 1507742099432084}} 2017-10-11 17:14:59,455 | TRACE | propagator.py ( 141) | propagate | Node2 already propagated SafeRequest: {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': '3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS'}, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '5hEHFtUkz1JBGstkZSUKhaeUqJbnjBSLimd7yJWuPbHdpc5iZ8yqXL3TSU5obg9MbeNt2WmnaJyZFQDx5AY3JHVB', 'reqId': 1507742099432084} 2017-10-11 17:14:59,455 | DEBUG | propagator.py ( 246) | tryForwarding | Node2 not forwarding request SafeRequest: {'operation': {'type': '110', 'data': {'action': 'in_progress', 'version': '1.1.40'}, 'signature': '3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS'}, 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'signature': '5hEHFtUkz1JBGstkZSUKhaeUqJbnjBSLimd7yJWuPbHdpc5iZ8yqXL3TSU5obg9MbeNt2WmnaJyZFQDx5AY3JHVB', 'reqId': 1507742099432084} to its replicas since already forwarded 2017-10-11 17:14:59,456 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"senderClient":"Node2","request":{"operation":{"type":"110","data":{"action":"in_progress","version":"1.1.40"},"signature":"3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS"},"identifier":"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb","signature":"5hEHFtUkz1JBGstkZSUKhaeUqJbnjBSLimd7yJWuPbHdpc5iZ8yqXL3TSU5obg9MbeNt2WmnaJyZFQDx5AY3JHVB","reqId":1507742099432084},"op":"PROPAGATE"}' to Node1 2017-10-11 17:14:59,456 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"senderClient":"Node2","request":{"operation":{"type":"110","data":{"action":"in_progress","version":"1.1.40"},"signature":"3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS"},"identifier":"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb","signature":"5hEHFtUkz1JBGstkZSUKhaeUqJbnjBSLimd7yJWuPbHdpc5iZ8yqXL3TSU5obg9MbeNt2WmnaJyZFQDx5AY3JHVB","reqId":1507742099432084},"op":"PROPAGATE"}' to Node1 2017-10-11 17:14:59,456 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"senderClient":"Node2","request":{"operation":{"type":"110","data":{"action":"in_progress","version":"1.1.40"},"signature":"3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS"},"identifier":"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb","signature":"5hEHFtUkz1JBGstkZSUKhaeUqJbnjBSLimd7yJWuPbHdpc5iZ8yqXL3TSU5obg9MbeNt2WmnaJyZFQDx5AY3JHVB","reqId":1507742099432084},"op":"PROPAGATE"}' to Node3 2017-10-11 17:14:59,457 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"senderClient":"Node2","request":{"operation":{"type":"110","data":{"action":"in_progress","version":"1.1.40"},"signature":"3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS"},"identifier":"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb","signature":"5hEHFtUkz1JBGstkZSUKhaeUqJbnjBSLimd7yJWuPbHdpc5iZ8yqXL3TSU5obg9MbeNt2WmnaJyZFQDx5AY3JHVB","reqId":1507742099432084},"op":"PROPAGATE"}' to Node3 2017-10-11 17:14:59,457 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"senderClient":"Node2","request":{"operation":{"type":"110","data":{"action":"in_progress","version":"1.1.40"},"signature":"3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS"},"identifier":"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb","signature":"5hEHFtUkz1JBGstkZSUKhaeUqJbnjBSLimd7yJWuPbHdpc5iZ8yqXL3TSU5obg9MbeNt2WmnaJyZFQDx5AY3JHVB","reqId":1507742099432084},"op":"PROPAGATE"}' to Node4 2017-10-11 17:14:59,457 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"senderClient":"Node2","request":{"operation":{"type":"110","data":{"action":"in_progress","version":"1.1.40"},"signature":"3gvc5cqDb5BFedoR6WCt2MwWeqzvhUSNPEmYTiFbWM41hKVQXqCae8xotWNUPRo1E9c5LN7D4Too7v7P4mr4tXRS"},"identifier":"8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb","signature":"5hEHFtUkz1JBGstkZSUKhaeUqJbnjBSLimd7yJWuPbHdpc5iZ8yqXL3TSU5obg9MbeNt2WmnaJyZFQDx5AY3JHVB","reqId":1507742099432084},"op":"PROPAGATE"}' to Node4 2017-10-11 17:14:59,458 | DEBUG | replica.py ( 667) | create3PCBatch | Node2:0 creating batch 2 for ledger 2 with state root b'\xbc q\xa4\xde\x84o(W\x02D\x7f%\x89\xdd\x166x\xe0\x97*\x8a\x1b\r(\xb0N\xd5\xc0\x94T\x7f' 2017-10-11 17:14:59,458 | DEBUG | replica.py ( 700) | create3PCBatch | Node2:0 created a PRE-PREPARE with 1 requests for ledger 2 2017-10-11 17:14:59,458 | DEBUG | replica.py (2193) | send | Node2:0 sending PrePrepare 2017-10-11 17:14:59,459 | TRACE | replica.py (2194) | send | Node2:0 sending PREPREPARE{'ledgerId': 2, 'reqIdr': [('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1507742099432084)], 'ppSeqNo': 2, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': 'GavLfumt6ugt9aC1DU8Q27QC7KyKz3UWHVKLejGJPsSb', 'instId': 0, 'ppTime': 1507742099, 'discarded': 1, 'digest': 'a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b', 'viewNo': 1} 2017-10-11 17:14:59,459 | DEBUG | replica.py ( 614) | trackBatches | Node2:0 tracking batch for PREPREPARE{'ledgerId': 2, 'reqIdr': [('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1507742099432084)], 'ppSeqNo': 2, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': 'GavLfumt6ugt9aC1DU8Q27QC7KyKz3UWHVKLejGJPsSb', 'instId': 0, 'ppTime': 1507742099, 'discarded': 1, 'digest': 'a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b', 'viewNo': 1} with state root b'\xbc q\xa4\xde\x84o(W\x02D\x7f%\x89\xdd\x166x\xe0\x97*\x8a\x1b\r(\xb0N\xd5\xc0\x94T\x7f' 2017-10-11 17:14:59,459 | DEBUG | node.py (2608) | send | Node2 sending message PREPREPARE{'ledgerId': 2, 'reqIdr': [('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1507742099432084)], 'ppSeqNo': 2, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': 'GavLfumt6ugt9aC1DU8Q27QC7KyKz3UWHVKLejGJPsSb', 'instId': 0, 'ppTime': 1507742099, 'discarded': 1, 'digest': 'a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b', 'viewNo': 1} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:14:59,459 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 1 messages through listener 2017-10-11 17:14:59,460 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PREPREPARE{'ledgerId': 2, 'reqIdr': [['8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1507742099432084]], 'ppSeqNo': 2, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507742099, 'discarded': 1, 'digest': 'a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b', 'viewNo': 1} 2017-10-11 17:14:59,460 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'ledgerId': 2, 'reqIdr': [['8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1507742099432084]], 'stateRootHash': None, 'ppSeqNo': 2, 'op': 'PREPREPARE', 'txnRootHash': None, 'instId': 1, 'ppTime': 1507742099, 'discarded': 1, 'digest': 'a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b', 'viewNo': 1}, 'Node3') 2017-10-11 17:14:59,460 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPREPARE{'ledgerId': 2, 'reqIdr': [['8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1507742099432084]], 'ppSeqNo': 2, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507742099, 'discarded': 1, 'digest': 'a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b', 'viewNo': 1} 2017-10-11 17:14:59,461 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","digest":"a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b","discarded":1,"ledgerId":2,"reqIdr":[["8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb",1507742099432084]],"ppSeqNo":2,"op":"PREPREPARE","instId":0,"ppTime":1507742099,"viewNo":1,"txnRootHash":"GavLfumt6ugt9aC1DU8Q27QC7KyKz3UWHVKLejGJPsSb"}' to Node1 2017-10-11 17:14:59,461 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","digest":"a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b","discarded":1,"ledgerId":2,"reqIdr":[["8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb",1507742099432084]],"ppSeqNo":2,"op":"PREPREPARE","instId":0,"ppTime":1507742099,"viewNo":1,"txnRootHash":"GavLfumt6ugt9aC1DU8Q27QC7KyKz3UWHVKLejGJPsSb"}' to Node1 2017-10-11 17:14:59,461 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","digest":"a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b","discarded":1,"ledgerId":2,"reqIdr":[["8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb",1507742099432084]],"ppSeqNo":2,"op":"PREPREPARE","instId":0,"ppTime":1507742099,"viewNo":1,"txnRootHash":"GavLfumt6ugt9aC1DU8Q27QC7KyKz3UWHVKLejGJPsSb"}' to Node3 2017-10-11 17:14:59,461 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","digest":"a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b","discarded":1,"ledgerId":2,"reqIdr":[["8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb",1507742099432084]],"ppSeqNo":2,"op":"PREPREPARE","instId":0,"ppTime":1507742099,"viewNo":1,"txnRootHash":"GavLfumt6ugt9aC1DU8Q27QC7KyKz3UWHVKLejGJPsSb"}' to Node3 2017-10-11 17:14:59,462 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","digest":"a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b","discarded":1,"ledgerId":2,"reqIdr":[["8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb",1507742099432084]],"ppSeqNo":2,"op":"PREPREPARE","instId":0,"ppTime":1507742099,"viewNo":1,"txnRootHash":"GavLfumt6ugt9aC1DU8Q27QC7KyKz3UWHVKLejGJPsSb"}' to Node4 2017-10-11 17:14:59,462 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","digest":"a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b","discarded":1,"ledgerId":2,"reqIdr":[["8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb",1507742099432084]],"ppSeqNo":2,"op":"PREPREPARE","instId":0,"ppTime":1507742099,"viewNo":1,"txnRootHash":"GavLfumt6ugt9aC1DU8Q27QC7KyKz3UWHVKLejGJPsSb"}' to Node4 2017-10-11 17:14:59,462 | DEBUG | replica.py ( 823) | processPrePrepare | Node2:1 received PRE-PREPARE(1, 2) from Node3:1 at 2271184.920049715 2017-10-11 17:14:59,463 | DEBUG | replica.py ( 934) | doPrepare | Node2:1 Sending PREPARE(1, 2) at 2271184.920556363 2017-10-11 17:14:59,463 | DEBUG | replica.py (2193) | send | Node2:1 sending Prepare 2017-10-11 17:14:59,463 | TRACE | replica.py (2194) | send | Node2:1 sending PREPARE{'digest': 'a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b', 'ppSeqNo': 2, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507742099, 'viewNo': 1} 2017-10-11 17:14:59,463 | DEBUG | replica.py ( 917) | tryCommit | Node2:1 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': 'a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b', 'ppSeqNo': 2, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507742099, 'viewNo': 1} 2017-10-11 17:14:59,463 | DEBUG | replica.py ( 614) | trackBatches | Node2:1 tracking batch for PREPREPARE{'ledgerId': 2, 'reqIdr': [('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1507742099432084)], 'ppSeqNo': 2, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507742099, 'discarded': 1, 'digest': 'a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b', 'viewNo': 1} with state root None 2017-10-11 17:14:59,463 | DEBUG | replica.py ( 842) | processPrePrepare | Node2:1 processed incoming PRE-PREPARE(1, 2) 2017-10-11 17:14:59,463 | DEBUG | node.py (2608) | send | Node2 sending message PREPARE{'digest': 'a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b', 'ppSeqNo': 2, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507742099, 'viewNo': 1} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:14:59,466 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":null,"viewNo":1,"digest":"a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b","ppSeqNo":2,"op":"PREPARE","instId":1,"ppTime":1507742099,"txnRootHash":null}' to Node1 2017-10-11 17:14:59,466 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":null,"viewNo":1,"digest":"a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b","ppSeqNo":2,"op":"PREPARE","instId":1,"ppTime":1507742099,"txnRootHash":null}' to Node1 2017-10-11 17:14:59,467 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":null,"viewNo":1,"digest":"a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b","ppSeqNo":2,"op":"PREPARE","instId":1,"ppTime":1507742099,"txnRootHash":null}' to Node3 2017-10-11 17:14:59,467 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":null,"viewNo":1,"digest":"a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b","ppSeqNo":2,"op":"PREPARE","instId":1,"ppTime":1507742099,"txnRootHash":null}' to Node3 2017-10-11 17:14:59,467 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"stateRootHash":null,"viewNo":1,"digest":"a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b","ppSeqNo":2,"op":"PREPARE","instId":1,"ppTime":1507742099,"txnRootHash":null}' to Node4 2017-10-11 17:14:59,467 | TRACE | batched.py ( 96) | flushOutBoxes | Node2 sending msg b'{"stateRootHash":null,"viewNo":1,"digest":"a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b","ppSeqNo":2,"op":"PREPARE","instId":1,"ppTime":1507742099,"txnRootHash":null}' to Node4 2017-10-11 17:14:59,479 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 2 messages through listener 2017-10-11 17:14:59,480 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: BATCH{'messages': ['{"viewNo":1,"digest":"a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b","ppTime":1507742099,"instId":0,"ppSeqNo":2,"txnRootHash":"GavLfumt6ugt9aC1DU8Q27QC7KyKz3UWHVKLejGJPsSb","stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","op":"PREPARE"}', '{"viewNo":1,"digest":"a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b","ppTime":1507742099,"instId":1,"ppSeqNo":2,"txnRootHash":null,"stateRootHash":null,"op":"PREPARE"}'], 'signature': None} 2017-10-11 17:14:59,480 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"viewNo":1,"digest":"a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b","ppTime":1507742099,"instId":0,"ppSeqNo":2,"txnRootHash":"GavLfumt6ugt9aC1DU8Q27QC7KyKz3UWHVKLejGJPsSb","stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","op":"PREPARE"}', '{"viewNo":1,"digest":"a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b","ppTime":1507742099,"instId":1,"ppSeqNo":2,"txnRootHash":null,"stateRootHash":null,"op":"PREPARE"}'], 'op': 'BATCH', 'signature': None}, 'Node1') 2017-10-11 17:14:59,480 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"viewNo":1,"digest":"a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b","ppTime":1507742099,"instId":0,"ppSeqNo":2,"txnRootHash":"GavLfumt6ugt9aC1DU8Q27QC7KyKz3UWHVKLejGJPsSb","stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","op":"PREPARE"}', '{"viewNo":1,"digest":"a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b","ppTime":1507742099,"instId":1,"ppSeqNo":2,"txnRootHash":null,"stateRootHash":null,"op":"PREPARE"}'], 'signature': None} 2017-10-11 17:14:59,480 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PREPARE{'digest': 'a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b', 'ppSeqNo': 2, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': 'GavLfumt6ugt9aC1DU8Q27QC7KyKz3UWHVKLejGJPsSb', 'instId': 0, 'ppTime': 1507742099, 'viewNo': 1} 2017-10-11 17:14:59,481 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'digest': 'a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b', 'ppSeqNo': 2, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'op': 'PREPARE', 'txnRootHash': 'GavLfumt6ugt9aC1DU8Q27QC7KyKz3UWHVKLejGJPsSb', 'instId': 0, 'ppTime': 1507742099, 'viewNo': 1}, 'Node1') 2017-10-11 17:14:59,481 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'digest': 'a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b', 'ppSeqNo': 2, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': 'GavLfumt6ugt9aC1DU8Q27QC7KyKz3UWHVKLejGJPsSb', 'instId': 0, 'ppTime': 1507742099, 'viewNo': 1} 2017-10-11 17:14:59,481 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: PREPARE{'digest': 'a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b', 'ppSeqNo': 2, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507742099, 'viewNo': 1} 2017-10-11 17:14:59,481 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'digest': 'a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b', 'ppSeqNo': 2, 'stateRootHash': None, 'op': 'PREPARE', 'txnRootHash': None, 'instId': 1, 'ppTime': 1507742099, 'viewNo': 1}, 'Node1') 2017-10-11 17:14:59,481 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'digest': 'a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b', 'ppSeqNo': 2, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507742099, 'viewNo': 1} 2017-10-11 17:14:59,481 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: PREPARE{'digest': 'a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b', 'ppSeqNo': 2, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': 'GavLfumt6ugt9aC1DU8Q27QC7KyKz3UWHVKLejGJPsSb', 'instId': 0, 'ppTime': 1507742099, 'viewNo': 1} 2017-10-11 17:14:59,482 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'digest': 'a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b', 'ppSeqNo': 2, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'op': 'PREPARE', 'txnRootHash': 'GavLfumt6ugt9aC1DU8Q27QC7KyKz3UWHVKLejGJPsSb', 'instId': 0, 'ppTime': 1507742099, 'viewNo': 1}, 'Node3') 2017-10-11 17:14:59,482 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'digest': 'a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b', 'ppSeqNo': 2, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': 'GavLfumt6ugt9aC1DU8Q27QC7KyKz3UWHVKLejGJPsSb', 'instId': 0, 'ppTime': 1507742099, 'viewNo': 1} 2017-10-11 17:14:59,483 | DEBUG | replica.py ( 917) | tryCommit | Node2:0 cannot send COMMIT since does not have prepare quorum for PREPARE{'digest': 'a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b', 'ppSeqNo': 2, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': 'GavLfumt6ugt9aC1DU8Q27QC7KyKz3UWHVKLejGJPsSb', 'instId': 0, 'ppTime': 1507742099, 'viewNo': 1} 2017-10-11 17:14:59,483 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (1, 2) 2017-10-11 17:14:59,483 | DEBUG | replica.py ( 953) | doCommit | Node2:0 Sending COMMIT(1, 2) at 2271184.940860836 2017-10-11 17:14:59,483 | DEBUG | replica.py (2193) | send | Node2:0 sending Commit 2017-10-11 17:14:59,483 | TRACE | replica.py (2194) | send | Node2:0 sending COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 1} 2017-10-11 17:14:59,483 | DEBUG | replica.py ( 929) | tryOrder | Node2:0 cannot return request to node: no quorum (3): COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 1} commits where f is 1 2017-10-11 17:14:59,483 | DEBUG | replica.py ( 876) | processPrepare | Node2:0 processed incoming PREPARE (1, 2) 2017-10-11 17:14:59,484 | DEBUG | replica.py ( 953) | doCommit | Node2:1 Sending COMMIT(1, 2) at 2271184.941501694 2017-10-11 17:14:59,484 | DEBUG | replica.py (2193) | send | Node2:1 sending Commit 2017-10-11 17:14:59,484 | TRACE | replica.py (2194) | send | Node2:1 sending COMMIT{'instId': 1, 'ppSeqNo': 2, 'viewNo': 1} 2017-10-11 17:14:59,484 | DEBUG | replica.py ( 929) | tryOrder | Node2:1 cannot return request to node: no quorum (3): COMMIT{'instId': 1, 'ppSeqNo': 2, 'viewNo': 1} commits where f is 1 2017-10-11 17:14:59,484 | DEBUG | replica.py ( 876) | processPrepare | Node2:1 processed incoming PREPARE (1, 2) 2017-10-11 17:14:59,484 | DEBUG | node.py (2608) | send | Node2 sending message COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 1} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:14:59,484 | DEBUG | node.py (2608) | send | Node2 sending message COMMIT{'instId': 1, 'ppSeqNo': 2, 'viewNo': 1} to all recipients: ['Node1', 'Node3', 'Node4'] 2017-10-11 17:14:59,485 | TRACE | zstack.py ( 479) | _receiveFromListener | Node2 got 3 messages through listener 2017-10-11 17:14:59,485 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node3: COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 1} 2017-10-11 17:14:59,486 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'ppSeqNo': 2, 'op': 'COMMIT', 'viewNo': 1}, 'Node3') 2017-10-11 17:14:59,486 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 1} 2017-10-11 17:14:59,490 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node1: COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 1} 2017-10-11 17:14:59,490 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'instId': 0, 'ppSeqNo': 2, 'op': 'COMMIT', 'viewNo': 1}, 'Node1') 2017-10-11 17:14:59,491 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox COMMIT{'instId': 0, 'ppSeqNo': 2, 'viewNo': 1} 2017-10-11 17:14:59,492 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: BATCH{'messages': ['{"txnRootHash":"GavLfumt6ugt9aC1DU8Q27QC7KyKz3UWHVKLejGJPsSb","ppSeqNo":2,"digest":"a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b","op":"PREPARE","viewNo":1,"ppTime":1507742099,"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","instId":0}', '{"txnRootHash":null,"ppSeqNo":2,"digest":"a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b","op":"PREPARE","viewNo":1,"ppTime":1507742099,"stateRootHash":null,"instId":1}'], 'signature': None} 2017-10-11 17:14:59,492 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'messages': ['{"txnRootHash":"GavLfumt6ugt9aC1DU8Q27QC7KyKz3UWHVKLejGJPsSb","ppSeqNo":2,"digest":"a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b","op":"PREPARE","viewNo":1,"ppTime":1507742099,"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","instId":0}', '{"txnRootHash":null,"ppSeqNo":2,"digest":"a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b","op":"PREPARE","viewNo":1,"ppTime":1507742099,"stateRootHash":null,"instId":1}'], 'signature': None, 'op': 'BATCH'}, 'Node4') 2017-10-11 17:14:59,492 | DEBUG | node.py (1272) | unpackNodeMsg | Node2 processing a batch BATCH{'messages': ['{"txnRootHash":"GavLfumt6ugt9aC1DU8Q27QC7KyKz3UWHVKLejGJPsSb","ppSeqNo":2,"digest":"a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b","op":"PREPARE","viewNo":1,"ppTime":1507742099,"stateRootHash":"DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA","instId":0}', '{"txnRootHash":null,"ppSeqNo":2,"digest":"a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b","op":"PREPARE","viewNo":1,"ppTime":1507742099,"stateRootHash":null,"instId":1}'], 'signature': None} 2017-10-11 17:14:59,495 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'digest': 'a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b', 'ppSeqNo': 2, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': 'GavLfumt6ugt9aC1DU8Q27QC7KyKz3UWHVKLejGJPsSb', 'instId': 0, 'ppTime': 1507742099, 'viewNo': 1} 2017-10-11 17:14:59,496 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'ppSeqNo': 2, 'digest': 'a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b', 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': 'GavLfumt6ugt9aC1DU8Q27QC7KyKz3UWHVKLejGJPsSb', 'op': 'PREPARE', 'instId': 0, 'ppTime': 1507742099, 'viewNo': 1}, 'Node4') 2017-10-11 17:14:59,496 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'digest': 'a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b', 'ppSeqNo': 2, 'stateRootHash': 'DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA', 'txnRootHash': 'GavLfumt6ugt9aC1DU8Q27QC7KyKz3UWHVKLejGJPsSb', 'instId': 0, 'ppTime': 1507742099, 'viewNo': 1} 2017-10-11 17:14:59,496 | DEBUG | node.py (1257) | validateNodeMsg | Node2 received node message from Node4: PREPARE{'digest': 'a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b', 'ppSeqNo': 2, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507742099, 'viewNo': 1} 2017-10-11 17:14:59,496 | DEBUG | node.py (1219) | handleOneNodeMsg | Node2 msg validated ({'ppSeqNo': 2, 'digest': 'a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b', 'stateRootHash': None, 'txnRootHash': None, 'op': 'PREPARE', 'instId': 1, 'ppTime': 1507742099, 'viewNo': 1}, 'Node4') 2017-10-11 17:14:59,496 | DEBUG | node.py (1286) | postToNodeInBox | Node2 appending to nodeInbox PREPARE{'digest': 'a531031e988f0aab87b0281ff9a676a3a5aea71d899c87bf7fe18dfad945881b', 'ppSeqNo': 2, 'stateRootHash': None, 'txnRootHash': None, 'instId': 1, 'ppTime': 1507742099, 'viewNo': 1} 2017-10-11 17:14:59,497 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node1 into one transmission 2017-10-11 17:14:59,500 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"instId":0,"ppSeqNo":2,"op":"COMMIT","viewNo":1}', b'{"instId":1,"ppSeqNo":2,"op":"COMMIT","viewNo":1}']) 2017-10-11 17:14:59,500 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node1: b'{"messages":["{\\"instId\\":0,\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1}","{\\"instId\\":1,\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:14:59,500 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"instId\\":0,\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1}","{\\"instId\\":1,\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1}"],"signature":null,"op":"BATCH"}' to Node1 2017-10-11 17:14:59,501 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node3 into one transmission 2017-10-11 17:14:59,501 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"instId":0,"ppSeqNo":2,"op":"COMMIT","viewNo":1}', b'{"instId":1,"ppSeqNo":2,"op":"COMMIT","viewNo":1}']) 2017-10-11 17:14:59,501 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node3: b'{"messages":["{\\"instId\\":0,\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1}","{\\"instId\\":1,\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:14:59,501 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"instId\\":0,\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1}","{\\"instId\\":1,\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1}"],"signature":null,"op":"BATCH"}' to Node3 2017-10-11 17:14:59,509 | DEBUG | batched.py ( 100) | flushOutBoxes | Node2 batching 2 msgs to Node4 into one transmission 2017-10-11 17:14:59,510 | TRACE | batched.py ( 101) | flushOutBoxes | messages: deque([b'{"instId":0,"ppSeqNo":2,"op":"COMMIT","viewNo":1}', b'{"instId":1,"ppSeqNo":2,"op":"COMMIT","viewNo":1}']) 2017-10-11 17:14:59,510 | TRACE | batched.py ( 110) | flushOutBoxes | Node2 sending payload to Node4: b'{"messages":["{\\"instId\\":0,\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1}","{\\"instId\\":1,\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1}"],"signature":null,"op":"BATCH"}' 2017-10-11 17:14:59,510 | DEBUG | zstack.py ( 723) | transmit | Node2 transmitting message b'{"messages":["{\\"instId\\":0,\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1}","{\\"instId\\":1,\\"ppSeqNo\\":2,\\"op\\":\\"COMMIT\\",\\"viewNo\\":1}"],"signature":null,"op":"BATCH"}' to Node4 2017-10-11 17:14:59,511 | DEBUG | replica.py ( 894) | processCommit | Node2:0 received COMMIT(1, 2) from Node3:0 2017