2017-11-17 13:43:44,267 | INFO     | node.py              (2420) | initStateFromLedger | Node2 found state to be empty, recreating from ledger
2017-11-17 13:43:44,350 | INFO     | node.py              (2420) | initStateFromLedger | Node2 found state to be empty, recreating from ledger
2017-11-17 13:43:44,396 | INFO     | plugin_loader.py     ( 117) | _load | plugin FirebaseStatsConsumer successfully loaded from module plugin_firebase_stats_consumer
2017-11-17 13:43:44,398 | DISPLAY  | replicas.py          (  36) | grow | Node2 added replica Node2:0 to instance 0 (master)
2017-11-17 13:43:44,399 | DISPLAY  | replicas.py          (  36) | grow | Node2 added replica Node2:1 to instance 1 (backup)
2017-11-17 13:43:44,487 | INFO     | node.py              (2420) | initStateFromLedger | Node2 found state to be empty, recreating from ledger
2017-11-17 13:43:44,489 | INFO     | stacks.py            (  84) | start | CONNECTION: Node2 listening for other nodes at 0.0.0.0:9703
2017-11-17 13:43:44,489 | INFO     | node.py              ( 621) | start | Node2 first time running...
2017-11-17 13:43:44,490 | INFO     | zstack.py            ( 590) | connect | CONNECTION: Node2 looking for Node1 at 10.0.0.2:9701
2017-11-17 13:43:44,492 | INFO     | zstack.py            ( 590) | connect | CONNECTION: Node2 looking for Node4 at 10.0.0.5:9707
2017-11-17 13:43:44,492 | INFO     | zstack.py            ( 590) | connect | CONNECTION: Node2 looking for Node3 at 10.0.0.4:9705
2017-11-17 13:43:44,572 | INFO     | keep_in_touch.py     (  98) | _connsChanged | CONNECTION: Node2 now connected to Node1
2017-11-17 13:43:44,572 | INFO     | keep_in_touch.py     (  98) | _connsChanged | CONNECTION: Node2 now connected to Node3
2017-11-17 13:43:44,597 | INFO     | node.py              (1489) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 0
2017-11-17 13:43:44,597 | INFO     | ledger_manager.py    ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 0, caught up 0 in total
2017-11-17 13:43:44,623 | INFO     | node.py              (1489) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 2
2017-11-17 13:43:44,624 | INFO     | upgrader.py          ( 151) | should_notify_about_upgrade_result | Node's 'Node2' last upgrade txn is None
2017-11-17 13:43:44,624 | INFO     | ledger_manager.py    ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 2, caught up 0 in total
2017-11-17 13:43:44,652 | INFO     | node.py              (1489) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 1
2017-11-17 13:43:44,652 | INFO     | ledger_manager.py    ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 1, caught up 0 in total
2017-11-17 13:43:44,652 | INFO     | node.py              (1537) | allLedgersCaughtUp | CATCH-UP: Node2 does not need any more catchups
2017-11-17 13:43:44,680 | DISPLAY  | primary_selector.py  ( 291) | _startSelection | PRIMARY SELECTION: Node2:0 selected primary Node1:0 for instance 0 (view 0)
2017-11-17 13:43:44,680 | INFO     | node.py              ( 510) | start_participating | Node2 started participating
2017-11-17 13:43:44,681 | DISPLAY  | primary_selector.py  ( 315) | _startSelection | VIEW CHANGE: Node2:0 declares view change 0 as completed for instance 0, new primary is Node1:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')]
2017-11-17 13:43:44,681 | DISPLAY  | primary_selector.py  ( 291) | _startSelection | PRIMARY SELECTION: Node2:1 selected primary Node2:1 for instance 1 (view 0)
2017-11-17 13:43:44,681 | DISPLAY  | primary_selector.py  ( 315) | _startSelection | VIEW CHANGE: Node2:1 declares view change 0 as completed for instance 1, new primary is Node2:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')]
2017-11-17 13:43:45,137 | INFO     | keep_in_touch.py     (  98) | _connsChanged | CONNECTION: Node2 now connected to Node4
2017-11-17 13:45:38,705 | INFO     | message_processor.py (  29) | discard | Node2 discarding message ({'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'KNpuf1rjcogYVHpD982n5b3tY527KEDAdKDbuVfSDTY95x1jUkULk3buoEzHuGj6KLjfHSNN6wcdkF9dfztPSHe', 'operation': {'action': 'start', 'justification': None, 'reinstall': False, 'force': False, 'name': 'upgrade_to_the_latest_full', 'type': '109', 'timeout': 10, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-11-17T13:50:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-11-17T13:55:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-11-17T14:00:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-11-17T13:45:00.258870+00:00'}, 'version': '1.2.45'}, 'reqId': 1510926338687540}, b'=cQu#Y-Ir.lyHj/JuuLEhk!lc&(9BYFL7fT8TyD}') because InvalidClientRequest("{'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-11-17T13:50:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-11-17T13:55:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-11-17T14:00:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-11-17T13:45:00.258870+00:00'} not a valid schedule since 2017-11-17 13:45:00.258870+00:00 is less than current time",)
2017-11-17 13:46:06,339 | INFO     | propagator.py        ( 148) | propagate | Node2 propagating request ('V4SGRU86Z58d6TV7PBUe6f', 1510926366313517) from client b'=cQu#Y-Ir.lyHj/JuuLEhk!lc&(9BYFL7fT8TyD}'
2017-11-17 13:46:06,401 | INFO     | replica.py           (1483) | order_3pc_key | Node2:1 ordered batch request, view no 0, ppSeqNo 1, ledger 2, state root None, txn root None, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1510926366313517)], discarded []
2017-11-17 13:46:06,409 | INFO     | replica.py           (1483) | order_3pc_key | Node2:0 ordered batch request, view no 0, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root XjH6SHaWYaxmk1gyUBigBTfZctf6HT46vK3WgUhceQG, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1510926366313517)], discarded []
2017-11-17 13:46:06,411 | INFO     | upgrader.py          ( 292) | handleUpgradeTxn | Node 'Node2' handles upgrade txn {'seqNo': 1, 'justification': None, 'auditPath': [], 'name': 'upgrade_to_the_latest_full', 'type': '109', 'timeout': 10, 'rootHash': 'XjH6SHaWYaxmk1gyUBigBTfZctf6HT46vK3WgUhceQG', 'reqId': 1510926366313517, 'action': 'start', 'reinstall': False, 'txnTime': 1510926366, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj', 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-11-17T13:52:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-11-17T13:57:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-11-17T13:47:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-11-17T14:02:00.258870+00:00'}, 'version': '1.2.45', 'force': False}
2017-11-17 13:46:06,411 | INFO     | upgrader.py          ( 321) | handleUpgradeTxn | Node 'Node2' schedules upgrade to 1.2.45
2017-11-17 13:46:06,412 | INFO     | upgrader.py          ( 359) | _scheduleUpgrade | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb's upgrader processing upgrade for version 1.2.45
2017-11-17 13:46:06,412 | INFO     | node.py              (2290) | executeBatch | Node2 committed batch request, view no 0, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root XjH6SHaWYaxmk1gyUBigBTfZctf6HT46vK3WgUhceQG, requests: [('V4SGRU86Z58d6TV7PBUe6f', 1510926366313517)]
2017-11-17 13:46:59,434 | INFO     | propagator.py        ( 148) | propagate | Node2 propagating request ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926419422000) from client Node1
2017-11-17 13:46:59,466 | INFO     | replica.py           (1483) | order_3pc_key | Node2:0 ordered batch request, view no 0, ppSeqNo 2, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 2PDWCTTViFKEEVQvxGsg6s7VZm4rCmmBu8G1VLJhEdom, requests ordered [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926419422000)], discarded []
2017-11-17 13:46:59,467 | INFO     | replica.py           (1483) | order_3pc_key | Node2:1 ordered batch request, view no 0, ppSeqNo 2, ledger 2, state root None, txn root None, requests ordered [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926419422000)], discarded []
2017-11-17 13:46:59,468 | INFO     | node.py              (2290) | executeBatch | Node2 committed batch request, view no 0, ppSeqNo 2, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 2PDWCTTViFKEEVQvxGsg6s7VZm4rCmmBu8G1VLJhEdom, requests: [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926419422000)]
2017-11-17 13:47:30,420 | INFO     | keep_in_touch.py     (  93) | _connsChanged | CONNECTION: Node2 disconnected from Node1
2017-11-17 13:47:30,420 | INFO     | node.py              ( 823) | onConnsChanged | Node2 lost connection to primary of master
2017-11-17 13:47:32,421 | INFO     | node.py              (2048) | sendInstanceChange | VIEW CHANGE: Node2 sending an instance change with view_no 1 since Primary of master protocol instance disconnected
2017-11-17 13:47:32,422 | INFO     | node.py              (2051) | sendInstanceChange | MONITORING: Node2 metrics for monitor: Node2 Monitor metrics:: None
            Delta: 0.4
            Lambda: 60
            Omega: 5
            instances started: [1381072.607847083, 1381072.608642995]
            ordered request counts: {0: 2, 1: 2}
            ordered request durations: {0: 0.08592366566881537, 1: 0.07085515977814794}
            master request latencies: {}
            client avg request latencies: [{'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': (1, 0.02899836585856974), 'V4SGRU86Z58d6TV7PBUe6f': (1, 0.05692529981024563)}, {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': (1, 0.02907823189161718), 'V4SGRU86Z58d6TV7PBUe6f': (1, 0.04177692788653076)}]
            throughput: {0: 23.276474350021452, 1: 28.22659642942206}
            master throughput: 23.276474350021452
            total requests: 2
            avg backup throughput: 28.22659642942206
            master throughput ratio: 0.824629154571366
2017-11-17 13:47:32,422 | INFO     | node.py              (2108) | propose_view_change | Node2 sent view change since was disconnected from primary for too long
2017-11-17 13:47:32,435 | INFO     | node.py              (1954) | do_view_change_if_possible | VIEW CHANGE: Node2 initiating a view change to 1 from 0
2017-11-17 13:47:32,435 | INFO     | node.py              (2159) | startViewChange | VIEW CHANGE: Node2 changed to view 1, will start catchup now
2017-11-17 13:47:32,436 | INFO     | message_processor.py (  29) | discard | Node2 discarding message INSTANCE_CHANGE{'viewNo': 1, 'reason': 26} because Received instance change request with view no 1 which is not more than its view no 1
2017-11-17 13:47:32,464 | INFO     | node.py              (1489) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 0
2017-11-17 13:47:32,465 | INFO     | ledger_manager.py    ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 0, caught up 0 in total
2017-11-17 13:47:32,493 | INFO     | node.py              (1489) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 2
2017-11-17 13:47:32,494 | INFO     | upgrader.py          ( 214) | processLedger | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb found upgrade START txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)])
2017-11-17 13:47:32,494 | INFO     | upgrader.py          ( 292) | handleUpgradeTxn | Node 'Node2' handles upgrade txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)])
2017-11-17 13:47:32,494 | INFO     | upgrader.py          ( 321) | handleUpgradeTxn | Node 'Node2' schedules upgrade to 1.2.45
2017-11-17 13:47:32,494 | INFO     | upgrader.py          ( 326) | handleUpgradeTxn | Node 'Node2' cancels previous upgrade and schedules a new one to 1.2.45
2017-11-17 13:47:32,494 | INFO     | upgrader.py          ( 390) | _cancelScheduledUpgrade | Cancelling upgrade of node 'Node2' to version 1.2.45 due to some reason
2017-11-17 13:47:32,494 | INFO     | upgrader.py          ( 359) | _scheduleUpgrade | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb's upgrader processing upgrade for version 1.2.45
2017-11-17 13:47:32,495 | INFO     | upgrader.py          ( 151) | should_notify_about_upgrade_result | Node's 'Node2' last upgrade txn is None
2017-11-17 13:47:32,496 | INFO     | ledger_manager.py    ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 2, caught up 0 in total
2017-11-17 13:47:32,511 | INFO     | node.py              (1489) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 1
2017-11-17 13:47:32,511 | INFO     | ledger_manager.py    ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 1, caught up 0 in total
2017-11-17 13:47:32,512 | INFO     | node.py              (1537) | allLedgersCaughtUp | CATCH-UP: Node2 does not need any more catchups
2017-11-17 13:47:32,526 | DISPLAY  | primary_selector.py  ( 291) | _startSelection | PRIMARY SELECTION: Node2:0 selected primary Node2:0 for instance 0 (view 1)
2017-11-17 13:47:32,526 | INFO     | node.py              ( 510) | start_participating | Node2 started participating
2017-11-17 13:47:32,526 | DISPLAY  | primary_selector.py  ( 315) | _startSelection | VIEW CHANGE: Node2:0 declares view change 1 as completed for instance 0, new primary is Node2:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 2, '2PDWCTTViFKEEVQvxGsg6s7VZm4rCmmBu8G1VLJhEdom')]
2017-11-17 13:47:32,526 | DISPLAY  | primary_selector.py  ( 291) | _startSelection | PRIMARY SELECTION: Node2:1 selected primary Node3:1 for instance 1 (view 1)
2017-11-17 13:47:32,527 | DISPLAY  | primary_selector.py  ( 315) | _startSelection | VIEW CHANGE: Node2:1 declares view change 1 as completed for instance 1, new primary is Node3:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 2, '2PDWCTTViFKEEVQvxGsg6s7VZm4rCmmBu8G1VLJhEdom')]
2017-11-17 13:47:44,612 | INFO     | keep_in_touch.py     (  98) | _connsChanged | CONNECTION: Node2 now connected to Node1
2017-11-17 13:47:44,666 | INFO     | message_processor.py (  29) | discard | Node2 discarding message {'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'operation': {'type': '110', 'data': {'action': 'complete', 'version': '1.2.45'}, 'signature': '5Rjt3r36mqM669ZoEcccqa6zJU5UEdxDLrfZD3UAJrWeN3hbvPCsVnzqmUootfcgMsfm43MzMkq1vXcXyveN6VjN'}, 'reqId': 1510926464650411, 'signature': '485QNuqefeLN4d7RrAyUYtK7egvRBsMhzADLYScsLmJivg8FKytaWwbPnj82pxd6FrSbch5znp8M5bn1gsRwraFn', 'protocolVersion': 1} because 'SafeRequest' object has no attribute 'schema_is_strict'
2017-11-17 13:51:59,501 | INFO     | upgrader.py          ( 421) | _callUpgradeAgent | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb's upgrader calling agent for upgrade
2017-11-17 13:51:59,501 | INFO     | node.py              ( 254) | notify_upgrade_start | Node2 is about to be upgraded, sending NODE_UPGRADE
2017-11-17 13:51:59,505 | INFO     | upgrader.py          ( 433) | _sendUpdateRequest | Sending message to control tool: {"version": "1.2.45"}
2017-11-17 13:51:59,507 | INFO     | upgrader.py          ( 452) | _sendUpdateRequest | Waiting 10 minutes for upgrade to be performed
2017-11-17 13:51:59,522 | INFO     | propagator.py        ( 148) | propagate | Node2 propagating request ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926719502400) from client Node2
2017-11-17 13:51:59,543 | INFO     | replica.py           (1483) | order_3pc_key | Node2:1 ordered batch request, view no 1, ppSeqNo 1, ledger 2, state root None, txn root None, requests ordered [('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926719502400)], discarded []
2017-11-17 13:51:59,546 | INFO     | replica.py           (1483) | order_3pc_key | Node2:0 ordered batch request, view no 1, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root BvBE3KNL7XqXP3gkNvaS9mSi1veDkzTj32rY5mpWBQbk, requests ordered [('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926719502400)], discarded []
2017-11-17 13:51:59,547 | INFO     | node.py              (2290) | executeBatch | Node2 committed batch request, view no 1, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root BvBE3KNL7XqXP3gkNvaS9mSi1veDkzTj32rY5mpWBQbk, requests: [('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926719502400)]
2017-11-17 13:52:32,632 | INFO     | node.py              (2503) | initStateFromLedger | Node2 found state to be empty, recreating from ledger
2017-11-17 13:52:32,635 | INFO     | pool_manager.py      ( 409) | _order_node | Node2 node Node1 ordered, NYM Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv
2017-11-17 13:52:32,635 | INFO     | pool_manager.py      ( 409) | _order_node | Node2 node Node2 ordered, NYM 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb
2017-11-17 13:52:32,635 | INFO     | pool_manager.py      ( 409) | _order_node | Node2 node Node3 ordered, NYM DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya
2017-11-17 13:52:32,635 | INFO     | pool_manager.py      ( 409) | _order_node | Node2 node Node4 ordered, NYM 4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA
2017-11-17 13:52:32,650 | WARNING  | node.py              ( 623) | _create_bls_bft | BLS: Transactions will not be BLS signed by this Node, since BLS keys were not found. Please make sure that a script to init BLS keys was called (init_bls_keys), and NODE txn was sent with BLS public keys.
2017-11-17 13:52:32,656 | INFO     | node.py              (2503) | initStateFromLedger | Node2 found state to be empty, recreating from ledger
2017-11-17 13:52:32,668 | INFO     | node.py              ( 479) | setPoolParams | Node2 updated its pool parameters: f 1, totalNodes 4, allNodeNames {'Node1', 'Node2', 'Node3', 'Node4'}, requiredNumberOfInstances 2, minimumNodes 3, quorums {'propagate_primary': Quorum(2), 'propagate': Quorum(2), 'commit': Quorum(3), 'view_change_done': Quorum(3), 'f': 1, 'consistency_proof': Quorum(2), 'election': Quorum(3), 'ledger_status': Quorum(2), 'view_change': Quorum(3), 'same_consistency_proof': Quorum(2), 'reply': Quorum(2), 'timestamp': Quorum(2), 'checkpoint': Quorum(2), 'bls_signatures': Quorum(3), 'prepare': Quorum(2)}
2017-11-17 13:52:32,705 | INFO     | plugin_loader.py     ( 117) | _load | plugin FirebaseStatsConsumer successfully loaded from module plugin_firebase_stats_consumer
2017-11-17 13:52:32,706 | DISPLAY  | replicas.py          (  40) | grow | Node2 added replica Node2:0 to instance 0 (master)
2017-11-17 13:52:32,707 | DISPLAY  | replicas.py          (  40) | grow | Node2 added replica Node2:1 to instance 1 (backup)
2017-11-17 13:52:32,748 | INFO     | upgrader.py          ( 140) | check_upgrade_succeeded | Node 'Node2' successfully upgraded to version 1.2.45
2017-11-17 13:52:32,748 | INFO     | node.py              (2503) | initStateFromLedger | Node2 found state to be empty, recreating from ledger
2017-11-17 13:52:32,750 | INFO     | stacks.py            (  84) | start | CONNECTION: Node2 listening for other nodes at 0.0.0.0:9703
2017-11-17 13:52:32,750 | INFO     | node.py              ( 680) | start | Node2 first time running...
2017-11-17 13:52:32,751 | INFO     | zstack.py            ( 585) | connect | CONNECTION: Node2 looking for Node1 at 10.0.0.2:9701
2017-11-17 13:52:32,752 | INFO     | zstack.py            ( 585) | connect | CONNECTION: Node2 looking for Node4 at 10.0.0.5:9707
2017-11-17 13:52:32,753 | INFO     | zstack.py            ( 585) | connect | CONNECTION: Node2 looking for Node3 at 10.0.0.4:9705
2017-11-17 13:52:32,819 | INFO     | keep_in_touch.py     (  98) | _connsChanged | CONNECTION: Node2 now connected to Node3
2017-11-17 13:52:32,845 | INFO     | node.py              (2044) | _start_view_change_if_possible | VIEW CHANGE: Node2 starting view change for 2 after 2 view change indications from other nodes
2017-11-17 13:52:32,846 | INFO     | node.py              (2241) | startViewChange | VIEW CHANGE: Node2 changed to view 2, will start catchup now
2017-11-17 13:52:32,847 | INFO     | keep_in_touch.py     (  98) | _connsChanged | CONNECTION: Node2 now connected to Node4
2017-11-17 13:52:32,862 | INFO     | node.py              (1551) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 0
2017-11-17 13:52:32,862 | INFO     | ledger_manager.py    ( 865) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 0, caught up 0 in total
2017-11-17 13:52:32,863 | INFO     | node.py              (1551) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 2
2017-11-17 13:52:32,863 | INFO     | upgrader.py          ( 212) | processLedger | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb found upgrade START txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)])
2017-11-17 13:52:32,864 | INFO     | upgrader.py          ( 290) | handleUpgradeTxn | Node 'Node2' handles upgrade txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)])
2017-11-17 13:52:32,864 | INFO     | upgrader.py          ( 310) | handleUpgradeTxn | Node 'Node2' has already performed an upgrade with upgrade_id 15109263663135171. Last recorded event is ('succeeded', datetime.datetime(2017, 11, 17, 13, 52, 0, 258870, tzinfo=tzlocal()), '1.2.45', '15109263663135171')
2017-11-17 13:52:32,864 | INFO     | node.py              (1551) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 1
2017-11-17 13:52:32,865 | INFO     | ledger_manager.py    ( 865) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 1, caught up 0 in total
2017-11-17 13:52:32,865 | INFO     | node.py              (1588) | allLedgersCaughtUp | CATCH-UP: Node2 caught up till (1, 1)
2017-11-17 13:52:32,866 | INFO     | node.py              (1598) | allLedgersCaughtUp | CATCH-UP: Node2 does not need any more catchups
2017-11-17 13:52:32,866 | DISPLAY  | primary_selector.py  ( 327) | _start_selection | PRIMARY SELECTION: Node2:0 selected primary Node3:0 for instance 0 (view 2)
2017-11-17 13:52:32,866 | INFO     | node.py              ( 532) | start_participating | Node2 started participating
2017-11-17 13:52:32,867 | DISPLAY  | primary_selector.py  ( 351) | _start_selection | VIEW CHANGE: Node2:0 declares view change 2 as completed for instance 0, new primary is Node3:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 3, 'BvBE3KNL7XqXP3gkNvaS9mSi1veDkzTj32rY5mpWBQbk')]
2017-11-17 13:52:32,867 | DISPLAY  | primary_selector.py  ( 327) | _start_selection | PRIMARY SELECTION: Node2:1 selected primary Node4:1 for instance 1 (view 2)
2017-11-17 13:52:32,870 | DISPLAY  | primary_selector.py  ( 351) | _start_selection | VIEW CHANGE: Node2:1 declares view change 2 as completed for instance 1, new primary is Node4:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 3, 'BvBE3KNL7XqXP3gkNvaS9mSi1veDkzTj32rY5mpWBQbk')]
2017-11-17 13:52:32,872 | INFO     | upgrader.py          ( 149) | should_notify_about_upgrade_result | Node's 'Node2' last upgrade txn is OrderedDict([('data', OrderedDict([('action', 'in_progress'), ('version', '1.2.45')])), ('identifier', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb'), ('reqId', 1510926719502400), ('signature', '59fs2tWoGeAcMnUon3maEHpzyNZhNULgk4zctNgfGRSB3AE7D3ciyXpicauLCRwpKzzknWaQhK2V1Btj73K6B2Kp'), ('txnTime', 1510926719), ('type', '110'), ('seqNo', 3)])
2017-11-17 13:52:32,873 | INFO     | ledger_manager.py    ( 865) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 2, caught up 0 in total
2017-11-17 13:52:32,874 | INFO     | node.py              (1598) | allLedgersCaughtUp | CATCH-UP: Node2 does not need any more catchups
2017-11-17 13:52:32,874 | INFO     | node.py              (1685) | no_more_catchups_needed | Node2 starting to participate since catchup is done, primaries are selected but mode was not set to participating
2017-11-17 13:52:32,874 | INFO     | node.py              ( 532) | start_participating | Node2 started participating
2017-11-17 13:52:32,920 | INFO     | message_processor.py (  29) | discard | Node2 discarding message INSTANCE_CHANGE{'viewNo': 2, 'reason': 26} because Received instance change request with view no 2 which is not more than its view no 2
2017-11-17 13:52:32,922 | INFO     | propagator.py        ( 179) | propagate | Node2 propagating request ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926748095846) from client Node1
2017-11-17 13:52:32,925 | INFO     | propagator.py        ( 179) | propagate | Node2 propagating request ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926752873422) from client Node2
2017-11-17 13:52:32,927 | INFO     | keep_in_touch.py     (  98) | _connsChanged | CONNECTION: Node2 now connected to Node1
2017-11-17 13:57:00,104 | INFO     | propagator.py        ( 179) | propagate | Node2 propagating request ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927020097708) from client Node3
2017-11-17 13:57:00,152 | INFO     | replica.py           (1619) | order_3pc_key | Node2:0 ordered batch request, view no 2, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 3aErFDAegiipoxjKyq8YBzyNpDPHaoNhZ8BqVu4e3uCg, requests ordered [('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927020097708)], discarded []
2017-11-17 13:57:00,153 | INFO     | replica.py           (1619) | order_3pc_key | Node2:1 ordered batch request, view no 2, ppSeqNo 1, ledger 2, state root None, txn root None, requests ordered [('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927020097708)], discarded []
2017-11-17 13:57:00,155 | INFO     | node.py              (2374) | executeBatch | Node2 committed batch request, view no 2, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 3aErFDAegiipoxjKyq8YBzyNpDPHaoNhZ8BqVu4e3uCg, requests: [('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927020097708)]
2017-11-17 13:57:25,919 | INFO     | keep_in_touch.py     (  93) | _connsChanged | CONNECTION: Node2 disconnected from Node3
2017-11-17 13:57:25,919 | INFO     | node.py              ( 884) | onConnsChanged | Node2 lost connection to primary of master
2017-11-17 13:57:27,920 | INFO     | node.py              (2128) | sendInstanceChange | VIEW CHANGE: Node2 sending an instance change with view_no 3 since Primary of master protocol instance disconnected
2017-11-17 13:57:27,920 | INFO     | node.py              (2131) | sendInstanceChange | MONITORING: Node2 metrics for monitor: Node2 Monitor metrics:: None
            Delta: 0.4
            Lambda: 60
            Omega: 5
            instances started: [1381600.91592481, 1381600.916206756]
            ordered request counts: {0: 1, 1: 1}
            ordered request durations: {0: 0.04119092784821987, 1: 0.04124294384382665}
            master request latencies: {}
            client avg request latencies: [{'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': (1, 0.04119092784821987)}, {'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': (1, 0.04124294384382665)}]
            throughput: {0: 24.27719044554653, 1: 24.246571820543856}
            master throughput: 24.27719044554653
            total requests: 1
            avg backup throughput: 24.246571820543856
            master throughput ratio: 1.0012628022315604
2017-11-17 13:57:27,920 | INFO     | node.py              (2191) | propose_view_change | Node2 sent view change since was disconnected from primary for too long
2017-11-17 13:57:27,945 | INFO     | node.py              (2031) | do_view_change_if_possible | VIEW CHANGE: Node2 initiating a view change to 3 from 2
2017-11-17 13:57:27,945 | INFO     | node.py              (2241) | startViewChange | VIEW CHANGE: Node2 changed to view 3, will start catchup now
2017-11-17 13:57:27,975 | INFO     | node.py              (1551) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 0
2017-11-17 13:57:27,975 | INFO     | ledger_manager.py    ( 865) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 0, caught up 0 in total
2017-11-17 13:57:27,994 | INFO     | node.py              (1551) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 2
2017-11-17 13:57:27,994 | INFO     | upgrader.py          ( 212) | processLedger | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb found upgrade START txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)])
2017-11-17 13:57:27,995 | INFO     | upgrader.py          ( 290) | handleUpgradeTxn | Node 'Node2' handles upgrade txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)])
2017-11-17 13:57:27,995 | INFO     | upgrader.py          ( 310) | handleUpgradeTxn | Node 'Node2' has already performed an upgrade with upgrade_id 15109263663135171. Last recorded event is ('succeeded', datetime.datetime(2017, 11, 17, 13, 52, 0, 258870, tzinfo=tzlocal()), '1.2.45', '15109263663135171')
2017-11-17 13:57:27,995 | INFO     | upgrader.py          ( 149) | should_notify_about_upgrade_result | Node's 'Node2' last upgrade txn is OrderedDict([('data', OrderedDict([('action', 'in_progress'), ('version', '1.2.45')])), ('identifier', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb'), ('reqId', 1510926719502400), ('signature', '59fs2tWoGeAcMnUon3maEHpzyNZhNULgk4zctNgfGRSB3AE7D3ciyXpicauLCRwpKzzknWaQhK2V1Btj73K6B2Kp'), ('txnTime', 1510926719), ('type', '110'), ('seqNo', 3)])
2017-11-17 13:57:27,996 | INFO     | ledger_manager.py    ( 865) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 2, caught up 0 in total
2017-11-17 13:57:28,013 | INFO     | propagator.py        ( 179) | propagate | Node2 propagating request ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927048003928) from client Node1
2017-11-17 13:57:28,013 | INFO     | node.py              (1551) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 1
2017-11-17 13:57:28,014 | INFO     | ledger_manager.py    ( 865) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 1, caught up 0 in total
2017-11-17 13:57:28,014 | INFO     | node.py              (1598) | allLedgersCaughtUp | CATCH-UP: Node2 does not need any more catchups
2017-11-17 13:57:28,015 | INFO     | propagator.py        ( 179) | propagate | Node2 propagating request ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927047995990) from client Node2
2017-11-17 13:57:28,033 | DISPLAY  | primary_selector.py  ( 327) | _start_selection | PRIMARY SELECTION: Node2:0 selected primary Node4:0 for instance 0 (view 3)
2017-11-17 13:57:28,033 | INFO     | node.py              ( 532) | start_participating | Node2 started participating
2017-11-17 13:57:28,034 | DISPLAY  | primary_selector.py  ( 351) | _start_selection | VIEW CHANGE: Node2:0 declares view change 3 as completed for instance 0, new primary is Node4:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 4, '3aErFDAegiipoxjKyq8YBzyNpDPHaoNhZ8BqVu4e3uCg')]
2017-11-17 13:57:28,034 | DISPLAY  | primary_selector.py  ( 327) | _start_selection | PRIMARY SELECTION: Node2:1 selected primary Node1:1 for instance 1 (view 3)
2017-11-17 13:57:28,034 | DISPLAY  | primary_selector.py  ( 351) | _start_selection | VIEW CHANGE: Node2:1 declares view change 3 as completed for instance 1, new primary is Node1:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 4, '3aErFDAegiipoxjKyq8YBzyNpDPHaoNhZ8BqVu4e3uCg')]
2017-11-17 13:57:28,152 | INFO     | keep_in_touch.py     (  93) | _connsChanged | CONNECTION: Node2 disconnected from Node4
2017-11-17 13:57:28,152 | INFO     | node.py              ( 884) | onConnsChanged | Node2 lost connection to primary of master
2017-11-17 13:57:30,156 | INFO     | node.py              (2128) | sendInstanceChange | VIEW CHANGE: Node2 sending an instance change with view_no 4 since Primary of master protocol instance disconnected
2017-11-17 13:57:30,156 | INFO     | node.py              (2131) | sendInstanceChange | MONITORING: Node2 metrics for monitor: Node2 Monitor metrics:: None
            Delta: 0.4
            Lambda: 60
            Omega: 5
            instances started: [1381600.91592481, 1381600.916206756]
            ordered request counts: {0: 0, 1: 0}
            ordered request durations: {0: 0, 1: 0}
            master request latencies: {}
            client avg request latencies: [{}, {}]
            throughput: {0: 0, 1: 0}
            master throughput: None
            total requests: 1
            avg backup throughput: None
            master throughput ratio: None
2017-11-17 13:57:30,156 | INFO     | node.py              (2191) | propose_view_change | Node2 sent view change since was disconnected from primary for too long
2017-11-17 13:57:32,933 | INFO     | keep_in_touch.py     (  98) | _connsChanged | CONNECTION: Node2 now connected to Node3
2017-11-17 13:57:37,501 | INFO     | propagator.py        ( 179) | propagate | Node2 propagating request ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927057486576) from client Node3
2017-11-17 13:57:37,539 | INFO     | replica.py           (1619) | order_3pc_key | Node2:1 ordered batch request, view no 3, ppSeqNo 1, ledger 2, state root None, txn root None, requests ordered [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926748095846), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926752873422), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927048003928), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927047995990)], discarded []
2017-11-17 13:57:37,553 | INFO     | replica.py           (1619) | order_3pc_key | Node2:1 ordered batch request, view no 3, ppSeqNo 2, ledger 2, state root None, txn root None, requests ordered [('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927057486576)], discarded []
2017-11-17 13:57:42,608 | INFO     | node.py              (2031) | do_view_change_if_possible | VIEW CHANGE: Node2 initiating a view change to 4 from 3
2017-11-17 13:57:42,609 | INFO     | node.py              (2241) | startViewChange | VIEW CHANGE: Node2 changed to view 4, will start catchup now
2017-11-17 13:57:42,643 | INFO     | node.py              (1551) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 0
2017-11-17 13:57:42,644 | INFO     | ledger_manager.py    ( 865) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 0, caught up 0 in total
2017-11-17 13:57:42,660 | INFO     | propagator.py        ( 179) | propagate | Node2 propagating request ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927062651136) from client Node3
2017-11-17 13:57:42,661 | INFO     | node.py              (1551) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 2
2017-11-17 13:57:42,661 | INFO     | upgrader.py          ( 212) | processLedger | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb found upgrade START txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)])
2017-11-17 13:57:42,661 | INFO     | upgrader.py          ( 290) | handleUpgradeTxn | Node 'Node2' handles upgrade txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)])
2017-11-17 13:57:42,661 | INFO     | upgrader.py          ( 310) | handleUpgradeTxn | Node 'Node2' has already performed an upgrade with upgrade_id 15109263663135171. Last recorded event is ('succeeded', datetime.datetime(2017, 11, 17, 13, 52, 0, 258870, tzinfo=tzlocal()), '1.2.45', '15109263663135171')
2017-11-17 13:57:42,662 | INFO     | upgrader.py          ( 149) | should_notify_about_upgrade_result | Node's 'Node2' last upgrade txn is OrderedDict([('data', OrderedDict([('action', 'in_progress'), ('version', '1.2.45')])), ('identifier', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb'), ('reqId', 1510926719502400), ('signature', '59fs2tWoGeAcMnUon3maEHpzyNZhNULgk4zctNgfGRSB3AE7D3ciyXpicauLCRwpKzzknWaQhK2V1Btj73K6B2Kp'), ('txnTime', 1510926719), ('type', '110'), ('seqNo', 3)])
2017-11-17 13:57:42,662 | INFO     | ledger_manager.py    ( 865) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 2, caught up 0 in total
2017-11-17 13:57:42,664 | INFO     | propagator.py        ( 179) | propagate | Node2 propagating request ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927062654152) from client Node1
2017-11-17 13:57:42,685 | INFO     | propagator.py        ( 179) | propagate | Node2 propagating request ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927062662523) from client Node2
2017-11-17 13:57:42,685 | INFO     | node.py              (1551) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 1
2017-11-17 13:57:42,685 | INFO     | ledger_manager.py    ( 865) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 1, caught up 0 in total
2017-11-17 13:57:42,686 | INFO     | node.py              (1598) | allLedgersCaughtUp | CATCH-UP: Node2 does not need any more catchups
2017-11-17 13:57:42,687 | DISPLAY  | primary_selector.py  ( 327) | _start_selection | PRIMARY SELECTION: Node2:0 selected primary Node1:0 for instance 0 (view 4)
2017-11-17 13:57:42,687 | INFO     | node.py              ( 532) | start_participating | Node2 started participating
2017-11-17 13:57:42,688 | DISPLAY  | primary_selector.py  ( 351) | _start_selection | VIEW CHANGE: Node2:0 declares view change 4 as completed for instance 0, new primary is Node1:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 4, '3aErFDAegiipoxjKyq8YBzyNpDPHaoNhZ8BqVu4e3uCg')]
2017-11-17 13:57:42,688 | DISPLAY  | primary_selector.py  ( 327) | _start_selection | PRIMARY SELECTION: Node2:1 selected primary Node2:1 for instance 1 (view 4)
2017-11-17 13:57:42,688 | DISPLAY  | primary_selector.py  ( 351) | _start_selection | VIEW CHANGE: Node2:1 declares view change 4 as completed for instance 1, new primary is Node2:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 4, '3aErFDAegiipoxjKyq8YBzyNpDPHaoNhZ8BqVu4e3uCg')]
2017-11-17 13:57:42,725 | INFO     | replica.py           (1619) | order_3pc_key | Node2:0 ordered batch request, view no 4, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 8PHonJAkhGQDvCHhZ336fu17U8UV3wYpJGCjV7gdLdBd, requests ordered [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926748095846), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926752873422), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927048003928), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927047995990), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927057486576), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927062651136), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927062654152), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927062662523)], discarded []
2017-11-17 13:57:42,726 | INFO     | replica.py           (1619) | order_3pc_key | Node2:1 ordered batch request, view no 4, ppSeqNo 1, ledger 2, state root None, txn root None, requests ordered [('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927062651136), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927062654152), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927062662523)], discarded []
2017-11-17 13:57:42,729 | INFO     | node.py              (2374) | executeBatch | Node2 committed batch request, view no 4, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 8PHonJAkhGQDvCHhZ336fu17U8UV3wYpJGCjV7gdLdBd, requests: [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926748095846), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926752873422), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927048003928), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927047995990), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927057486576), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927062651136), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927062654152), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927062662523)]
2017-11-17 13:57:44,670 | INFO     | keep_in_touch.py     (  98) | _connsChanged | CONNECTION: Node2 now connected to Node4