2017-11-21 13:09:42,108 | INFO | node.py (2420) | initStateFromLedger | Node1 found state to be empty, recreating from ledger 2017-11-21 13:09:42,188 | INFO | node.py (2420) | initStateFromLedger | Node1 found state to be empty, recreating from ledger 2017-11-21 13:09:42,243 | INFO | plugin_loader.py ( 117) | _load | plugin FirebaseStatsConsumer successfully loaded from module plugin_firebase_stats_consumer 2017-11-21 13:09:42,244 | DISPLAY | replicas.py ( 36) | grow | Node1 added replica Node1:0 to instance 0 (master) 2017-11-21 13:09:42,244 | DISPLAY | replicas.py ( 36) | grow | Node1 added replica Node1:1 to instance 1 (backup) 2017-11-21 13:09:42,323 | INFO | node.py (2420) | initStateFromLedger | Node1 found state to be empty, recreating from ledger 2017-11-21 13:09:42,324 | INFO | stacks.py ( 84) | start | CONNECTION: Node1 listening for other nodes at 0.0.0.0:9701 2017-11-21 13:09:42,325 | INFO | node.py ( 621) | start | Node1 first time running... 2017-11-21 13:09:42,327 | INFO | zstack.py ( 590) | connect | CONNECTION: Node1 looking for Node2 at 10.0.0.3:9703 2017-11-21 13:09:42,329 | INFO | zstack.py ( 590) | connect | CONNECTION: Node1 looking for Node3 at 10.0.0.4:9705 2017-11-21 13:09:42,331 | INFO | zstack.py ( 590) | connect | CONNECTION: Node1 looking for Node4 at 10.0.0.5:9707 2017-11-21 13:09:42,386 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node1 now connected to Node4 2017-11-21 13:09:42,548 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node1 now connected to Node3 2017-11-21 13:09:42,562 | INFO | node.py (1489) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 0 2017-11-21 13:09:42,563 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 0, caught up 0 in total 2017-11-21 13:09:42,564 | INFO | node.py (1489) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 2 2017-11-21 13:09:42,565 | INFO | node.py (1489) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 1 2017-11-21 13:09:42,565 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 1, caught up 0 in total 2017-11-21 13:09:42,565 | INFO | node.py (1537) | allLedgersCaughtUp | CATCH-UP: Node1 does not need any more catchups 2017-11-21 13:09:42,565 | DISPLAY | primary_selector.py ( 291) | _startSelection | PRIMARY SELECTION: Node1:0 selected primary Node1:0 for instance 0 (view 0) 2017-11-21 13:09:42,566 | INFO | node.py ( 510) | start_participating | Node1 started participating 2017-11-21 13:09:42,566 | DISPLAY | primary_selector.py ( 315) | _startSelection | VIEW CHANGE: Node1:0 declares view change 0 as completed for instance 0, new primary is Node1:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-11-21 13:09:42,566 | DISPLAY | primary_selector.py ( 291) | _startSelection | PRIMARY SELECTION: Node1:1 selected primary Node2:1 for instance 1 (view 0) 2017-11-21 13:09:42,566 | DISPLAY | primary_selector.py ( 315) | _startSelection | VIEW CHANGE: Node1:1 declares view change 0 as completed for instance 1, new primary is Node2:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-11-21 13:09:42,566 | INFO | upgrader.py ( 151) | should_notify_about_upgrade_result | Node's 'Node1' last upgrade txn is None 2017-11-21 13:09:42,566 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 2, caught up 0 in total 2017-11-21 13:09:42,567 | INFO | node.py (1537) | allLedgersCaughtUp | CATCH-UP: Node1 does not need any more catchups 2017-11-21 13:09:42,567 | INFO | node.py (1624) | no_more_catchups_needed | Node1 starting to participate since catchup is done, primaries are selected but mode was not set to participating 2017-11-21 13:09:42,567 | INFO | node.py ( 510) | start_participating | Node1 started participating 2017-11-21 13:09:42,568 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node1 now connected to Node2 2017-11-21 13:10:42,327 | INFO | node.py (2048) | sendInstanceChange | VIEW CHANGE: Node1 sending an instance change with view_no 1 since Primary of master protocol instance disconnected 2017-11-21 13:10:42,327 | INFO | node.py (2051) | sendInstanceChange | MONITORING: Node1 metrics for monitor: Node1 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [1724630.453797027, 1724630.454056903] ordered request counts: {0: 0, 1: 0} ordered request durations: {0: 0, 1: 0} master request latencies: {} client avg request latencies: [{}, {}] throughput: {0: 0, 1: 0} master throughput: None total requests: 0 avg backup throughput: None master throughput ratio: None 2017-11-21 13:10:42,328 | INFO | node.py (2108) | propose_view_change | Node1 sent view change since was disconnected from primary for too long 2017-11-21 13:30:22,990 | INFO | propagator.py ( 148) | propagate | Node1 propagating request ('V4SGRU86Z58d6TV7PBUe6f', 1511271022953271) from client MyAS5>/6=HGPt>CF{HKRU.&xuO$0^KZ!y>e}9RLl 2017-11-21 13:30:23,027 | INFO | replica.py (1483) | order_3pc_key | Node1:0 ordered batch request, view no 0, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root ZNYrpy1fYTShLY4hAkJtafsNJvDd6gnzpC4UKBzVd3G, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1511271022953271)], discarded [] 2017-11-21 13:30:23,028 | INFO | upgrader.py ( 292) | handleUpgradeTxn | Node 'Node1' handles upgrade txn {'auditPath': [], 'name': 'upgrade_to_the_latest_full', 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'force': False, 'reinstall': False, 'action': 'start', 'txnTime': 1511271022, 'version': '1.2.47', 'justification': None, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-11-21T13:35:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-11-21T13:45:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-11-21T13:40:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-11-21T13:50:00.258870+00:00'}, 'seqNo': 1, 'type': '109', 'rootHash': 'ZNYrpy1fYTShLY4hAkJtafsNJvDd6gnzpC4UKBzVd3G', 'signature': '3NRsExGbSDFBeVe8Negs1fJuWJTugu2xq1V6eSVHhGNeSULAeBdzn9VNyv4wwYSnKjhc94n4k2SSm6k7AVMoSHzS', 'timeout': 10, 'reqId': 1511271022953271} 2017-11-21 13:30:23,028 | INFO | upgrader.py ( 321) | handleUpgradeTxn | Node 'Node1' schedules upgrade to 1.2.47 2017-11-21 13:30:23,028 | INFO | upgrader.py ( 359) | _scheduleUpgrade | Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv's upgrader processing upgrade for version 1.2.47 2017-11-21 13:30:23,029 | INFO | node.py (2290) | executeBatch | Node1 committed batch request, view no 0, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root ZNYrpy1fYTShLY4hAkJtafsNJvDd6gnzpC4UKBzVd3G, requests: [('V4SGRU86Z58d6TV7PBUe6f', 1511271022953271)] 2017-11-21 13:30:23,031 | INFO | replica.py (1483) | order_3pc_key | Node1:1 ordered batch request, view no 0, ppSeqNo 1, ledger 2, state root None, txn root None, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1511271022953271)], discarded [] 2017-11-21 13:35:00,030 | INFO | upgrader.py ( 421) | _callUpgradeAgent | Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv's upgrader calling agent for upgrade 2017-11-21 13:35:00,031 | INFO | node.py ( 254) | notify_upgrade_start | Node1 is about to be upgraded, sending NODE_UPGRADE 2017-11-21 13:35:00,033 | INFO | upgrader.py ( 433) | _sendUpdateRequest | Sending message to control tool: {"version": "1.2.47"} 2017-11-21 13:35:00,034 | INFO | upgrader.py ( 452) | _sendUpdateRequest | Waiting 10 minutes for upgrade to be performed 2017-11-21 13:35:00,046 | INFO | propagator.py ( 148) | propagate | Node1 propagating request ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1511271300031735) from client Node1 2017-11-21 13:35:00,067 | INFO | replica.py (1483) | order_3pc_key | Node1:0 ordered batch request, view no 0, ppSeqNo 2, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root GeSC6DwsVQaSWEVVoWjmCTGE1yAdUaq3c7nE1t1FyHfG, requests ordered [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1511271300031735)], discarded [] 2017-11-21 13:35:00,068 | INFO | node.py (2290) | executeBatch | Node1 committed batch request, view no 0, ppSeqNo 2, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root GeSC6DwsVQaSWEVVoWjmCTGE1yAdUaq3c7nE1t1FyHfG, requests: [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1511271300031735)] 2017-11-21 13:35:00,085 | INFO | replica.py (1483) | order_3pc_key | Node1:1 ordered batch request, view no 0, ppSeqNo 2, ledger 2, state root None, txn root None, requests ordered [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1511271300031735)], discarded [] 2017-11-21 13:35:29,725 | INFO | node.py (2503) | initStateFromLedger | Node1 found state to be empty, recreating from ledger 2017-11-21 13:35:29,727 | INFO | pool_manager.py ( 409) | _order_node | Node1 node Node1 ordered, NYM Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv 2017-11-21 13:35:29,728 | INFO | pool_manager.py ( 409) | _order_node | Node1 node Node2 ordered, NYM 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb 2017-11-21 13:35:29,728 | INFO | pool_manager.py ( 409) | _order_node | Node1 node Node3 ordered, NYM DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya 2017-11-21 13:35:29,728 | INFO | pool_manager.py ( 409) | _order_node | Node1 node Node4 ordered, NYM 4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA 2017-11-21 13:35:29,747 | WARNING | node.py ( 623) | _create_bls_bft | BLS: Transactions will not be BLS signed by this Node, since BLS keys were not found. Please make sure that a script to init BLS keys was called (init_bls_keys), and NODE txn was sent with BLS public keys. 2017-11-21 13:35:29,751 | INFO | node.py (2503) | initStateFromLedger | Node1 found state to be empty, recreating from ledger 2017-11-21 13:35:29,762 | INFO | node.py ( 479) | setPoolParams | Node1 updated its pool parameters: f 1, totalNodes 4, allNodeNames {'Node2', 'Node1', 'Node4', 'Node3'}, requiredNumberOfInstances 2, minimumNodes 3, quorums {'same_consistency_proof': Quorum(2), 'prepare': Quorum(2), 'bls_signatures': Quorum(3), 'propagate': Quorum(2), 'timestamp': Quorum(2), 'reply': Quorum(2), 'ledger_status': Quorum(2), 'election': Quorum(3), 'consistency_proof': Quorum(2), 'checkpoint': Quorum(2), 'f': 1, 'commit': Quorum(3), 'view_change_done': Quorum(3), 'view_change': Quorum(3), 'propagate_primary': Quorum(2)} 2017-11-21 13:35:29,793 | INFO | plugin_loader.py ( 117) | _load | plugin FirebaseStatsConsumer successfully loaded from module plugin_firebase_stats_consumer 2017-11-21 13:35:29,794 | DISPLAY | replicas.py ( 40) | grow | Node1 added replica Node1:0 to instance 0 (master) 2017-11-21 13:35:29,794 | DISPLAY | replicas.py ( 40) | grow | Node1 added replica Node1:1 to instance 1 (backup) 2017-11-21 13:35:29,830 | INFO | upgrader.py ( 152) | _update_upgrade_log_for_started_upgrade | Node 'Node1' successfully upgraded to version 1.2.47 2017-11-21 13:35:29,831 | INFO | node.py (2503) | initStateFromLedger | Node1 found state to be empty, recreating from ledger 2017-11-21 13:35:29,832 | INFO | stacks.py ( 84) | start | CONNECTION: Node1 listening for other nodes at 0.0.0.0:9701 2017-11-21 13:35:29,832 | INFO | node.py ( 680) | start | Node1 first time running... 2017-11-21 13:35:29,834 | INFO | zstack.py ( 585) | connect | CONNECTION: Node1 looking for Node2 at 10.0.0.3:9703 2017-11-21 13:35:29,835 | INFO | zstack.py ( 585) | connect | CONNECTION: Node1 looking for Node3 at 10.0.0.4:9705 2017-11-21 13:35:29,835 | INFO | zstack.py ( 585) | connect | CONNECTION: Node1 looking for Node4 at 10.0.0.5:9707 2017-11-21 13:35:29,967 | INFO | node.py (2044) | _start_view_change_if_possible | VIEW CHANGE: Node1 starting view change for 1 after 2 view change indications from other nodes 2017-11-21 13:35:29,967 | INFO | node.py (2241) | startViewChange | VIEW CHANGE: Node1 changed to view 1, will start catchup now 2017-11-21 13:35:29,968 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node1 now connected to Node2 2017-11-21 13:35:29,968 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node1 now connected to Node4 2017-11-21 13:35:29,984 | INFO | message_processor.py ( 29) | discard | Node1 discarding message INSTANCE_CHANGE{'viewNo': 1, 'reason': 26} because Received instance change request with view no 1 which is not more than its view no 1 2017-11-21 13:35:29,987 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node1 now connected to Node3 2017-11-21 13:35:29,989 | INFO | node.py (1551) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 0 2017-11-21 13:35:29,989 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 0, caught up 0 in total 2017-11-21 13:35:29,990 | INFO | node.py (1551) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 2 2017-11-21 13:35:29,990 | INFO | upgrader.py ( 224) | processLedger | Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv found upgrade START txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1511271022953271), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-21T13:50:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-21T13:40:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-21T13:45:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-21T13:35:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', '3NRsExGbSDFBeVe8Negs1fJuWJTugu2xq1V6eSVHhGNeSULAeBdzn9VNyv4wwYSnKjhc94n4k2SSm6k7AVMoSHzS'), ('timeout', 10), ('txnTime', 1511271022), ('type', '109'), ('version', '1.2.47'), ('seqNo', 1)]) 2017-11-21 13:35:29,990 | INFO | upgrader.py ( 305) | handleUpgradeTxn | Node 'Node1' handles upgrade txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1511271022953271), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-21T13:50:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-21T13:40:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-21T13:45:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-21T13:35:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', '3NRsExGbSDFBeVe8Negs1fJuWJTugu2xq1V6eSVHhGNeSULAeBdzn9VNyv4wwYSnKjhc94n4k2SSm6k7AVMoSHzS'), ('timeout', 10), ('txnTime', 1511271022), ('type', '109'), ('version', '1.2.47'), ('seqNo', 1)]) 2017-11-21 13:35:29,991 | INFO | upgrader.py ( 325) | handleUpgradeTxn | Node 'Node1' has already performed an upgrade with upgrade_id 15112710229532711. Last recorded event is ('succeeded', datetime.datetime(2017, 11, 21, 13, 35, 0, 258870, tzinfo=tzlocal()), '1.2.47', '15112710229532711') 2017-11-21 13:35:29,994 | INFO | node.py (1551) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 1 2017-11-21 13:35:29,994 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 1, caught up 0 in total 2017-11-21 13:35:29,994 | INFO | node.py (1588) | allLedgersCaughtUp | CATCH-UP: Node1 caught up till (0, 2) 2017-11-21 13:35:29,995 | INFO | node.py (1598) | allLedgersCaughtUp | CATCH-UP: Node1 does not need any more catchups 2017-11-21 13:35:29,996 | DISPLAY | primary_selector.py ( 327) | _start_selection | PRIMARY SELECTION: Node1:0 selected primary Node2:0 for instance 0 (view 1) 2017-11-21 13:35:29,996 | INFO | node.py ( 532) | start_participating | Node1 started participating 2017-11-21 13:35:29,996 | DISPLAY | primary_selector.py ( 351) | _start_selection | VIEW CHANGE: Node1:0 declares view change 1 as completed for instance 0, new primary is Node2:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 2, 'GeSC6DwsVQaSWEVVoWjmCTGE1yAdUaq3c7nE1t1FyHfG')] 2017-11-21 13:35:29,996 | DISPLAY | primary_selector.py ( 327) | _start_selection | PRIMARY SELECTION: Node1:1 selected primary Node3:1 for instance 1 (view 1) 2017-11-21 13:35:29,996 | DISPLAY | primary_selector.py ( 351) | _start_selection | VIEW CHANGE: Node1:1 declares view change 1 as completed for instance 1, new primary is Node3:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 2, 'GeSC6DwsVQaSWEVVoWjmCTGE1yAdUaq3c7nE1t1FyHfG')] 2017-11-21 13:35:29,997 | INFO | node.py ( 242) | acknowledge_upgrade | Node1 found the first run after upgrade, sending NODE_UPGRADE complete to version 1.2.47 2017-11-21 13:35:29,998 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 2, caught up 0 in total 2017-11-21 13:35:29,998 | INFO | node.py (1598) | allLedgersCaughtUp | CATCH-UP: Node1 does not need any more catchups 2017-11-21 13:35:29,999 | INFO | node.py (1685) | no_more_catchups_needed | Node1 starting to participate since catchup is done, primaries are selected but mode was not set to participating 2017-11-21 13:35:29,999 | INFO | node.py ( 532) | start_participating | Node1 started participating 2017-11-21 13:35:30,027 | INFO | propagator.py ( 179) | propagate | Node1 propagating request ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1511271329997860) from client Node1 2017-11-21 13:35:30,066 | INFO | replica.py (1619) | order_3pc_key | Node1:1 ordered batch request, view no 1, ppSeqNo 1, ledger 2, state root None, txn root None, requests ordered [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1511271329997860)], discarded [] 2017-11-21 13:35:30,070 | INFO | replica.py (1619) | order_3pc_key | Node1:0 ordered batch request, view no 1, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root Sufjfa1upHQkafvtKeThr49DC7inH3CCejhogJdUrhD, requests ordered [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1511271329997860)], discarded [] 2017-11-21 13:35:30,072 | INFO | node.py (2374) | executeBatch | Node1 committed batch request, view no 1, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root Sufjfa1upHQkafvtKeThr49DC7inH3CCejhogJdUrhD, requests: [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1511271329997860)] 2017-11-21 13:39:59,308 | INFO | propagator.py ( 179) | propagate | Node1 propagating request ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1511271599296461) from client Node2 2017-11-21 13:39:59,332 | INFO | replica.py (1619) | order_3pc_key | Node1:1 ordered batch request, view no 1, ppSeqNo 2, ledger 2, state root None, txn root None, requests ordered [('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1511271599296461)], discarded [] 2017-11-21 13:39:59,351 | INFO | replica.py (1619) | order_3pc_key | Node1:0 ordered batch request, view no 1, ppSeqNo 2, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root GJoBk7sZCyo7GfzTzL7PE7CEdMJUCGuDkFF7KqtG1vDB, requests ordered [('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1511271599296461)], discarded [] 2017-11-21 13:39:59,352 | INFO | node.py (2374) | executeBatch | Node1 committed batch request, view no 1, ppSeqNo 2, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root GJoBk7sZCyo7GfzTzL7PE7CEdMJUCGuDkFF7KqtG1vDB, requests: [('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1511271599296461)] 2017-11-21 13:40:22,230 | INFO | keep_in_touch.py ( 93) | _connsChanged | CONNECTION: Node1 disconnected from Node2 2017-11-21 13:40:22,230 | INFO | node.py ( 884) | onConnsChanged | Node1 lost connection to primary of master 2017-11-21 13:40:24,237 | INFO | node.py (2128) | sendInstanceChange | VIEW CHANGE: Node1 sending an instance change with view_no 2 since Primary of master protocol instance disconnected 2017-11-21 13:40:24,237 | INFO | node.py (2131) | sendInstanceChange | MONITORING: Node1 metrics for monitor: Node1 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [1726178.003684762, 1726178.003981781] ordered request counts: {0: 2, 1: 2} ordered request durations: {0: 0.08378025097772479, 1: 0.0592197198420763} master request latencies: {} client avg request latencies: [{'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': (1, 0.044939704006537795), '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': (1, 0.038840546971186996)}, {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': (1, 0.0401502619497478), '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': (1, 0.0190694578923285)}] throughput: {0: 23.871974321630443, 1: 33.772533968979985} master throughput: 23.871974321630443 total requests: 2 avg backup throughput: 33.772533968979985 master throughput ratio: 0.7068458157020971 2017-11-21 13:40:24,237 | INFO | node.py (2191) | propose_view_change | Node1 sent view change since was disconnected from primary for too long 2017-11-21 13:40:24,250 | INFO | node.py (2031) | do_view_change_if_possible | VIEW CHANGE: Node1 initiating a view change to 2 from 1 2017-11-21 13:40:24,250 | INFO | node.py (2241) | startViewChange | VIEW CHANGE: Node1 changed to view 2, will start catchup now 2017-11-21 13:40:24,267 | INFO | node.py (1551) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 0 2017-11-21 13:40:24,267 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 0, caught up 0 in total 2017-11-21 13:40:24,296 | INFO | node.py (1551) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 2 2017-11-21 13:40:24,297 | INFO | upgrader.py ( 224) | processLedger | Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv found upgrade START txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1511271022953271), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-21T13:50:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-21T13:40:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-21T13:45:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-21T13:35:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', '3NRsExGbSDFBeVe8Negs1fJuWJTugu2xq1V6eSVHhGNeSULAeBdzn9VNyv4wwYSnKjhc94n4k2SSm6k7AVMoSHzS'), ('timeout', 10), ('txnTime', 1511271022), ('type', '109'), ('version', '1.2.47'), ('seqNo', 1)]) 2017-11-21 13:40:24,297 | INFO | upgrader.py ( 305) | handleUpgradeTxn | Node 'Node1' handles upgrade txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1511271022953271), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-21T13:50:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-21T13:40:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-21T13:45:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-21T13:35:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', '3NRsExGbSDFBeVe8Negs1fJuWJTugu2xq1V6eSVHhGNeSULAeBdzn9VNyv4wwYSnKjhc94n4k2SSm6k7AVMoSHzS'), ('timeout', 10), ('txnTime', 1511271022), ('type', '109'), ('version', '1.2.47'), ('seqNo', 1)]) 2017-11-21 13:40:24,297 | INFO | upgrader.py ( 325) | handleUpgradeTxn | Node 'Node1' has already performed an upgrade with upgrade_id 15112710229532711. Last recorded event is ('succeeded', datetime.datetime(2017, 11, 21, 13, 35, 0, 258870, tzinfo=tzlocal()), '1.2.47', '15112710229532711') 2017-11-21 13:40:24,297 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 2, caught up 0 in total 2017-11-21 13:40:24,326 | INFO | node.py (1551) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 1 2017-11-21 13:40:24,326 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 1, caught up 0 in total 2017-11-21 13:40:24,326 | INFO | node.py (1598) | allLedgersCaughtUp | CATCH-UP: Node1 does not need any more catchups 2017-11-21 13:40:24,327 | DISPLAY | primary_selector.py ( 327) | _start_selection | PRIMARY SELECTION: Node1:0 selected primary Node3:0 for instance 0 (view 2) 2017-11-21 13:40:24,328 | INFO | node.py ( 532) | start_participating | Node1 started participating 2017-11-21 13:40:24,328 | DISPLAY | primary_selector.py ( 351) | _start_selection | VIEW CHANGE: Node1:0 declares view change 2 as completed for instance 0, new primary is Node3:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 4, 'GJoBk7sZCyo7GfzTzL7PE7CEdMJUCGuDkFF7KqtG1vDB')] 2017-11-21 13:40:24,328 | DISPLAY | primary_selector.py ( 327) | _start_selection | PRIMARY SELECTION: Node1:1 selected primary Node4:1 for instance 1 (view 2) 2017-11-21 13:40:24,328 | DISPLAY | primary_selector.py ( 351) | _start_selection | VIEW CHANGE: Node1:1 declares view change 2 as completed for instance 1, new primary is Node4:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 4, 'GJoBk7sZCyo7GfzTzL7PE7CEdMJUCGuDkFF7KqtG1vDB')] 2017-11-21 13:40:28,854 | INFO | propagator.py ( 179) | propagate | Node1 propagating request ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1511271628844985) from client Node2 2017-11-21 13:40:28,902 | INFO | replica.py (1619) | order_3pc_key | Node1:0 ordered batch request, view no 2, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 44Kh4HRALU5B6nY6yWJx8rhrcMf3BvHdu1sPScXEjiEo, requests ordered [('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1511271628844985)], discarded [] 2017-11-21 13:40:28,902 | INFO | replica.py (1619) | order_3pc_key | Node1:1 ordered batch request, view no 2, ppSeqNo 1, ledger 2, state root None, txn root None, requests ordered [('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1511271628844985)], discarded [] 2017-11-21 13:40:28,903 | INFO | node.py (2374) | executeBatch | Node1 committed batch request, view no 2, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 44Kh4HRALU5B6nY6yWJx8rhrcMf3BvHdu1sPScXEjiEo, requests: [('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1511271628844985)] 2017-11-21 13:40:30,009 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node1 now connected to Node2 2017-11-21 13:44:59,286 | INFO | propagator.py ( 179) | propagate | Node1 propagating request ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1511271899274670) from client Node3 2017-11-21 13:44:59,307 | INFO | replica.py (1619) | order_3pc_key | Node1:1 ordered batch request, view no 2, ppSeqNo 2, ledger 2, state root None, txn root None, requests ordered [('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1511271899274670)], discarded [] 2017-11-21 13:44:59,323 | INFO | replica.py (1619) | order_3pc_key | Node1:0 ordered batch request, view no 2, ppSeqNo 2, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 3ru9mawZCCRe7brebbogXMNBwceurrrDPC1FGvNkrsks, requests ordered [('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1511271899274670)], discarded [] 2017-11-21 13:44:59,324 | INFO | node.py (2374) | executeBatch | Node1 committed batch request, view no 2, ppSeqNo 2, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 3ru9mawZCCRe7brebbogXMNBwceurrrDPC1FGvNkrsks, requests: [('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1511271899274670)] 2017-11-21 13:45:42,714 | INFO | keep_in_touch.py ( 93) | _connsChanged | CONNECTION: Node1 disconnected from Node3 2017-11-21 13:45:42,714 | INFO | node.py ( 884) | onConnsChanged | Node1 lost connection to primary of master 2017-11-21 13:45:44,723 | INFO | node.py (2128) | sendInstanceChange | VIEW CHANGE: Node1 sending an instance change with view_no 3 since Primary of master protocol instance disconnected 2017-11-21 13:45:44,723 | INFO | node.py (2131) | sendInstanceChange | MONITORING: Node1 metrics for monitor: Node1 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [1726178.003684762, 1726178.003981781] ordered request counts: {0: 2, 1: 2} ordered request durations: {0: 0.06967695313505828, 1: 0.05291926092468202} master request latencies: {} client avg request latencies: [{'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': (1, 0.03732627001591027), '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': (1, 0.032350683119148016)}, {'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': (1, 0.020526513922959566), '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': (1, 0.032392747001722455)}] throughput: {0: 28.70389576483491, 1: 37.79342275483636} master throughput: 28.70389576483491 total requests: 4 avg backup throughput: 37.79342275483636 master throughput ratio: 0.759494474767087 2017-11-21 13:45:44,724 | INFO | node.py (2191) | propose_view_change | Node1 sent view change since was disconnected from primary for too long 2017-11-21 13:45:44,725 | INFO | node.py (2031) | do_view_change_if_possible | VIEW CHANGE: Node1 initiating a view change to 3 from 2 2017-11-21 13:45:44,725 | INFO | node.py (2241) | startViewChange | VIEW CHANGE: Node1 changed to view 3, will start catchup now 2017-11-21 13:45:44,743 | INFO | node.py (1551) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 0 2017-11-21 13:45:44,744 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 0, caught up 0 in total 2017-11-21 13:45:44,760 | INFO | node.py (1551) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 2 2017-11-21 13:45:44,760 | INFO | upgrader.py ( 224) | processLedger | Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv found upgrade START txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1511271022953271), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-21T13:50:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-21T13:40:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-21T13:45:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-21T13:35:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', '3NRsExGbSDFBeVe8Negs1fJuWJTugu2xq1V6eSVHhGNeSULAeBdzn9VNyv4wwYSnKjhc94n4k2SSm6k7AVMoSHzS'), ('timeout', 10), ('txnTime', 1511271022), ('type', '109'), ('version', '1.2.47'), ('seqNo', 1)]) 2017-11-21 13:45:44,761 | INFO | upgrader.py ( 305) | handleUpgradeTxn | Node 'Node1' handles upgrade txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1511271022953271), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-21T13:50:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-21T13:40:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-21T13:45:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-21T13:35:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', '3NRsExGbSDFBeVe8Negs1fJuWJTugu2xq1V6eSVHhGNeSULAeBdzn9VNyv4wwYSnKjhc94n4k2SSm6k7AVMoSHzS'), ('timeout', 10), ('txnTime', 1511271022), ('type', '109'), ('version', '1.2.47'), ('seqNo', 1)]) 2017-11-21 13:45:44,761 | INFO | upgrader.py ( 325) | handleUpgradeTxn | Node 'Node1' has already performed an upgrade with upgrade_id 15112710229532711. Last recorded event is ('succeeded', datetime.datetime(2017, 11, 21, 13, 35, 0, 258870, tzinfo=tzlocal()), '1.2.47', '15112710229532711') 2017-11-21 13:45:44,761 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 2, caught up 0 in total 2017-11-21 13:45:44,778 | INFO | node.py (1551) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 1 2017-11-21 13:45:44,778 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 1, caught up 0 in total 2017-11-21 13:45:44,779 | INFO | node.py (1598) | allLedgersCaughtUp | CATCH-UP: Node1 does not need any more catchups 2017-11-21 13:45:44,817 | DISPLAY | primary_selector.py ( 327) | _start_selection | PRIMARY SELECTION: Node1:0 selected primary Node4:0 for instance 0 (view 3) 2017-11-21 13:45:44,817 | INFO | node.py ( 532) | start_participating | Node1 started participating 2017-11-21 13:45:44,818 | DISPLAY | primary_selector.py ( 351) | _start_selection | VIEW CHANGE: Node1:0 declares view change 3 as completed for instance 0, new primary is Node4:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 6, '3ru9mawZCCRe7brebbogXMNBwceurrrDPC1FGvNkrsks')] 2017-11-21 13:45:44,818 | DISPLAY | primary_selector.py ( 327) | _start_selection | PRIMARY SELECTION: Node1:1 selected primary Node1:1 for instance 1 (view 3) 2017-11-21 13:45:44,819 | DISPLAY | primary_selector.py ( 351) | _start_selection | VIEW CHANGE: Node1:1 declares view change 3 as completed for instance 1, new primary is Node1:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 6, '3ru9mawZCCRe7brebbogXMNBwceurrrDPC1FGvNkrsks')] 2017-11-21 13:45:49,714 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node1 now connected to Node3 2017-11-21 13:45:49,746 | INFO | propagator.py ( 179) | propagate | Node1 propagating request ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1511271949735623) from client Node3 2017-11-21 13:45:49,827 | INFO | replica.py (1619) | order_3pc_key | Node1:0 ordered batch request, view no 3, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 8rGzHyRHHaJnkbvGHDNZ2FYkrz3MgQ6jmuU3mGUjhANi, requests ordered [('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1511271949735623)], discarded [] 2017-11-21 13:45:49,828 | INFO | node.py (2374) | executeBatch | Node1 committed batch request, view no 3, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 8rGzHyRHHaJnkbvGHDNZ2FYkrz3MgQ6jmuU3mGUjhANi, requests: [('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1511271949735623)] 2017-11-21 13:45:49,832 | INFO | replica.py (1619) | order_3pc_key | Node1:1 ordered batch request, view no 3, ppSeqNo 1, ledger 2, state root None, txn root None, requests ordered [('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1511271949735623)], discarded [] 2017-11-21 13:49:59,801 | INFO | propagator.py ( 179) | propagate | Node1 propagating request ('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 1511272199791267) from client Node4 2017-11-21 13:49:59,861 | INFO | replica.py (1619) | order_3pc_key | Node1:0 ordered batch request, view no 3, ppSeqNo 2, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root ATezckL25jagwcHSbqTSshUXpLCExPPpkaatxvqjnfz7, requests ordered [('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 1511272199791267)], discarded [] 2017-11-21 13:49:59,861 | INFO | replica.py (1619) | order_3pc_key | Node1:1 ordered batch request, view no 3, ppSeqNo 2, ledger 2, state root None, txn root None, requests ordered [('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 1511272199791267)], discarded [] 2017-11-21 13:49:59,862 | INFO | node.py (2374) | executeBatch | Node1 committed batch request, view no 3, ppSeqNo 2, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root ATezckL25jagwcHSbqTSshUXpLCExPPpkaatxvqjnfz7, requests: [('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 1511272199791267)] 2017-11-21 13:50:32,405 | INFO | keep_in_touch.py ( 93) | _connsChanged | CONNECTION: Node1 disconnected from Node4 2017-11-21 13:50:32,406 | INFO | node.py ( 884) | onConnsChanged | Node1 lost connection to primary of master 2017-11-21 13:50:34,410 | INFO | node.py (2128) | sendInstanceChange | VIEW CHANGE: Node1 sending an instance change with view_no 4 since Primary of master protocol instance disconnected 2017-11-21 13:50:34,410 | INFO | node.py (2131) | sendInstanceChange | MONITORING: Node1 metrics for monitor: Node1 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [1726178.003684762, 1726178.003981781] ordered request counts: {0: 2, 1: 2} ordered request durations: {0: 0.10458710789680481, 1: 0.10782596189528704} master request latencies: {} client avg request latencies: [{'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': (1, 0.05758804199285805), '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': (1, 0.04699906590394676)}, {'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': (1, 0.06077529489994049), '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': (1, 0.047050666995346546)}] throughput: {0: 19.1228158060684, 1: 18.548408609999314} master throughput: 19.1228158060684 total requests: 6 avg backup throughput: 18.548408609999314 master throughput ratio: 1.030968004217862 2017-11-21 13:50:34,411 | INFO | node.py (2191) | propose_view_change | Node1 sent view change since was disconnected from primary for too long 2017-11-21 13:50:34,424 | INFO | node.py (2031) | do_view_change_if_possible | VIEW CHANGE: Node1 initiating a view change to 4 from 3 2017-11-21 13:50:34,425 | INFO | node.py (2241) | startViewChange | VIEW CHANGE: Node1 changed to view 4, will start catchup now 2017-11-21 13:50:34,441 | INFO | node.py (1551) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 0 2017-11-21 13:50:34,442 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 0, caught up 0 in total 2017-11-21 13:50:34,469 | INFO | node.py (1551) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 2 2017-11-21 13:50:34,470 | INFO | upgrader.py ( 224) | processLedger | Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv found upgrade START txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1511271022953271), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-21T13:50:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-21T13:40:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-21T13:45:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-21T13:35:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', '3NRsExGbSDFBeVe8Negs1fJuWJTugu2xq1V6eSVHhGNeSULAeBdzn9VNyv4wwYSnKjhc94n4k2SSm6k7AVMoSHzS'), ('timeout', 10), ('txnTime', 1511271022), ('type', '109'), ('version', '1.2.47'), ('seqNo', 1)]) 2017-11-21 13:50:34,470 | INFO | upgrader.py ( 305) | handleUpgradeTxn | Node 'Node1' handles upgrade txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1511271022953271), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-21T13:50:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-21T13:40:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-21T13:45:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-21T13:35:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', '3NRsExGbSDFBeVe8Negs1fJuWJTugu2xq1V6eSVHhGNeSULAeBdzn9VNyv4wwYSnKjhc94n4k2SSm6k7AVMoSHzS'), ('timeout', 10), ('txnTime', 1511271022), ('type', '109'), ('version', '1.2.47'), ('seqNo', 1)]) 2017-11-21 13:50:34,471 | INFO | upgrader.py ( 325) | handleUpgradeTxn | Node 'Node1' has already performed an upgrade with upgrade_id 15112710229532711. Last recorded event is ('succeeded', datetime.datetime(2017, 11, 21, 13, 35, 0, 258870, tzinfo=tzlocal()), '1.2.47', '15112710229532711') 2017-11-21 13:50:34,471 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 2, caught up 0 in total 2017-11-21 13:50:34,485 | INFO | node.py (1551) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 1 2017-11-21 13:50:34,485 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 1, caught up 0 in total 2017-11-21 13:50:34,486 | INFO | node.py (1598) | allLedgersCaughtUp | CATCH-UP: Node1 does not need any more catchups 2017-11-21 13:50:34,525 | DISPLAY | primary_selector.py ( 327) | _start_selection | PRIMARY SELECTION: Node1:0 selected primary Node1:0 for instance 0 (view 4) 2017-11-21 13:50:34,525 | INFO | node.py ( 532) | start_participating | Node1 started participating 2017-11-21 13:50:34,525 | DISPLAY | primary_selector.py ( 351) | _start_selection | VIEW CHANGE: Node1:0 declares view change 4 as completed for instance 0, new primary is Node1:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 8, 'ATezckL25jagwcHSbqTSshUXpLCExPPpkaatxvqjnfz7')] 2017-11-21 13:50:34,526 | DISPLAY | primary_selector.py ( 327) | _start_selection | PRIMARY SELECTION: Node1:1 selected primary Node2:1 for instance 1 (view 4) 2017-11-21 13:50:34,526 | DISPLAY | primary_selector.py ( 351) | _start_selection | VIEW CHANGE: Node1:1 declares view change 4 as completed for instance 1, new primary is Node2:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 8, 'ATezckL25jagwcHSbqTSshUXpLCExPPpkaatxvqjnfz7')] 2017-11-21 13:50:44,025 | INFO | propagator.py ( 179) | propagate | Node1 propagating request ('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 1511272244012595) from client Node4 2017-11-21 13:50:44,061 | INFO | replica.py (1619) | order_3pc_key | Node1:0 ordered batch request, view no 4, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 4k5nzTnx5EMvpLGBsdBgtciuYMZMKjYhqwTUbbhDhyWW, requests ordered [('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 1511272244012595)], discarded [] 2017-11-21 13:50:44,061 | INFO | replica.py (1619) | order_3pc_key | Node1:1 ordered batch request, view no 4, ppSeqNo 1, ledger 2, state root None, txn root None, requests ordered [('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 1511272244012595)], discarded [] 2017-11-21 13:50:44,062 | INFO | node.py (2374) | executeBatch | Node1 committed batch request, view no 4, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 4k5nzTnx5EMvpLGBsdBgtciuYMZMKjYhqwTUbbhDhyWW, requests: [('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', 1511272244012595)] 2017-11-21 13:50:45,295 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node1 now connected to Node4 2017-11-21 14:39:49,446 | INFO | propagator.py ( 179) | propagate | Node1 propagating request ('V4SGRU86Z58d6TV7PBUe6f', 1511275189432767) from client b'yg4t8ejbZ%PBSioTlg:nv%PU[llx1L4:u:iAG[Hu' 2017-11-21 14:39:49,504 | INFO | replica.py (2025) | enqueue_prepare | Node1:1 queueing prepare due to unavailability of PRE-PREPARE. Prepare PREPARE{'ppTime': 1511275189, 'txnRootHash': None, 'stateRootHash': None, 'digest': 'a8eb42083175738c7050a4c2d9c5b5e668c55c79777174e0d15deece64e7fda8', 'instId': 1, 'ppSeqNo': 2, 'viewNo': 4} from Node4:1 2017-11-21 14:39:49,509 | INFO | replica.py (1619) | order_3pc_key | Node1:1 ordered batch request, view no 4, ppSeqNo 2, ledger 1, state root None, txn root None, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1511275189432767)], discarded [] 2017-11-21 14:39:49,537 | INFO | replica.py (1619) | order_3pc_key | Node1:0 ordered batch request, view no 4, ppSeqNo 2, ledger 1, state root DneLpHxR3mGWEMP1r9d6s7KcipxQkSu7CBFqsRg2N5qv, txn root D8n7Qh8qBHg95PCbEKsjycmBnRN7MmVjZMCm16ojR3L6, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1511275189432767)], discarded [] 2017-11-21 14:39:49,545 | INFO | node.py (2374) | executeBatch | Node1 committed batch request, view no 4, ppSeqNo 2, ledger 1, state root DneLpHxR3mGWEMP1r9d6s7KcipxQkSu7CBFqsRg2N5qv, txn root D8n7Qh8qBHg95PCbEKsjycmBnRN7MmVjZMCm16ojR3L6, requests: [('V4SGRU86Z58d6TV7PBUe6f', 1511275189432767)] 2017-11-21 14:42:40,360 | INFO | pool_manager.py ( 409) | _order_node | Node1 node Node1 ordered, NYM Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv 2017-11-21 14:42:40,361 | INFO | pool_manager.py ( 409) | _order_node | Node1 node Node2 ordered, NYM 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb 2017-11-21 14:42:40,361 | INFO | pool_manager.py ( 409) | _order_node | Node1 node Node3 ordered, NYM DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya 2017-11-21 14:42:40,361 | INFO | pool_manager.py ( 409) | _order_node | Node1 node Node4 ordered, NYM 4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA 2017-11-21 14:42:40,382 | INFO | node.py ( 617) | _create_bls_bft | BLS: BLS Signatures will be used for Node Node1 2017-11-21 14:42:40,391 | INFO | node.py ( 479) | setPoolParams | Node1 updated its pool parameters: f 1, totalNodes 4, allNodeNames {'Node3', 'Node2', 'Node1', 'Node4'}, requiredNumberOfInstances 2, minimumNodes 3, quorums {'same_consistency_proof': Quorum(2), 'reply': Quorum(2), 'f': 1, 'timestamp': Quorum(2), 'ledger_status': Quorum(2), 'prepare': Quorum(2), 'view_change': Quorum(3), 'bls_signatures': Quorum(3), 'propagate_primary': Quorum(2), 'view_change_done': Quorum(3), 'election': Quorum(3), 'commit': Quorum(3), 'consistency_proof': Quorum(2), 'checkpoint': Quorum(2), 'propagate': Quorum(2)} 2017-11-21 14:42:40,433 | INFO | plugin_loader.py ( 117) | _load | plugin FirebaseStatsConsumer successfully loaded from module plugin_firebase_stats_consumer 2017-11-21 14:42:40,434 | DISPLAY | replicas.py ( 40) | grow | Node1 added replica Node1:0 to instance 0 (master) 2017-11-21 14:42:40,434 | DISPLAY | replicas.py ( 40) | grow | Node1 added replica Node1:1 to instance 1 (backup) 2017-11-21 14:42:40,487 | INFO | node.py (2503) | initStateFromLedger | Node1 found state to be empty, recreating from ledger 2017-11-21 14:42:40,489 | INFO | stacks.py ( 84) | start | CONNECTION: Node1 listening for other nodes at 0.0.0.0:9701 2017-11-21 14:42:40,489 | INFO | node.py ( 680) | start | Node1 first time running... 2017-11-21 14:42:40,490 | INFO | zstack.py ( 585) | connect | CONNECTION: Node1 looking for Node3 at 10.0.0.4:9705 2017-11-21 14:42:40,492 | INFO | zstack.py ( 585) | connect | CONNECTION: Node1 looking for Node2 at 10.0.0.3:9703 2017-11-21 14:42:40,492 | INFO | zstack.py ( 585) | connect | CONNECTION: Node1 looking for Node4 at 10.0.0.5:9707 2017-11-21 14:42:41,515 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node1 now connected to Node2 2017-11-21 14:42:42,263 | INFO | node.py (1551) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 0 2017-11-21 14:42:42,264 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 0, caught up 0 in total 2017-11-21 14:42:42,265 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node1 now connected to Node3 2017-11-21 14:42:42,281 | INFO | node.py (1551) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 2 2017-11-21 14:42:42,281 | INFO | upgrader.py ( 224) | processLedger | Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv found upgrade START txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1511271022953271), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-21T13:50:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-21T13:40:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-21T13:45:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-21T13:35:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', '3NRsExGbSDFBeVe8Negs1fJuWJTugu2xq1V6eSVHhGNeSULAeBdzn9VNyv4wwYSnKjhc94n4k2SSm6k7AVMoSHzS'), ('timeout', 10), ('txnTime', 1511271022), ('type', '109'), ('version', '1.2.47'), ('seqNo', 1)]) 2017-11-21 14:42:42,281 | INFO | upgrader.py ( 305) | handleUpgradeTxn | Node 'Node1' handles upgrade txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1511271022953271), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-21T13:50:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-21T13:40:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-21T13:45:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-21T13:35:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', '3NRsExGbSDFBeVe8Negs1fJuWJTugu2xq1V6eSVHhGNeSULAeBdzn9VNyv4wwYSnKjhc94n4k2SSm6k7AVMoSHzS'), ('timeout', 10), ('txnTime', 1511271022), ('type', '109'), ('version', '1.2.47'), ('seqNo', 1)]) 2017-11-21 14:42:42,281 | INFO | upgrader.py ( 325) | handleUpgradeTxn | Node 'Node1' has already performed an upgrade with upgrade_id 15112710229532711. Last recorded event is ('succeeded', datetime.datetime(2017, 11, 21, 13, 35, 0, 258870, tzinfo=tzlocal()), '1.2.47', '15112710229532711') 2017-11-21 14:42:42,282 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 2, caught up 0 in total 2017-11-21 14:42:42,299 | INFO | node.py (1551) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 1 2017-11-21 14:42:42,299 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 1, caught up 0 in total 2017-11-21 14:42:42,299 | INFO | node.py (1598) | allLedgersCaughtUp | CATCH-UP: Node1 does not need any more catchups 2017-11-21 14:42:42,301 | DISPLAY | primary_selector.py ( 327) | _start_selection | PRIMARY SELECTION: Node1:0 selected primary Node1:0 for instance 0 (view 0) 2017-11-21 14:42:42,301 | INFO | node.py ( 532) | start_participating | Node1 started participating 2017-11-21 14:42:42,301 | DISPLAY | primary_selector.py ( 351) | _start_selection | VIEW CHANGE: Node1:0 declares view change 0 as completed for instance 0, new primary is Node1:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 16, 'D8n7Qh8qBHg95PCbEKsjycmBnRN7MmVjZMCm16ojR3L6'), (2, 9, '4k5nzTnx5EMvpLGBsdBgtciuYMZMKjYhqwTUbbhDhyWW')] 2017-11-21 14:42:42,301 | DISPLAY | primary_selector.py ( 327) | _start_selection | PRIMARY SELECTION: Node1:1 selected primary Node2:1 for instance 1 (view 0) 2017-11-21 14:42:42,302 | DISPLAY | primary_selector.py ( 351) | _start_selection | VIEW CHANGE: Node1:1 declares view change 0 as completed for instance 1, new primary is Node2:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 16, 'D8n7Qh8qBHg95PCbEKsjycmBnRN7MmVjZMCm16ojR3L6'), (2, 9, '4k5nzTnx5EMvpLGBsdBgtciuYMZMKjYhqwTUbbhDhyWW')] 2017-11-21 14:42:42,929 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node1 now connected to Node4 2017-11-21 14:43:40,495 | INFO | node.py (2128) | sendInstanceChange | VIEW CHANGE: Node1 sending an instance change with view_no 1 since Primary of master protocol instance disconnected 2017-11-21 14:43:40,495 | INFO | node.py (2131) | sendInstanceChange | MONITORING: Node1 metrics for monitor: Node1 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [1730208.643603919, 1730208.643878167] ordered request counts: {0: 0, 1: 0} ordered request durations: {0: 0, 1: 0} master request latencies: {} client avg request latencies: [{}, {}] throughput: {0: 0, 1: 0} master throughput: None total requests: 0 avg backup throughput: None master throughput ratio: None 2017-11-21 14:43:40,495 | INFO | node.py (2191) | propose_view_change | Node1 sent view change since was disconnected from primary for too long 2017-11-21 14:43:58,086 | INFO | propagator.py ( 179) | propagate | Node1 propagating request ('V4SGRU86Z58d6TV7PBUe6f', 1511275438071041) from client b'yg4t8ejbZ%PBSioTlg:nv%PU[llx1L4:u:iAG[Hu' 2017-11-21 14:43:58,147 | WARNING | node.py (2591) | reportSuspiciousNode | Node1 raised suspicion on node Node3 for Commit message has invalid BLS signature; suspicion code is 31 2017-11-21 14:43:58,147 | INFO | replica.py (1619) | order_3pc_key | Node1:1 ordered batch request, view no 0, ppSeqNo 1, ledger 1, state root None, txn root None, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1511275438071041)], discarded [] 2017-11-21 14:43:58,152 | WARNING | node.py (2591) | reportSuspiciousNode | Node1 raised suspicion on node Node4 for Commit message has invalid BLS signature; suspicion code is 31 2017-11-21 14:43:58,153 | WARNING | node.py (2591) | reportSuspiciousNode | Node1 raised suspicion on node Node2 for Commit message has invalid BLS signature; suspicion code is 31 2017-11-21 14:44:55,218 | INFO | propagator.py ( 179) | propagate | Node1 propagating request ('V4SGRU86Z58d6TV7PBUe6f', 1511275495206223) from client b'yg4t8ejbZ%PBSioTlg:nv%PU[llx1L4:u:iAG[Hu' 2017-11-21 14:44:55,270 | WARNING | node.py (2591) | reportSuspiciousNode | Node1 raised suspicion on node Node4 for Commit message has invalid BLS signature; suspicion code is 31 2017-11-21 14:44:55,270 | INFO | replica.py (1619) | order_3pc_key | Node1:1 ordered batch request, view no 0, ppSeqNo 2, ledger 1, state root None, txn root None, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1511275495206223)], discarded [] 2017-11-21 14:44:55,273 | WARNING | node.py (2591) | reportSuspiciousNode | Node1 raised suspicion on node Node3 for Commit message has invalid BLS signature; suspicion code is 31 2017-11-21 14:44:55,286 | WARNING | node.py (2591) | reportSuspiciousNode | Node1 raised suspicion on node Node2 for Commit message has invalid BLS signature; suspicion code is 31 2017-11-21 14:45:07,336 | INFO | pool_manager.py ( 409) | _order_node | Node1 node Node1 ordered, NYM Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv 2017-11-21 14:45:07,336 | INFO | pool_manager.py ( 409) | _order_node | Node1 node Node2 ordered, NYM 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb 2017-11-21 14:45:07,336 | INFO | pool_manager.py ( 409) | _order_node | Node1 node Node3 ordered, NYM DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya 2017-11-21 14:45:07,336 | INFO | pool_manager.py ( 409) | _order_node | Node1 node Node4 ordered, NYM 4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA 2017-11-21 14:45:07,360 | INFO | node.py ( 617) | _create_bls_bft | BLS: BLS Signatures will be used for Node Node1 2017-11-21 14:45:07,365 | INFO | node.py ( 479) | setPoolParams | Node1 updated its pool parameters: f 1, totalNodes 4, allNodeNames {'Node2', 'Node4', 'Node1', 'Node3'}, requiredNumberOfInstances 2, minimumNodes 3, quorums {'view_change_done': Quorum(3), 'propagate': Quorum(2), 'f': 1, 'ledger_status': Quorum(2), 'timestamp': Quorum(2), 'same_consistency_proof': Quorum(2), 'view_change': Quorum(3), 'reply': Quorum(2), 'propagate_primary': Quorum(2), 'prepare': Quorum(2), 'bls_signatures': Quorum(3), 'election': Quorum(3), 'checkpoint': Quorum(2), 'consistency_proof': Quorum(2), 'commit': Quorum(3)} 2017-11-21 14:45:07,406 | INFO | plugin_loader.py ( 117) | _load | plugin FirebaseStatsConsumer successfully loaded from module plugin_firebase_stats_consumer 2017-11-21 14:45:07,407 | DISPLAY | replicas.py ( 40) | grow | Node1 added replica Node1:0 to instance 0 (master) 2017-11-21 14:45:07,407 | DISPLAY | replicas.py ( 40) | grow | Node1 added replica Node1:1 to instance 1 (backup) 2017-11-21 14:45:07,440 | INFO | node.py (2503) | initStateFromLedger | Node1 found state to be empty, recreating from ledger 2017-11-21 14:45:07,442 | INFO | stacks.py ( 84) | start | CONNECTION: Node1 listening for other nodes at 0.0.0.0:9701 2017-11-21 14:45:07,443 | INFO | node.py ( 680) | start | Node1 first time running... 2017-11-21 14:45:07,444 | INFO | zstack.py ( 585) | connect | CONNECTION: Node1 looking for Node4 at 10.0.0.5:9707 2017-11-21 14:45:07,446 | INFO | zstack.py ( 585) | connect | CONNECTION: Node1 looking for Node2 at 10.0.0.3:9703 2017-11-21 14:45:07,446 | INFO | zstack.py ( 585) | connect | CONNECTION: Node1 looking for Node3 at 10.0.0.4:9705 2017-11-21 14:45:07,531 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node1 now connected to Node3 2017-11-21 14:45:07,564 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node1 now connected to Node2 2017-11-21 14:45:07,660 | INFO | node.py (2031) | do_view_change_if_possible | VIEW CHANGE: Node1 initiating a view change to 1 from 0 2017-11-21 14:45:07,661 | INFO | node.py (2241) | startViewChange | VIEW CHANGE: Node1 changed to view 1, will start catchup now 2017-11-21 14:45:07,683 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node1 now connected to Node4 2017-11-21 14:45:07,721 | INFO | keep_in_touch.py ( 93) | _connsChanged | CONNECTION: Node1 disconnected from Node2 2017-11-21 14:45:11,701 | INFO | keep_in_touch.py ( 93) | _connsChanged | CONNECTION: Node1 disconnected from Node3 2017-11-21 14:45:12,980 | INFO | node.py (1551) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 0 2017-11-21 14:45:12,981 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 0, caught up 0 in total 2017-11-21 14:45:13,053 | INFO | node.py (1551) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 2 2017-11-21 14:45:13,054 | INFO | upgrader.py ( 224) | processLedger | Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv found upgrade START txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1511271022953271), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-21T13:50:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-21T13:40:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-21T13:45:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-21T13:35:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', '3NRsExGbSDFBeVe8Negs1fJuWJTugu2xq1V6eSVHhGNeSULAeBdzn9VNyv4wwYSnKjhc94n4k2SSm6k7AVMoSHzS'), ('timeout', 10), ('txnTime', 1511271022), ('type', '109'), ('version', '1.2.47'), ('seqNo', 1)]) 2017-11-21 14:45:13,054 | INFO | upgrader.py ( 305) | handleUpgradeTxn | Node 'Node1' handles upgrade txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1511271022953271), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-21T13:50:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-21T13:40:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-21T13:45:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-21T13:35:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', '3NRsExGbSDFBeVe8Negs1fJuWJTugu2xq1V6eSVHhGNeSULAeBdzn9VNyv4wwYSnKjhc94n4k2SSm6k7AVMoSHzS'), ('timeout', 10), ('txnTime', 1511271022), ('type', '109'), ('version', '1.2.47'), ('seqNo', 1)]) 2017-11-21 14:45:13,054 | INFO | upgrader.py ( 325) | handleUpgradeTxn | Node 'Node1' has already performed an upgrade with upgrade_id 15112710229532711. Last recorded event is ('succeeded', datetime.datetime(2017, 11, 21, 13, 35, 0, 258870, tzinfo=tzlocal()), '1.2.47', '15112710229532711') 2017-11-21 14:45:13,055 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 2, caught up 0 in total 2017-11-21 14:45:13,088 | INFO | node.py (1551) | preLedgerCatchUp | Node1 reverted 0 batches before starting catch up for ledger 1 2017-11-21 14:45:13,089 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node1 completed catching up ledger 1, caught up 0 in total 2017-11-21 14:45:13,089 | INFO | node.py (1598) | allLedgersCaughtUp | CATCH-UP: Node1 does not need any more catchups 2017-11-21 14:45:13,216 | INFO | propagator.py ( 179) | propagate | Node1 propagating request ('V4SGRU86Z58d6TV7PBUe6f', 1511275438071041) from client b'yg4t8ejbZ%PBSioTlg:nv%PU[llx1L4:u:iAG[Hu' 2017-11-21 14:45:13,216 | INFO | propagator.py ( 179) | propagate | Node1 propagating request ('V4SGRU86Z58d6TV7PBUe6f', 1511275495206223) from client b'yg4t8ejbZ%PBSioTlg:nv%PU[llx1L4:u:iAG[Hu' 2017-11-21 14:45:13,918 | INFO | keep_in_touch.py ( 93) | _connsChanged | CONNECTION: Node1 disconnected from Node4 2017-11-21 14:45:22,470 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node1 now connected to Node4 2017-11-21 14:45:22,472 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node1 now connected to Node2 2017-11-21 14:45:22,472 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node1 now connected to Node3 2017-11-21 14:46:07,444 | INFO | node.py (2128) | sendInstanceChange | VIEW CHANGE: Node1 sending an instance change with view_no 2 since Primary of master protocol instance disconnected 2017-11-21 14:46:07,444 | INFO | node.py (2131) | sendInstanceChange | MONITORING: Node1 metrics for monitor: Node1 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [1730355.616697451, 1730355.617007189] ordered request counts: {0: 0, 1: 0} ordered request durations: {0: 0, 1: 0} master request latencies: {} client avg request latencies: [{}, {}] throughput: {0: 0, 1: 0} master throughput: None total requests: 0 avg backup throughput: None master throughput ratio: None 2017-11-21 14:46:07,444 | INFO | node.py (2191) | propose_view_change | Node1 sent view change since was disconnected from primary for too long 2017-11-21 14:46:07,661 | INFO | node.py (1085) | _check_view_change_completed | view change to view 1 is not completed in time, starting view change for view 2 2017-11-21 14:46:07,661 | INFO | node.py (1087) | _check_view_change_completed | VIEW CHANGE: Node1 initiating a view change to 2 from 1 2017-11-21 14:46:10,061 | INFO | message_processor.py ( 29) | discard | Node1 discarding message INSTANCE_CHANGE{'viewNo': 1, 'reason': 26} because Received instance change request with view no 1 which is not more than its view no 1 2017-11-21 14:46:14,122 | INFO | message_processor.py ( 29) | discard | Node1 discarding message INSTANCE_CHANGE{'viewNo': 1, 'reason': 26} because Received instance change request with view no 1 which is not more than its view no 1 2017-11-21 14:46:16,058 | INFO | message_processor.py ( 29) | discard | Node1 discarding message INSTANCE_CHANGE{'viewNo': 1, 'reason': 26} because Received instance change request with view no 1 which is not more than its view no 1 2017-11-21 14:46:16,132 | DISPLAY | primary_selector.py ( 327) | _start_selection | PRIMARY SELECTION: Node1:0 selected primary Node2:0 for instance 0 (view 1) 2017-11-21 14:46:16,132 | INFO | node.py ( 532) | start_participating | Node1 started participating 2017-11-21 14:46:16,133 | DISPLAY | primary_selector.py ( 351) | _start_selection | VIEW CHANGE: Node1:0 declares view change 1 as completed for instance 0, new primary is Node2:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 16, 'D8n7Qh8qBHg95PCbEKsjycmBnRN7MmVjZMCm16ojR3L6'), (2, 9, '4k5nzTnx5EMvpLGBsdBgtciuYMZMKjYhqwTUbbhDhyWW')] 2017-11-21 14:46:16,133 | DISPLAY | primary_selector.py ( 327) | _start_selection | PRIMARY SELECTION: Node1:1 selected primary Node3:1 for instance 1 (view 1) 2017-11-21 14:46:16,133 | DISPLAY | primary_selector.py ( 351) | _start_selection | VIEW CHANGE: Node1:1 declares view change 1 as completed for instance 1, new primary is Node3:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 16, 'D8n7Qh8qBHg95PCbEKsjycmBnRN7MmVjZMCm16ojR3L6'), (2, 9, '4k5nzTnx5EMvpLGBsdBgtciuYMZMKjYhqwTUbbhDhyWW')] 2017-11-21 14:46:16,154 | WARNING | node.py (2591) | reportSuspiciousNode | Node1 raised suspicion on node Node3 for Commit message has invalid BLS signature; suspicion code is 31 2017-11-21 14:46:16,157 | WARNING | node.py (2591) | reportSuspiciousNode | Node1 raised suspicion on node Node4 for Commit message has invalid BLS signature; suspicion code is 31 2017-11-21 14:46:16,171 | WARNING | node.py (2591) | reportSuspiciousNode | Node1 raised suspicion on node Node2 for Commit message has invalid BLS signature; suspicion code is 31 2017-11-21 14:46:16,172 | INFO | replica.py (1619) | order_3pc_key | Node1:1 ordered batch request, view no 1, ppSeqNo 1, ledger 1, state root None, txn root None, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1511275438071041), ('V4SGRU86Z58d6TV7PBUe6f', 1511275495206223)], discarded [] 2017-11-21 15:02:31,042 | INFO | propagator.py ( 179) | propagate | Node1 propagating request ('V4SGRU86Z58d6TV7PBUe6f', 1511276551030155) from client b'yg4t8ejbZ%PBSioTlg:nv%PU[llx1L4:u:iAG[Hu' 2017-11-21 15:02:31,117 | INFO | replica.py (1619) | order_3pc_key | Node1:1 ordered batch request, view no 1, ppSeqNo 2, ledger 1, state root None, txn root None, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1511276551030155)], discarded [] 2017-11-21 15:02:31,160 | WARNING | node.py (2591) | reportSuspiciousNode | Node1 raised suspicion on node Node4 for Commit message has invalid BLS signature; suspicion code is 31 2017-11-21 15:02:31,161 | WARNING | node.py (2591) | reportSuspiciousNode | Node1 raised suspicion on node Node2 for Commit message has invalid BLS signature; suspicion code is 31 2017-11-21 15:02:31,161 | WARNING | node.py (2591) | reportSuspiciousNode | Node1 raised suspicion on node Node3 for Commit message has invalid BLS signature; suspicion code is 31 2017-11-21 15:02:38,091 | INFO | monitor.py ( 343) | isMasterThroughputTooLow | MONITORING: Node1 master throughput ratio 0.0 is lower than Delta 0.4. 2017-11-21 15:02:38,091 | INFO | node.py (2128) | sendInstanceChange | VIEW CHANGE: Node1 sending an instance change with view_no 2 since Primary of master protocol instance degraded the performance 2017-11-21 15:02:38,091 | INFO | node.py (2131) | sendInstanceChange | MONITORING: Node1 metrics for monitor: Node1 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [1730355.616697451, 1730355.617007189] ordered request counts: {0: 0, 1: 3} ordered request durations: {0: 0, 1: 125.9544536753092} master request latencies: {} client avg request latencies: [{}, {'V4SGRU86Z58d6TV7PBUe6f': (3, 41.98481789176973)}] throughput: {0: 0, 1: 0.023818133559084215} master throughput: 0 total requests: 0 avg backup throughput: 0.023818133559084215 master throughput ratio: 0.0 2017-11-21 15:02:38,092 | INFO | node.py (2075) | checkPerformance | Node1 sent view change since performance degraded of master instance 2017-11-21 15:39:59,878 | INFO | propagator.py ( 179) | propagate | Node1 propagating request ('V4SGRU86Z58d6TV7PBUe6f', 1511278799838029) from client .pQ[L6htk(fAQ76HZO.x9sRh!s:V!o894BgpH44h 2017-11-21 15:39:59,911 | WARNING | node.py (2591) | reportSuspiciousNode | Node1 raised suspicion on node Node3 for Commit message has invalid BLS signature; suspicion code is 31 2017-11-21 15:39:59,913 | INFO | replica.py (1619) | order_3pc_key | Node1:1 ordered batch request, view no 1, ppSeqNo 3, ledger 1, state root None, txn root None, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1511278799838029)], discarded [] 2017-11-21 15:39:59,937 | WARNING | node.py (2591) | reportSuspiciousNode | Node1 raised suspicion on node Node4 for Commit message has invalid BLS signature; suspicion code is 31 2017-11-21 15:39:59,937 | WARNING | node.py (2591) | reportSuspiciousNode | Node1 raised suspicion on node Node2 for Commit message has invalid BLS signature; suspicion code is 31 2017-11-21 15:40:09,435 | INFO | monitor.py ( 343) | isMasterThroughputTooLow | MONITORING: Node1 master throughput ratio 0.0 is lower than Delta 0.4. 2017-11-21 15:40:09,435 | INFO | node.py (2128) | sendInstanceChange | VIEW CHANGE: Node1 sending an instance change with view_no 2 since Primary of master protocol instance degraded the performance 2017-11-21 15:40:09,436 | INFO | node.py (2131) | sendInstanceChange | MONITORING: Node1 metrics for monitor: Node1 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [1730355.616697451, 1730355.617007189] ordered request counts: {0: 0, 1: 4} ordered request durations: {0: 0, 1: 125.98922909423709} master request latencies: {} client avg request latencies: [{}, {'V4SGRU86Z58d6TV7PBUe6f': (4, 31.497307273559272)}] throughput: {0: 0, 1: 0.031748745736098524} master throughput: 0 total requests: 0 avg backup throughput: 0.031748745736098524 master throughput ratio: 0.0 2017-11-21 15:40:09,436 | INFO | node.py (2075) | checkPerformance | Node1 sent view change since performance degraded of master instance