2017-11-17 13:43:44,267 | INFO | node.py (2420) | initStateFromLedger | Node2 found state to be empty, recreating from ledger 2017-11-17 13:43:44,350 | INFO | node.py (2420) | initStateFromLedger | Node2 found state to be empty, recreating from ledger 2017-11-17 13:43:44,396 | INFO | plugin_loader.py ( 117) | _load | plugin FirebaseStatsConsumer successfully loaded from module plugin_firebase_stats_consumer 2017-11-17 13:43:44,398 | DISPLAY | replicas.py ( 36) | grow | Node2 added replica Node2:0 to instance 0 (master) 2017-11-17 13:43:44,399 | DISPLAY | replicas.py ( 36) | grow | Node2 added replica Node2:1 to instance 1 (backup) 2017-11-17 13:43:44,487 | INFO | node.py (2420) | initStateFromLedger | Node2 found state to be empty, recreating from ledger 2017-11-17 13:43:44,489 | INFO | stacks.py ( 84) | start | CONNECTION: Node2 listening for other nodes at 0.0.0.0:9703 2017-11-17 13:43:44,489 | INFO | node.py ( 621) | start | Node2 first time running... 2017-11-17 13:43:44,490 | INFO | zstack.py ( 590) | connect | CONNECTION: Node2 looking for Node1 at 10.0.0.2:9701 2017-11-17 13:43:44,492 | INFO | zstack.py ( 590) | connect | CONNECTION: Node2 looking for Node4 at 10.0.0.5:9707 2017-11-17 13:43:44,492 | INFO | zstack.py ( 590) | connect | CONNECTION: Node2 looking for Node3 at 10.0.0.4:9705 2017-11-17 13:43:44,572 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node2 now connected to Node1 2017-11-17 13:43:44,572 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node2 now connected to Node3 2017-11-17 13:43:44,597 | INFO | node.py (1489) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 0 2017-11-17 13:43:44,597 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 0, caught up 0 in total 2017-11-17 13:43:44,623 | INFO | node.py (1489) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 2 2017-11-17 13:43:44,624 | INFO | upgrader.py ( 151) | should_notify_about_upgrade_result | Node's 'Node2' last upgrade txn is None 2017-11-17 13:43:44,624 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 2, caught up 0 in total 2017-11-17 13:43:44,652 | INFO | node.py (1489) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 1 2017-11-17 13:43:44,652 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 1, caught up 0 in total 2017-11-17 13:43:44,652 | INFO | node.py (1537) | allLedgersCaughtUp | CATCH-UP: Node2 does not need any more catchups 2017-11-17 13:43:44,680 | DISPLAY | primary_selector.py ( 291) | _startSelection | PRIMARY SELECTION: Node2:0 selected primary Node1:0 for instance 0 (view 0) 2017-11-17 13:43:44,680 | INFO | node.py ( 510) | start_participating | Node2 started participating 2017-11-17 13:43:44,681 | DISPLAY | primary_selector.py ( 315) | _startSelection | VIEW CHANGE: Node2:0 declares view change 0 as completed for instance 0, new primary is Node1:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-11-17 13:43:44,681 | DISPLAY | primary_selector.py ( 291) | _startSelection | PRIMARY SELECTION: Node2:1 selected primary Node2:1 for instance 1 (view 0) 2017-11-17 13:43:44,681 | DISPLAY | primary_selector.py ( 315) | _startSelection | VIEW CHANGE: Node2:1 declares view change 0 as completed for instance 1, new primary is Node2:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-11-17 13:43:45,137 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node2 now connected to Node4 2017-11-17 13:45:38,705 | INFO | message_processor.py ( 29) | discard | Node2 discarding message ({'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'KNpuf1rjcogYVHpD982n5b3tY527KEDAdKDbuVfSDTY95x1jUkULk3buoEzHuGj6KLjfHSNN6wcdkF9dfztPSHe', 'operation': {'action': 'start', 'justification': None, 'reinstall': False, 'force': False, 'name': 'upgrade_to_the_latest_full', 'type': '109', 'timeout': 10, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-11-17T13:50:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-11-17T13:55:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-11-17T14:00:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-11-17T13:45:00.258870+00:00'}, 'version': '1.2.45'}, 'reqId': 1510926338687540}, b'=cQu#Y-Ir.lyHj/JuuLEhk!lc&(9BYFL7fT8TyD}') because InvalidClientRequest("{'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-11-17T13:50:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-11-17T13:55:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-11-17T14:00:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-11-17T13:45:00.258870+00:00'} not a valid schedule since 2017-11-17 13:45:00.258870+00:00 is less than current time",) 2017-11-17 13:46:06,339 | INFO | propagator.py ( 148) | propagate | Node2 propagating request ('V4SGRU86Z58d6TV7PBUe6f', 1510926366313517) from client b'=cQu#Y-Ir.lyHj/JuuLEhk!lc&(9BYFL7fT8TyD}' 2017-11-17 13:46:06,401 | INFO | replica.py (1483) | order_3pc_key | Node2:1 ordered batch request, view no 0, ppSeqNo 1, ledger 2, state root None, txn root None, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1510926366313517)], discarded [] 2017-11-17 13:46:06,409 | INFO | replica.py (1483) | order_3pc_key | Node2:0 ordered batch request, view no 0, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root XjH6SHaWYaxmk1gyUBigBTfZctf6HT46vK3WgUhceQG, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1510926366313517)], discarded [] 2017-11-17 13:46:06,411 | INFO | upgrader.py ( 292) | handleUpgradeTxn | Node 'Node2' handles upgrade txn {'seqNo': 1, 'justification': None, 'auditPath': [], 'name': 'upgrade_to_the_latest_full', 'type': '109', 'timeout': 10, 'rootHash': 'XjH6SHaWYaxmk1gyUBigBTfZctf6HT46vK3WgUhceQG', 'reqId': 1510926366313517, 'action': 'start', 'reinstall': False, 'txnTime': 1510926366, 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'signature': 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj', 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'schedule': {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-11-17T13:52:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-11-17T13:57:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-11-17T13:47:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-11-17T14:02:00.258870+00:00'}, 'version': '1.2.45', 'force': False} 2017-11-17 13:46:06,411 | INFO | upgrader.py ( 321) | handleUpgradeTxn | Node 'Node2' schedules upgrade to 1.2.45 2017-11-17 13:46:06,412 | INFO | upgrader.py ( 359) | _scheduleUpgrade | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb's upgrader processing upgrade for version 1.2.45 2017-11-17 13:46:06,412 | INFO | node.py (2290) | executeBatch | Node2 committed batch request, view no 0, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root XjH6SHaWYaxmk1gyUBigBTfZctf6HT46vK3WgUhceQG, requests: [('V4SGRU86Z58d6TV7PBUe6f', 1510926366313517)] 2017-11-17 13:46:59,434 | INFO | propagator.py ( 148) | propagate | Node2 propagating request ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926419422000) from client Node1 2017-11-17 13:46:59,466 | INFO | replica.py (1483) | order_3pc_key | Node2:0 ordered batch request, view no 0, ppSeqNo 2, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 2PDWCTTViFKEEVQvxGsg6s7VZm4rCmmBu8G1VLJhEdom, requests ordered [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926419422000)], discarded [] 2017-11-17 13:46:59,467 | INFO | replica.py (1483) | order_3pc_key | Node2:1 ordered batch request, view no 0, ppSeqNo 2, ledger 2, state root None, txn root None, requests ordered [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926419422000)], discarded [] 2017-11-17 13:46:59,468 | INFO | node.py (2290) | executeBatch | Node2 committed batch request, view no 0, ppSeqNo 2, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 2PDWCTTViFKEEVQvxGsg6s7VZm4rCmmBu8G1VLJhEdom, requests: [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926419422000)] 2017-11-17 13:47:30,420 | INFO | keep_in_touch.py ( 93) | _connsChanged | CONNECTION: Node2 disconnected from Node1 2017-11-17 13:47:30,420 | INFO | node.py ( 823) | onConnsChanged | Node2 lost connection to primary of master 2017-11-17 13:47:32,421 | INFO | node.py (2048) | sendInstanceChange | VIEW CHANGE: Node2 sending an instance change with view_no 1 since Primary of master protocol instance disconnected 2017-11-17 13:47:32,422 | INFO | node.py (2051) | sendInstanceChange | MONITORING: Node2 metrics for monitor: Node2 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [1381072.607847083, 1381072.608642995] ordered request counts: {0: 2, 1: 2} ordered request durations: {0: 0.08592366566881537, 1: 0.07085515977814794} master request latencies: {} client avg request latencies: [{'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': (1, 0.02899836585856974), 'V4SGRU86Z58d6TV7PBUe6f': (1, 0.05692529981024563)}, {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': (1, 0.02907823189161718), 'V4SGRU86Z58d6TV7PBUe6f': (1, 0.04177692788653076)}] throughput: {0: 23.276474350021452, 1: 28.22659642942206} master throughput: 23.276474350021452 total requests: 2 avg backup throughput: 28.22659642942206 master throughput ratio: 0.824629154571366 2017-11-17 13:47:32,422 | INFO | node.py (2108) | propose_view_change | Node2 sent view change since was disconnected from primary for too long 2017-11-17 13:47:32,435 | INFO | node.py (1954) | do_view_change_if_possible | VIEW CHANGE: Node2 initiating a view change to 1 from 0 2017-11-17 13:47:32,435 | INFO | node.py (2159) | startViewChange | VIEW CHANGE: Node2 changed to view 1, will start catchup now 2017-11-17 13:47:32,436 | INFO | message_processor.py ( 29) | discard | Node2 discarding message INSTANCE_CHANGE{'viewNo': 1, 'reason': 26} because Received instance change request with view no 1 which is not more than its view no 1 2017-11-17 13:47:32,464 | INFO | node.py (1489) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 0 2017-11-17 13:47:32,465 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 0, caught up 0 in total 2017-11-17 13:47:32,493 | INFO | node.py (1489) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 2 2017-11-17 13:47:32,494 | INFO | upgrader.py ( 214) | processLedger | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb found upgrade START txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)]) 2017-11-17 13:47:32,494 | INFO | upgrader.py ( 292) | handleUpgradeTxn | Node 'Node2' handles upgrade txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)]) 2017-11-17 13:47:32,494 | INFO | upgrader.py ( 321) | handleUpgradeTxn | Node 'Node2' schedules upgrade to 1.2.45 2017-11-17 13:47:32,494 | INFO | upgrader.py ( 326) | handleUpgradeTxn | Node 'Node2' cancels previous upgrade and schedules a new one to 1.2.45 2017-11-17 13:47:32,494 | INFO | upgrader.py ( 390) | _cancelScheduledUpgrade | Cancelling upgrade of node 'Node2' to version 1.2.45 due to some reason 2017-11-17 13:47:32,494 | INFO | upgrader.py ( 359) | _scheduleUpgrade | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb's upgrader processing upgrade for version 1.2.45 2017-11-17 13:47:32,495 | INFO | upgrader.py ( 151) | should_notify_about_upgrade_result | Node's 'Node2' last upgrade txn is None 2017-11-17 13:47:32,496 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 2, caught up 0 in total 2017-11-17 13:47:32,511 | INFO | node.py (1489) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 1 2017-11-17 13:47:32,511 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 1, caught up 0 in total 2017-11-17 13:47:32,512 | INFO | node.py (1537) | allLedgersCaughtUp | CATCH-UP: Node2 does not need any more catchups 2017-11-17 13:47:32,526 | DISPLAY | primary_selector.py ( 291) | _startSelection | PRIMARY SELECTION: Node2:0 selected primary Node2:0 for instance 0 (view 1) 2017-11-17 13:47:32,526 | INFO | node.py ( 510) | start_participating | Node2 started participating 2017-11-17 13:47:32,526 | DISPLAY | primary_selector.py ( 315) | _startSelection | VIEW CHANGE: Node2:0 declares view change 1 as completed for instance 0, new primary is Node2:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 2, '2PDWCTTViFKEEVQvxGsg6s7VZm4rCmmBu8G1VLJhEdom')] 2017-11-17 13:47:32,526 | DISPLAY | primary_selector.py ( 291) | _startSelection | PRIMARY SELECTION: Node2:1 selected primary Node3:1 for instance 1 (view 1) 2017-11-17 13:47:32,527 | DISPLAY | primary_selector.py ( 315) | _startSelection | VIEW CHANGE: Node2:1 declares view change 1 as completed for instance 1, new primary is Node3:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 2, '2PDWCTTViFKEEVQvxGsg6s7VZm4rCmmBu8G1VLJhEdom')] 2017-11-17 13:47:44,612 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node2 now connected to Node1 2017-11-17 13:47:44,666 | INFO | message_processor.py ( 29) | discard | Node2 discarding message {'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'operation': {'type': '110', 'data': {'action': 'complete', 'version': '1.2.45'}, 'signature': '5Rjt3r36mqM669ZoEcccqa6zJU5UEdxDLrfZD3UAJrWeN3hbvPCsVnzqmUootfcgMsfm43MzMkq1vXcXyveN6VjN'}, 'reqId': 1510926464650411, 'signature': '485QNuqefeLN4d7RrAyUYtK7egvRBsMhzADLYScsLmJivg8FKytaWwbPnj82pxd6FrSbch5znp8M5bn1gsRwraFn', 'protocolVersion': 1} because 'SafeRequest' object has no attribute 'schema_is_strict' 2017-11-17 13:51:59,501 | INFO | upgrader.py ( 421) | _callUpgradeAgent | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb's upgrader calling agent for upgrade 2017-11-17 13:51:59,501 | INFO | node.py ( 254) | notify_upgrade_start | Node2 is about to be upgraded, sending NODE_UPGRADE 2017-11-17 13:51:59,505 | INFO | upgrader.py ( 433) | _sendUpdateRequest | Sending message to control tool: {"version": "1.2.45"} 2017-11-17 13:51:59,507 | INFO | upgrader.py ( 452) | _sendUpdateRequest | Waiting 10 minutes for upgrade to be performed 2017-11-17 13:51:59,522 | INFO | propagator.py ( 148) | propagate | Node2 propagating request ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926719502400) from client Node2 2017-11-17 13:51:59,543 | INFO | replica.py (1483) | order_3pc_key | Node2:1 ordered batch request, view no 1, ppSeqNo 1, ledger 2, state root None, txn root None, requests ordered [('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926719502400)], discarded [] 2017-11-17 13:51:59,546 | INFO | replica.py (1483) | order_3pc_key | Node2:0 ordered batch request, view no 1, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root BvBE3KNL7XqXP3gkNvaS9mSi1veDkzTj32rY5mpWBQbk, requests ordered [('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926719502400)], discarded [] 2017-11-17 13:51:59,547 | INFO | node.py (2290) | executeBatch | Node2 committed batch request, view no 1, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root BvBE3KNL7XqXP3gkNvaS9mSi1veDkzTj32rY5mpWBQbk, requests: [('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926719502400)] 2017-11-17 13:52:32,632 | INFO | node.py (2503) | initStateFromLedger | Node2 found state to be empty, recreating from ledger 2017-11-17 13:52:32,635 | INFO | pool_manager.py ( 409) | _order_node | Node2 node Node1 ordered, NYM Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv 2017-11-17 13:52:32,635 | INFO | pool_manager.py ( 409) | _order_node | Node2 node Node2 ordered, NYM 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb 2017-11-17 13:52:32,635 | INFO | pool_manager.py ( 409) | _order_node | Node2 node Node3 ordered, NYM DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya 2017-11-17 13:52:32,635 | INFO | pool_manager.py ( 409) | _order_node | Node2 node Node4 ordered, NYM 4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA 2017-11-17 13:52:32,650 | WARNING | node.py ( 623) | _create_bls_bft | BLS: Transactions will not be BLS signed by this Node, since BLS keys were not found. Please make sure that a script to init BLS keys was called (init_bls_keys), and NODE txn was sent with BLS public keys. 2017-11-17 13:52:32,656 | INFO | node.py (2503) | initStateFromLedger | Node2 found state to be empty, recreating from ledger 2017-11-17 13:52:32,668 | INFO | node.py ( 479) | setPoolParams | Node2 updated its pool parameters: f 1, totalNodes 4, allNodeNames {'Node1', 'Node2', 'Node3', 'Node4'}, requiredNumberOfInstances 2, minimumNodes 3, quorums {'propagate_primary': Quorum(2), 'propagate': Quorum(2), 'commit': Quorum(3), 'view_change_done': Quorum(3), 'f': 1, 'consistency_proof': Quorum(2), 'election': Quorum(3), 'ledger_status': Quorum(2), 'view_change': Quorum(3), 'same_consistency_proof': Quorum(2), 'reply': Quorum(2), 'timestamp': Quorum(2), 'checkpoint': Quorum(2), 'bls_signatures': Quorum(3), 'prepare': Quorum(2)} 2017-11-17 13:52:32,705 | INFO | plugin_loader.py ( 117) | _load | plugin FirebaseStatsConsumer successfully loaded from module plugin_firebase_stats_consumer 2017-11-17 13:52:32,706 | DISPLAY | replicas.py ( 40) | grow | Node2 added replica Node2:0 to instance 0 (master) 2017-11-17 13:52:32,707 | DISPLAY | replicas.py ( 40) | grow | Node2 added replica Node2:1 to instance 1 (backup) 2017-11-17 13:52:32,748 | INFO | upgrader.py ( 140) | check_upgrade_succeeded | Node 'Node2' successfully upgraded to version 1.2.45 2017-11-17 13:52:32,748 | INFO | node.py (2503) | initStateFromLedger | Node2 found state to be empty, recreating from ledger 2017-11-17 13:52:32,750 | INFO | stacks.py ( 84) | start | CONNECTION: Node2 listening for other nodes at 0.0.0.0:9703 2017-11-17 13:52:32,750 | INFO | node.py ( 680) | start | Node2 first time running... 2017-11-17 13:52:32,751 | INFO | zstack.py ( 585) | connect | CONNECTION: Node2 looking for Node1 at 10.0.0.2:9701 2017-11-17 13:52:32,752 | INFO | zstack.py ( 585) | connect | CONNECTION: Node2 looking for Node4 at 10.0.0.5:9707 2017-11-17 13:52:32,753 | INFO | zstack.py ( 585) | connect | CONNECTION: Node2 looking for Node3 at 10.0.0.4:9705 2017-11-17 13:52:32,819 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node2 now connected to Node3 2017-11-17 13:52:32,845 | INFO | node.py (2044) | _start_view_change_if_possible | VIEW CHANGE: Node2 starting view change for 2 after 2 view change indications from other nodes 2017-11-17 13:52:32,846 | INFO | node.py (2241) | startViewChange | VIEW CHANGE: Node2 changed to view 2, will start catchup now 2017-11-17 13:52:32,847 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node2 now connected to Node4 2017-11-17 13:52:32,862 | INFO | node.py (1551) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 0 2017-11-17 13:52:32,862 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 0, caught up 0 in total 2017-11-17 13:52:32,863 | INFO | node.py (1551) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 2 2017-11-17 13:52:32,863 | INFO | upgrader.py ( 212) | processLedger | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb found upgrade START txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)]) 2017-11-17 13:52:32,864 | INFO | upgrader.py ( 290) | handleUpgradeTxn | Node 'Node2' handles upgrade txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)]) 2017-11-17 13:52:32,864 | INFO | upgrader.py ( 310) | handleUpgradeTxn | Node 'Node2' has already performed an upgrade with upgrade_id 15109263663135171. Last recorded event is ('succeeded', datetime.datetime(2017, 11, 17, 13, 52, 0, 258870, tzinfo=tzlocal()), '1.2.45', '15109263663135171') 2017-11-17 13:52:32,864 | INFO | node.py (1551) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 1 2017-11-17 13:52:32,865 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 1, caught up 0 in total 2017-11-17 13:52:32,865 | INFO | node.py (1588) | allLedgersCaughtUp | CATCH-UP: Node2 caught up till (1, 1) 2017-11-17 13:52:32,866 | INFO | node.py (1598) | allLedgersCaughtUp | CATCH-UP: Node2 does not need any more catchups 2017-11-17 13:52:32,866 | DISPLAY | primary_selector.py ( 327) | _start_selection | PRIMARY SELECTION: Node2:0 selected primary Node3:0 for instance 0 (view 2) 2017-11-17 13:52:32,866 | INFO | node.py ( 532) | start_participating | Node2 started participating 2017-11-17 13:52:32,867 | DISPLAY | primary_selector.py ( 351) | _start_selection | VIEW CHANGE: Node2:0 declares view change 2 as completed for instance 0, new primary is Node3:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 3, 'BvBE3KNL7XqXP3gkNvaS9mSi1veDkzTj32rY5mpWBQbk')] 2017-11-17 13:52:32,867 | DISPLAY | primary_selector.py ( 327) | _start_selection | PRIMARY SELECTION: Node2:1 selected primary Node4:1 for instance 1 (view 2) 2017-11-17 13:52:32,870 | DISPLAY | primary_selector.py ( 351) | _start_selection | VIEW CHANGE: Node2:1 declares view change 2 as completed for instance 1, new primary is Node4:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 3, 'BvBE3KNL7XqXP3gkNvaS9mSi1veDkzTj32rY5mpWBQbk')] 2017-11-17 13:52:32,872 | INFO | upgrader.py ( 149) | should_notify_about_upgrade_result | Node's 'Node2' last upgrade txn is OrderedDict([('data', OrderedDict([('action', 'in_progress'), ('version', '1.2.45')])), ('identifier', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb'), ('reqId', 1510926719502400), ('signature', '59fs2tWoGeAcMnUon3maEHpzyNZhNULgk4zctNgfGRSB3AE7D3ciyXpicauLCRwpKzzknWaQhK2V1Btj73K6B2Kp'), ('txnTime', 1510926719), ('type', '110'), ('seqNo', 3)]) 2017-11-17 13:52:32,873 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 2, caught up 0 in total 2017-11-17 13:52:32,874 | INFO | node.py (1598) | allLedgersCaughtUp | CATCH-UP: Node2 does not need any more catchups 2017-11-17 13:52:32,874 | INFO | node.py (1685) | no_more_catchups_needed | Node2 starting to participate since catchup is done, primaries are selected but mode was not set to participating 2017-11-17 13:52:32,874 | INFO | node.py ( 532) | start_participating | Node2 started participating 2017-11-17 13:52:32,920 | INFO | message_processor.py ( 29) | discard | Node2 discarding message INSTANCE_CHANGE{'viewNo': 2, 'reason': 26} because Received instance change request with view no 2 which is not more than its view no 2 2017-11-17 13:52:32,922 | INFO | propagator.py ( 179) | propagate | Node2 propagating request ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926748095846) from client Node1 2017-11-17 13:52:32,925 | INFO | propagator.py ( 179) | propagate | Node2 propagating request ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926752873422) from client Node2 2017-11-17 13:52:32,927 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node2 now connected to Node1 2017-11-17 13:57:00,104 | INFO | propagator.py ( 179) | propagate | Node2 propagating request ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927020097708) from client Node3 2017-11-17 13:57:00,152 | INFO | replica.py (1619) | order_3pc_key | Node2:0 ordered batch request, view no 2, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 3aErFDAegiipoxjKyq8YBzyNpDPHaoNhZ8BqVu4e3uCg, requests ordered [('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927020097708)], discarded [] 2017-11-17 13:57:00,153 | INFO | replica.py (1619) | order_3pc_key | Node2:1 ordered batch request, view no 2, ppSeqNo 1, ledger 2, state root None, txn root None, requests ordered [('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927020097708)], discarded [] 2017-11-17 13:57:00,155 | INFO | node.py (2374) | executeBatch | Node2 committed batch request, view no 2, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 3aErFDAegiipoxjKyq8YBzyNpDPHaoNhZ8BqVu4e3uCg, requests: [('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927020097708)] 2017-11-17 13:57:25,919 | INFO | keep_in_touch.py ( 93) | _connsChanged | CONNECTION: Node2 disconnected from Node3 2017-11-17 13:57:25,919 | INFO | node.py ( 884) | onConnsChanged | Node2 lost connection to primary of master 2017-11-17 13:57:27,920 | INFO | node.py (2128) | sendInstanceChange | VIEW CHANGE: Node2 sending an instance change with view_no 3 since Primary of master protocol instance disconnected 2017-11-17 13:57:27,920 | INFO | node.py (2131) | sendInstanceChange | MONITORING: Node2 metrics for monitor: Node2 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [1381600.91592481, 1381600.916206756] ordered request counts: {0: 1, 1: 1} ordered request durations: {0: 0.04119092784821987, 1: 0.04124294384382665} master request latencies: {} client avg request latencies: [{'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': (1, 0.04119092784821987)}, {'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': (1, 0.04124294384382665)}] throughput: {0: 24.27719044554653, 1: 24.246571820543856} master throughput: 24.27719044554653 total requests: 1 avg backup throughput: 24.246571820543856 master throughput ratio: 1.0012628022315604 2017-11-17 13:57:27,920 | INFO | node.py (2191) | propose_view_change | Node2 sent view change since was disconnected from primary for too long 2017-11-17 13:57:27,945 | INFO | node.py (2031) | do_view_change_if_possible | VIEW CHANGE: Node2 initiating a view change to 3 from 2 2017-11-17 13:57:27,945 | INFO | node.py (2241) | startViewChange | VIEW CHANGE: Node2 changed to view 3, will start catchup now 2017-11-17 13:57:27,975 | INFO | node.py (1551) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 0 2017-11-17 13:57:27,975 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 0, caught up 0 in total 2017-11-17 13:57:27,994 | INFO | node.py (1551) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 2 2017-11-17 13:57:27,994 | INFO | upgrader.py ( 212) | processLedger | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb found upgrade START txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)]) 2017-11-17 13:57:27,995 | INFO | upgrader.py ( 290) | handleUpgradeTxn | Node 'Node2' handles upgrade txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)]) 2017-11-17 13:57:27,995 | INFO | upgrader.py ( 310) | handleUpgradeTxn | Node 'Node2' has already performed an upgrade with upgrade_id 15109263663135171. Last recorded event is ('succeeded', datetime.datetime(2017, 11, 17, 13, 52, 0, 258870, tzinfo=tzlocal()), '1.2.45', '15109263663135171') 2017-11-17 13:57:27,995 | INFO | upgrader.py ( 149) | should_notify_about_upgrade_result | Node's 'Node2' last upgrade txn is OrderedDict([('data', OrderedDict([('action', 'in_progress'), ('version', '1.2.45')])), ('identifier', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb'), ('reqId', 1510926719502400), ('signature', '59fs2tWoGeAcMnUon3maEHpzyNZhNULgk4zctNgfGRSB3AE7D3ciyXpicauLCRwpKzzknWaQhK2V1Btj73K6B2Kp'), ('txnTime', 1510926719), ('type', '110'), ('seqNo', 3)]) 2017-11-17 13:57:27,996 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 2, caught up 0 in total 2017-11-17 13:57:28,013 | INFO | propagator.py ( 179) | propagate | Node2 propagating request ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927048003928) from client Node1 2017-11-17 13:57:28,013 | INFO | node.py (1551) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 1 2017-11-17 13:57:28,014 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 1, caught up 0 in total 2017-11-17 13:57:28,014 | INFO | node.py (1598) | allLedgersCaughtUp | CATCH-UP: Node2 does not need any more catchups 2017-11-17 13:57:28,015 | INFO | propagator.py ( 179) | propagate | Node2 propagating request ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927047995990) from client Node2 2017-11-17 13:57:28,033 | DISPLAY | primary_selector.py ( 327) | _start_selection | PRIMARY SELECTION: Node2:0 selected primary Node4:0 for instance 0 (view 3) 2017-11-17 13:57:28,033 | INFO | node.py ( 532) | start_participating | Node2 started participating 2017-11-17 13:57:28,034 | DISPLAY | primary_selector.py ( 351) | _start_selection | VIEW CHANGE: Node2:0 declares view change 3 as completed for instance 0, new primary is Node4:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 4, '3aErFDAegiipoxjKyq8YBzyNpDPHaoNhZ8BqVu4e3uCg')] 2017-11-17 13:57:28,034 | DISPLAY | primary_selector.py ( 327) | _start_selection | PRIMARY SELECTION: Node2:1 selected primary Node1:1 for instance 1 (view 3) 2017-11-17 13:57:28,034 | DISPLAY | primary_selector.py ( 351) | _start_selection | VIEW CHANGE: Node2:1 declares view change 3 as completed for instance 1, new primary is Node1:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 4, '3aErFDAegiipoxjKyq8YBzyNpDPHaoNhZ8BqVu4e3uCg')] 2017-11-17 13:57:28,152 | INFO | keep_in_touch.py ( 93) | _connsChanged | CONNECTION: Node2 disconnected from Node4 2017-11-17 13:57:28,152 | INFO | node.py ( 884) | onConnsChanged | Node2 lost connection to primary of master 2017-11-17 13:57:30,156 | INFO | node.py (2128) | sendInstanceChange | VIEW CHANGE: Node2 sending an instance change with view_no 4 since Primary of master protocol instance disconnected 2017-11-17 13:57:30,156 | INFO | node.py (2131) | sendInstanceChange | MONITORING: Node2 metrics for monitor: Node2 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [1381600.91592481, 1381600.916206756] ordered request counts: {0: 0, 1: 0} ordered request durations: {0: 0, 1: 0} master request latencies: {} client avg request latencies: [{}, {}] throughput: {0: 0, 1: 0} master throughput: None total requests: 1 avg backup throughput: None master throughput ratio: None 2017-11-17 13:57:30,156 | INFO | node.py (2191) | propose_view_change | Node2 sent view change since was disconnected from primary for too long 2017-11-17 13:57:32,933 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node2 now connected to Node3 2017-11-17 13:57:37,501 | INFO | propagator.py ( 179) | propagate | Node2 propagating request ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927057486576) from client Node3 2017-11-17 13:57:37,539 | INFO | replica.py (1619) | order_3pc_key | Node2:1 ordered batch request, view no 3, ppSeqNo 1, ledger 2, state root None, txn root None, requests ordered [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926748095846), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926752873422), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927048003928), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927047995990)], discarded [] 2017-11-17 13:57:37,553 | INFO | replica.py (1619) | order_3pc_key | Node2:1 ordered batch request, view no 3, ppSeqNo 2, ledger 2, state root None, txn root None, requests ordered [('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927057486576)], discarded [] 2017-11-17 13:57:42,608 | INFO | node.py (2031) | do_view_change_if_possible | VIEW CHANGE: Node2 initiating a view change to 4 from 3 2017-11-17 13:57:42,609 | INFO | node.py (2241) | startViewChange | VIEW CHANGE: Node2 changed to view 4, will start catchup now 2017-11-17 13:57:42,643 | INFO | node.py (1551) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 0 2017-11-17 13:57:42,644 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 0, caught up 0 in total 2017-11-17 13:57:42,660 | INFO | propagator.py ( 179) | propagate | Node2 propagating request ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927062651136) from client Node3 2017-11-17 13:57:42,661 | INFO | node.py (1551) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 2 2017-11-17 13:57:42,661 | INFO | upgrader.py ( 212) | processLedger | 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb found upgrade START txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)]) 2017-11-17 13:57:42,661 | INFO | upgrader.py ( 290) | handleUpgradeTxn | Node 'Node2' handles upgrade txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)]) 2017-11-17 13:57:42,661 | INFO | upgrader.py ( 310) | handleUpgradeTxn | Node 'Node2' has already performed an upgrade with upgrade_id 15109263663135171. Last recorded event is ('succeeded', datetime.datetime(2017, 11, 17, 13, 52, 0, 258870, tzinfo=tzlocal()), '1.2.45', '15109263663135171') 2017-11-17 13:57:42,662 | INFO | upgrader.py ( 149) | should_notify_about_upgrade_result | Node's 'Node2' last upgrade txn is OrderedDict([('data', OrderedDict([('action', 'in_progress'), ('version', '1.2.45')])), ('identifier', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb'), ('reqId', 1510926719502400), ('signature', '59fs2tWoGeAcMnUon3maEHpzyNZhNULgk4zctNgfGRSB3AE7D3ciyXpicauLCRwpKzzknWaQhK2V1Btj73K6B2Kp'), ('txnTime', 1510926719), ('type', '110'), ('seqNo', 3)]) 2017-11-17 13:57:42,662 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 2, caught up 0 in total 2017-11-17 13:57:42,664 | INFO | propagator.py ( 179) | propagate | Node2 propagating request ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927062654152) from client Node1 2017-11-17 13:57:42,685 | INFO | propagator.py ( 179) | propagate | Node2 propagating request ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927062662523) from client Node2 2017-11-17 13:57:42,685 | INFO | node.py (1551) | preLedgerCatchUp | Node2 reverted 0 batches before starting catch up for ledger 1 2017-11-17 13:57:42,685 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node2 completed catching up ledger 1, caught up 0 in total 2017-11-17 13:57:42,686 | INFO | node.py (1598) | allLedgersCaughtUp | CATCH-UP: Node2 does not need any more catchups 2017-11-17 13:57:42,687 | DISPLAY | primary_selector.py ( 327) | _start_selection | PRIMARY SELECTION: Node2:0 selected primary Node1:0 for instance 0 (view 4) 2017-11-17 13:57:42,687 | INFO | node.py ( 532) | start_participating | Node2 started participating 2017-11-17 13:57:42,688 | DISPLAY | primary_selector.py ( 351) | _start_selection | VIEW CHANGE: Node2:0 declares view change 4 as completed for instance 0, new primary is Node1:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 4, '3aErFDAegiipoxjKyq8YBzyNpDPHaoNhZ8BqVu4e3uCg')] 2017-11-17 13:57:42,688 | DISPLAY | primary_selector.py ( 327) | _start_selection | PRIMARY SELECTION: Node2:1 selected primary Node2:1 for instance 1 (view 4) 2017-11-17 13:57:42,688 | DISPLAY | primary_selector.py ( 351) | _start_selection | VIEW CHANGE: Node2:1 declares view change 4 as completed for instance 1, new primary is Node2:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 4, '3aErFDAegiipoxjKyq8YBzyNpDPHaoNhZ8BqVu4e3uCg')] 2017-11-17 13:57:42,725 | INFO | replica.py (1619) | order_3pc_key | Node2:0 ordered batch request, view no 4, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 8PHonJAkhGQDvCHhZ336fu17U8UV3wYpJGCjV7gdLdBd, requests ordered [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926748095846), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926752873422), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927048003928), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927047995990), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927057486576), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927062651136), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927062654152), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927062662523)], discarded [] 2017-11-17 13:57:42,726 | INFO | replica.py (1619) | order_3pc_key | Node2:1 ordered batch request, view no 4, ppSeqNo 1, ledger 2, state root None, txn root None, requests ordered [('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927062651136), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927062654152), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927062662523)], discarded [] 2017-11-17 13:57:42,729 | INFO | node.py (2374) | executeBatch | Node2 committed batch request, view no 4, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 8PHonJAkhGQDvCHhZ336fu17U8UV3wYpJGCjV7gdLdBd, requests: [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926748095846), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926752873422), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927048003928), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927047995990), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927057486576), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927062651136), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927062654152), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927062662523)] 2017-11-17 13:57:44,670 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node2 now connected to Node4