2017-11-17 13:43:44,268 | INFO | node.py (2420) | initStateFromLedger | Node3 found state to be empty, recreating from ledger 2017-11-17 13:43:44,351 | INFO | node.py (2420) | initStateFromLedger | Node3 found state to be empty, recreating from ledger 2017-11-17 13:43:44,394 | INFO | plugin_loader.py ( 117) | _load | plugin FirebaseStatsConsumer successfully loaded from module plugin_firebase_stats_consumer 2017-11-17 13:43:44,395 | DISPLAY | replicas.py ( 36) | grow | Node3 added replica Node3:0 to instance 0 (master) 2017-11-17 13:43:44,396 | DISPLAY | replicas.py ( 36) | grow | Node3 added replica Node3:1 to instance 1 (backup) 2017-11-17 13:43:44,487 | INFO | node.py (2420) | initStateFromLedger | Node3 found state to be empty, recreating from ledger 2017-11-17 13:43:44,488 | INFO | stacks.py ( 84) | start | CONNECTION: Node3 listening for other nodes at 0.0.0.0:9705 2017-11-17 13:43:44,488 | INFO | node.py ( 621) | start | Node3 first time running... 2017-11-17 13:43:44,490 | INFO | zstack.py ( 590) | connect | CONNECTION: Node3 looking for Node4 at 10.0.0.5:9707 2017-11-17 13:43:44,491 | INFO | zstack.py ( 590) | connect | CONNECTION: Node3 looking for Node2 at 10.0.0.3:9703 2017-11-17 13:43:44,491 | INFO | zstack.py ( 590) | connect | CONNECTION: Node3 looking for Node1 at 10.0.0.2:9701 2017-11-17 13:43:44,559 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node3 now connected to Node2 2017-11-17 13:43:44,596 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node3 now connected to Node1 2017-11-17 13:43:44,623 | INFO | node.py (1489) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 0 2017-11-17 13:43:44,624 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 0, caught up 0 in total 2017-11-17 13:43:44,651 | INFO | node.py (1489) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 2 2017-11-17 13:43:44,652 | INFO | upgrader.py ( 151) | should_notify_about_upgrade_result | Node's 'Node3' last upgrade txn is None 2017-11-17 13:43:44,652 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 2, caught up 0 in total 2017-11-17 13:43:44,667 | INFO | node.py (1489) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 1 2017-11-17 13:43:44,667 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 1, caught up 0 in total 2017-11-17 13:43:44,668 | INFO | node.py (1537) | allLedgersCaughtUp | CATCH-UP: Node3 does not need any more catchups 2017-11-17 13:43:44,671 | DISPLAY | primary_selector.py ( 291) | _startSelection | PRIMARY SELECTION: Node3:0 selected primary Node1:0 for instance 0 (view 0) 2017-11-17 13:43:44,671 | INFO | node.py ( 510) | start_participating | Node3 started participating 2017-11-17 13:43:44,672 | DISPLAY | primary_selector.py ( 315) | _startSelection | VIEW CHANGE: Node3:0 declares view change 0 as completed for instance 0, new primary is Node1:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-11-17 13:43:44,672 | DISPLAY | primary_selector.py ( 291) | _startSelection | PRIMARY SELECTION: Node3:1 selected primary Node2:1 for instance 1 (view 0) 2017-11-17 13:43:44,672 | DISPLAY | primary_selector.py ( 315) | _startSelection | VIEW CHANGE: Node3:1 declares view change 0 as completed for instance 1, new primary is Node2:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2017-11-17 13:43:45,136 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node3 now connected to Node4 2017-11-17 13:45:38,702 | INFO | message_processor.py ( 29) | discard | Node3 discarding message ({'signature': 'KNpuf1rjcogYVHpD982n5b3tY527KEDAdKDbuVfSDTY95x1jUkULk3buoEzHuGj6KLjfHSNN6wcdkF9dfztPSHe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'reinstall': False, 'justification': None, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'name': 'upgrade_to_the_latest_full', 'action': 'start', 'timeout': 10, 'type': '109', 'version': '1.2.45', 'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-11-17T13:45:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-11-17T14:00:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-11-17T13:50:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-11-17T13:55:00.258870+00:00'}, 'force': False}, 'reqId': 1510926338687540}, b'=cQu#Y-Ir.lyHj/JuuLEhk!lc&(9BYFL7fT8TyD}') because InvalidClientRequest("{'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-11-17T13:45:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-11-17T14:00:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-11-17T13:50:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-11-17T13:55:00.258870+00:00'} not a valid schedule since 2017-11-17 13:45:00.258870+00:00 is less than current time",) 2017-11-17 13:46:06,340 | INFO | propagator.py ( 148) | propagate | Node3 propagating request ('V4SGRU86Z58d6TV7PBUe6f', 1510926366313517) from client b'=cQu#Y-Ir.lyHj/JuuLEhk!lc&(9BYFL7fT8TyD}' 2017-11-17 13:46:06,396 | INFO | replica.py (1483) | order_3pc_key | Node3:1 ordered batch request, view no 0, ppSeqNo 1, ledger 2, state root None, txn root None, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1510926366313517)], discarded [] 2017-11-17 13:46:06,427 | INFO | replica.py (1483) | order_3pc_key | Node3:0 ordered batch request, view no 0, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root XjH6SHaWYaxmk1gyUBigBTfZctf6HT46vK3WgUhceQG, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1510926366313517)], discarded [] 2017-11-17 13:46:06,428 | INFO | upgrader.py ( 292) | handleUpgradeTxn | Node 'Node3' handles upgrade txn {'justification': None, 'signature': 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj', 'timeout': 10, 'reinstall': False, 'version': '1.2.45', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'schedule': {'4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-11-17T14:02:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-11-17T13:47:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-11-17T13:52:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-11-17T13:57:00.258870+00:00'}, 'rootHash': 'XjH6SHaWYaxmk1gyUBigBTfZctf6HT46vK3WgUhceQG', 'txnTime': 1510926366, 'reqId': 1510926366313517, 'auditPath': [], 'name': 'upgrade_to_the_latest_full', 'seqNo': 1, 'action': 'start', 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'type': '109', 'force': False} 2017-11-17 13:46:06,428 | INFO | upgrader.py ( 321) | handleUpgradeTxn | Node 'Node3' schedules upgrade to 1.2.45 2017-11-17 13:46:06,428 | INFO | upgrader.py ( 359) | _scheduleUpgrade | DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya's upgrader processing upgrade for version 1.2.45 2017-11-17 13:46:06,429 | INFO | node.py (2290) | executeBatch | Node3 committed batch request, view no 0, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root XjH6SHaWYaxmk1gyUBigBTfZctf6HT46vK3WgUhceQG, requests: [('V4SGRU86Z58d6TV7PBUe6f', 1510926366313517)] 2017-11-17 13:46:59,447 | INFO | propagator.py ( 148) | propagate | Node3 propagating request ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926419422000) from client Node1 2017-11-17 13:46:59,468 | INFO | replica.py (1483) | order_3pc_key | Node3:0 ordered batch request, view no 0, ppSeqNo 2, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 2PDWCTTViFKEEVQvxGsg6s7VZm4rCmmBu8G1VLJhEdom, requests ordered [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926419422000)], discarded [] 2017-11-17 13:46:59,469 | INFO | replica.py (1483) | order_3pc_key | Node3:1 ordered batch request, view no 0, ppSeqNo 2, ledger 2, state root None, txn root None, requests ordered [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926419422000)], discarded [] 2017-11-17 13:46:59,470 | INFO | node.py (2290) | executeBatch | Node3 committed batch request, view no 0, ppSeqNo 2, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 2PDWCTTViFKEEVQvxGsg6s7VZm4rCmmBu8G1VLJhEdom, requests: [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926419422000)] 2017-11-17 13:47:30,414 | INFO | keep_in_touch.py ( 93) | _connsChanged | CONNECTION: Node3 disconnected from Node1 2017-11-17 13:47:30,414 | INFO | node.py ( 823) | onConnsChanged | Node3 lost connection to primary of master 2017-11-17 13:47:32,419 | INFO | node.py (2048) | sendInstanceChange | VIEW CHANGE: Node3 sending an instance change with view_no 1 since Primary of master protocol instance disconnected 2017-11-17 13:47:32,419 | INFO | node.py (2051) | sendInstanceChange | MONITORING: Node3 metrics for monitor: Node3 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [1381072.604918195, 1381072.605166084] ordered request counts: {0: 2, 1: 2} ordered request durations: {0: 0.08513712720014155, 1: 0.05257508018985391} master request latencies: {} client avg request latencies: [{'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': (1, 0.021839564200490713), 'V4SGRU86Z58d6TV7PBUe6f': (1, 0.06329756299965084)}, {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': (1, 0.021925772074609995), 'V4SGRU86Z58d6TV7PBUe6f': (1, 0.030649308115243912)}] throughput: {0: 23.49151381744855, 1: 38.04083593934234} master throughput: 23.49151381744855 total requests: 2 avg backup throughput: 38.04083593934234 master throughput ratio: 0.6175341113667093 2017-11-17 13:47:32,419 | INFO | node.py (2108) | propose_view_change | Node3 sent view change since was disconnected from primary for too long 2017-11-17 13:47:32,432 | INFO | node.py (1954) | do_view_change_if_possible | VIEW CHANGE: Node3 initiating a view change to 1 from 0 2017-11-17 13:47:32,435 | INFO | node.py (2159) | startViewChange | VIEW CHANGE: Node3 changed to view 1, will start catchup now 2017-11-17 13:47:32,435 | INFO | message_processor.py ( 29) | discard | Node3 discarding message INSTANCE_CHANGE{'viewNo': 1, 'reason': 26} because Received instance change request with view no 1 which is not more than its view no 1 2017-11-17 13:47:32,465 | INFO | node.py (1489) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 0 2017-11-17 13:47:32,466 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 0, caught up 0 in total 2017-11-17 13:47:32,494 | INFO | node.py (1489) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 2 2017-11-17 13:47:32,494 | INFO | upgrader.py ( 214) | processLedger | DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya found upgrade START txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)]) 2017-11-17 13:47:32,495 | INFO | upgrader.py ( 292) | handleUpgradeTxn | Node 'Node3' handles upgrade txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)]) 2017-11-17 13:47:32,495 | INFO | upgrader.py ( 321) | handleUpgradeTxn | Node 'Node3' schedules upgrade to 1.2.45 2017-11-17 13:47:32,495 | INFO | upgrader.py ( 326) | handleUpgradeTxn | Node 'Node3' cancels previous upgrade and schedules a new one to 1.2.45 2017-11-17 13:47:32,495 | INFO | upgrader.py ( 390) | _cancelScheduledUpgrade | Cancelling upgrade of node 'Node3' to version 1.2.45 due to some reason 2017-11-17 13:47:32,495 | INFO | upgrader.py ( 359) | _scheduleUpgrade | DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya's upgrader processing upgrade for version 1.2.45 2017-11-17 13:47:32,496 | INFO | upgrader.py ( 151) | should_notify_about_upgrade_result | Node's 'Node3' last upgrade txn is None 2017-11-17 13:47:32,496 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 2, caught up 0 in total 2017-11-17 13:47:32,514 | INFO | node.py (1489) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 1 2017-11-17 13:47:32,514 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 1, caught up 0 in total 2017-11-17 13:47:32,515 | INFO | node.py (1537) | allLedgersCaughtUp | CATCH-UP: Node3 does not need any more catchups 2017-11-17 13:47:32,516 | DISPLAY | primary_selector.py ( 291) | _startSelection | PRIMARY SELECTION: Node3:0 selected primary Node2:0 for instance 0 (view 1) 2017-11-17 13:47:32,517 | INFO | node.py ( 510) | start_participating | Node3 started participating 2017-11-17 13:47:32,517 | DISPLAY | primary_selector.py ( 315) | _startSelection | VIEW CHANGE: Node3:0 declares view change 1 as completed for instance 0, new primary is Node2:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 2, '2PDWCTTViFKEEVQvxGsg6s7VZm4rCmmBu8G1VLJhEdom')] 2017-11-17 13:47:32,517 | DISPLAY | primary_selector.py ( 291) | _startSelection | PRIMARY SELECTION: Node3:1 selected primary Node3:1 for instance 1 (view 1) 2017-11-17 13:47:32,518 | DISPLAY | primary_selector.py ( 315) | _startSelection | VIEW CHANGE: Node3:1 declares view change 1 as completed for instance 1, new primary is Node3:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 2, '2PDWCTTViFKEEVQvxGsg6s7VZm4rCmmBu8G1VLJhEdom')] 2017-11-17 13:47:44,630 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node3 now connected to Node1 2017-11-17 13:47:44,658 | INFO | message_processor.py ( 29) | discard | Node3 discarding message {'signature': '485QNuqefeLN4d7RrAyUYtK7egvRBsMhzADLYScsLmJivg8FKytaWwbPnj82pxd6FrSbch5znp8M5bn1gsRwraFn', 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'operation': {'signature': '5Rjt3r36mqM669ZoEcccqa6zJU5UEdxDLrfZD3UAJrWeN3hbvPCsVnzqmUootfcgMsfm43MzMkq1vXcXyveN6VjN', 'type': '110', 'data': {'action': 'complete', 'version': '1.2.45'}}, 'protocolVersion': 1, 'reqId': 1510926464650411} because 'SafeRequest' object has no attribute 'schema_is_strict' 2017-11-17 13:51:59,520 | INFO | propagator.py ( 148) | propagate | Node3 propagating request ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926719502400) from client Node2 2017-11-17 13:51:59,542 | INFO | replica.py (1483) | order_3pc_key | Node3:1 ordered batch request, view no 1, ppSeqNo 1, ledger 2, state root None, txn root None, requests ordered [('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926719502400)], discarded [] 2017-11-17 13:51:59,545 | INFO | replica.py (1483) | order_3pc_key | Node3:0 ordered batch request, view no 1, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root BvBE3KNL7XqXP3gkNvaS9mSi1veDkzTj32rY5mpWBQbk, requests ordered [('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926719502400)], discarded [] 2017-11-17 13:51:59,546 | INFO | node.py (2290) | executeBatch | Node3 committed batch request, view no 1, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root BvBE3KNL7XqXP3gkNvaS9mSi1veDkzTj32rY5mpWBQbk, requests: [('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926719502400)] 2017-11-17 13:52:26,037 | INFO | keep_in_touch.py ( 93) | _connsChanged | CONNECTION: Node3 disconnected from Node2 2017-11-17 13:52:26,037 | INFO | node.py ( 823) | onConnsChanged | Node3 lost connection to primary of master 2017-11-17 13:52:28,048 | INFO | node.py (2048) | sendInstanceChange | VIEW CHANGE: Node3 sending an instance change with view_no 2 since Primary of master protocol instance disconnected 2017-11-17 13:52:28,049 | INFO | node.py (2051) | sendInstanceChange | MONITORING: Node3 metrics for monitor: Node3 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [1381072.604918195, 1381072.605166084] ordered request counts: {0: 1, 1: 1} ordered request durations: {0: 0.02551817218773067, 1: 0.02203888911753893} master request latencies: {} client avg request latencies: [{'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': (1, 0.02551817218773067)}, {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': (1, 0.02203888911753893)}] throughput: {0: 39.187759712696334, 1: 45.374337820148234} master throughput: 39.187759712696334 total requests: 3 avg backup throughput: 45.374337820148234 master throughput ratio: 0.8636546910728737 2017-11-17 13:52:28,049 | INFO | node.py (2108) | propose_view_change | Node3 sent view change since was disconnected from primary for too long 2017-11-17 13:52:28,051 | INFO | node.py (1954) | do_view_change_if_possible | VIEW CHANGE: Node3 initiating a view change to 2 from 1 2017-11-17 13:52:28,052 | INFO | node.py (2159) | startViewChange | VIEW CHANGE: Node3 changed to view 2, will start catchup now 2017-11-17 13:52:28,070 | INFO | node.py (1489) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 0 2017-11-17 13:52:28,070 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 0, caught up 0 in total 2017-11-17 13:52:28,086 | INFO | node.py (1489) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 2 2017-11-17 13:52:28,086 | INFO | upgrader.py ( 214) | processLedger | DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya found upgrade START txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)]) 2017-11-17 13:52:28,086 | INFO | upgrader.py ( 292) | handleUpgradeTxn | Node 'Node3' handles upgrade txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)]) 2017-11-17 13:52:28,087 | INFO | upgrader.py ( 321) | handleUpgradeTxn | Node 'Node3' schedules upgrade to 1.2.45 2017-11-17 13:52:28,087 | INFO | upgrader.py ( 326) | handleUpgradeTxn | Node 'Node3' cancels previous upgrade and schedules a new one to 1.2.45 2017-11-17 13:52:28,087 | INFO | upgrader.py ( 390) | _cancelScheduledUpgrade | Cancelling upgrade of node 'Node3' to version 1.2.45 due to some reason 2017-11-17 13:52:28,087 | INFO | upgrader.py ( 359) | _scheduleUpgrade | DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya's upgrader processing upgrade for version 1.2.45 2017-11-17 13:52:28,088 | INFO | upgrader.py ( 151) | should_notify_about_upgrade_result | Node's 'Node3' last upgrade txn is None 2017-11-17 13:52:28,088 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 2, caught up 0 in total 2017-11-17 13:52:28,102 | INFO | message_processor.py ( 29) | discard | Node3 discarding message {'signature': '5uX7Q8zCtLBdMp77gNeszGewZ4YqZo77WRK9dCbjso3EYqUL1F2GQSd8qMbjPiK49JT7k1HJATotnTN5paEvBeYM', 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'operation': {'signature': '5Rjt3r36mqM669ZoEcccqa6zJU5UEdxDLrfZD3UAJrWeN3hbvPCsVnzqmUootfcgMsfm43MzMkq1vXcXyveN6VjN', 'type': '110', 'data': {'action': 'complete', 'version': '1.2.45'}}, 'protocolVersion': 1, 'reqId': 1510926748095846} because 'SafeRequest' object has no attribute 'schema_is_strict' 2017-11-17 13:52:28,103 | INFO | node.py (1489) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 1 2017-11-17 13:52:28,104 | INFO | ledger_manager.py ( 848) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 1, caught up 0 in total 2017-11-17 13:52:28,104 | INFO | node.py (1537) | allLedgersCaughtUp | CATCH-UP: Node3 does not need any more catchups 2017-11-17 13:52:28,119 | DISPLAY | primary_selector.py ( 291) | _startSelection | PRIMARY SELECTION: Node3:0 selected primary Node3:0 for instance 0 (view 2) 2017-11-17 13:52:28,119 | INFO | node.py ( 510) | start_participating | Node3 started participating 2017-11-17 13:52:28,119 | DISPLAY | primary_selector.py ( 315) | _startSelection | VIEW CHANGE: Node3:0 declares view change 2 as completed for instance 0, new primary is Node3:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 3, 'BvBE3KNL7XqXP3gkNvaS9mSi1veDkzTj32rY5mpWBQbk')] 2017-11-17 13:52:28,120 | DISPLAY | primary_selector.py ( 291) | _startSelection | PRIMARY SELECTION: Node3:1 selected primary Node4:1 for instance 1 (view 2) 2017-11-17 13:52:28,120 | DISPLAY | primary_selector.py ( 315) | _startSelection | VIEW CHANGE: Node3:1 declares view change 2 as completed for instance 1, new primary is Node4:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 3, 'BvBE3KNL7XqXP3gkNvaS9mSi1veDkzTj32rY5mpWBQbk')] 2017-11-17 13:52:32,827 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node3 now connected to Node2 2017-11-17 13:52:32,892 | INFO | message_processor.py ( 29) | discard | Node3 discarding message {'signature': '2dvC3j8jPcADXgKBygm4FzmvmeRToTDNs2Tu4LFHkvRLpEm1EbyAtFXNwW1VNDFnYMcNwNWFtK67A1cyy5vZhgEs', 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'operation': {'signature': 'aXn8Rhnak15iQAyvXCt5Vz4Qm4XXhHBxg2CnNidCjAGWCbrxBfQEjhv98Wm67aLMuamt3KrpbTELweqMzyxswqP', 'type': '110', 'data': {'action': 'complete', 'version': '1.2.45'}}, 'protocolVersion': 1, 'reqId': 1510926752873422} because 'SafeRequest' object has no attribute 'schema_is_strict' 2017-11-17 13:52:32,894 | INFO | message_processor.py ( 29) | discard | Node3 discarding message {'senderClient': 'Node2', 'request': {'signature': '2dvC3j8jPcADXgKBygm4FzmvmeRToTDNs2Tu4LFHkvRLpEm1EbyAtFXNwW1VNDFnYMcNwNWFtK67A1cyy5vZhgEs', 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'operation': {'signature': 'aXn8Rhnak15iQAyvXCt5Vz4Qm4XXhHBxg2CnNidCjAGWCbrxBfQEjhv98Wm67aLMuamt3KrpbTELweqMzyxswqP', 'type': '110', 'data': {'action': 'complete', 'version': '1.2.45'}}, 'protocolVersion': 1, 'reqId': 1510926752873422}, 'op': 'PROPAGATE'} because InvalidNodeMsg('validation error [ClientMessageValidator]: unknown field - protocolVersion=1',) 2017-11-17 13:52:32,936 | INFO | message_processor.py ( 29) | discard | Node3 discarding message {'senderClient': 'Node1', 'request': {'signature': '5uX7Q8zCtLBdMp77gNeszGewZ4YqZo77WRK9dCbjso3EYqUL1F2GQSd8qMbjPiK49JT7k1HJATotnTN5paEvBeYM', 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'operation': {'signature': '5Rjt3r36mqM669ZoEcccqa6zJU5UEdxDLrfZD3UAJrWeN3hbvPCsVnzqmUootfcgMsfm43MzMkq1vXcXyveN6VjN', 'type': '110', 'data': {'action': 'complete', 'version': '1.2.45'}}, 'protocolVersion': 1, 'reqId': 1510926748095846}, 'op': 'PROPAGATE'} because InvalidNodeMsg('validation error [ClientMessageValidator]: unknown field - protocolVersion=1',) 2017-11-17 13:52:32,936 | INFO | message_processor.py ( 29) | discard | Node3 discarding message {'senderClient': 'Node2', 'request': {'signature': '2dvC3j8jPcADXgKBygm4FzmvmeRToTDNs2Tu4LFHkvRLpEm1EbyAtFXNwW1VNDFnYMcNwNWFtK67A1cyy5vZhgEs', 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'operation': {'signature': 'aXn8Rhnak15iQAyvXCt5Vz4Qm4XXhHBxg2CnNidCjAGWCbrxBfQEjhv98Wm67aLMuamt3KrpbTELweqMzyxswqP', 'type': '110', 'data': {'action': 'complete', 'version': '1.2.45'}}, 'protocolVersion': 1, 'reqId': 1510926752873422}, 'op': 'PROPAGATE'} because InvalidNodeMsg('validation error [ClientMessageValidator]: unknown field - protocolVersion=1',) 2017-11-17 13:52:32,948 | INFO | message_processor.py ( 29) | discard | Node3 discarding message {'senderClient': 'Node1', 'request': {'signature': '5uX7Q8zCtLBdMp77gNeszGewZ4YqZo77WRK9dCbjso3EYqUL1F2GQSd8qMbjPiK49JT7k1HJATotnTN5paEvBeYM', 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'operation': {'signature': '5Rjt3r36mqM669ZoEcccqa6zJU5UEdxDLrfZD3UAJrWeN3hbvPCsVnzqmUootfcgMsfm43MzMkq1vXcXyveN6VjN', 'type': '110', 'data': {'action': 'complete', 'version': '1.2.45'}}, 'protocolVersion': 1, 'reqId': 1510926748095846}, 'op': 'PROPAGATE'} because InvalidNodeMsg('validation error [ClientMessageValidator]: unknown field - protocolVersion=1',) 2017-11-17 13:57:00,096 | INFO | upgrader.py ( 421) | _callUpgradeAgent | DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya's upgrader calling agent for upgrade 2017-11-17 13:57:00,097 | INFO | node.py ( 254) | notify_upgrade_start | Node3 is about to be upgraded, sending NODE_UPGRADE 2017-11-17 13:57:00,099 | INFO | upgrader.py ( 433) | _sendUpdateRequest | Sending message to control tool: {"version": "1.2.45"} 2017-11-17 13:57:00,100 | INFO | upgrader.py ( 452) | _sendUpdateRequest | Waiting 10 minutes for upgrade to be performed 2017-11-17 13:57:00,117 | INFO | propagator.py ( 148) | propagate | Node3 propagating request ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927020097708) from client Node3 2017-11-17 13:57:00,145 | INFO | replica.py (1483) | order_3pc_key | Node3:1 ordered batch request, view no 2, ppSeqNo 1, ledger 2, state root None, txn root None, requests ordered [('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927020097708)], discarded [] 2017-11-17 13:57:00,160 | INFO | replica.py (1483) | order_3pc_key | Node3:0 ordered batch request, view no 2, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 3aErFDAegiipoxjKyq8YBzyNpDPHaoNhZ8BqVu4e3uCg, requests ordered [('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927020097708)], discarded [] 2017-11-17 13:57:00,161 | INFO | node.py (2290) | executeBatch | Node3 committed batch request, view no 2, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 3aErFDAegiipoxjKyq8YBzyNpDPHaoNhZ8BqVu4e3uCg, requests: [('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927020097708)] 2017-11-17 13:57:32,525 | INFO | node.py (2503) | initStateFromLedger | Node3 found state to be empty, recreating from ledger 2017-11-17 13:57:32,527 | INFO | pool_manager.py ( 409) | _order_node | Node3 node Node1 ordered, NYM Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv 2017-11-17 13:57:32,527 | INFO | pool_manager.py ( 409) | _order_node | Node3 node Node2 ordered, NYM 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb 2017-11-17 13:57:32,527 | INFO | pool_manager.py ( 409) | _order_node | Node3 node Node3 ordered, NYM DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya 2017-11-17 13:57:32,527 | INFO | pool_manager.py ( 409) | _order_node | Node3 node Node4 ordered, NYM 4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA 2017-11-17 13:57:32,540 | WARNING | node.py ( 623) | _create_bls_bft | BLS: Transactions will not be BLS signed by this Node, since BLS keys were not found. Please make sure that a script to init BLS keys was called (init_bls_keys), and NODE txn was sent with BLS public keys. 2017-11-17 13:57:32,543 | INFO | node.py (2503) | initStateFromLedger | Node3 found state to be empty, recreating from ledger 2017-11-17 13:57:32,554 | INFO | node.py ( 479) | setPoolParams | Node3 updated its pool parameters: f 1, totalNodes 4, allNodeNames {'Node3', 'Node2', 'Node4', 'Node1'}, requiredNumberOfInstances 2, minimumNodes 3, quorums {'election': Quorum(3), 'timestamp': Quorum(2), 'f': 1, 'view_change': Quorum(3), 'bls_signatures': Quorum(3), 'propagate_primary': Quorum(2), 'checkpoint': Quorum(2), 'same_consistency_proof': Quorum(2), 'view_change_done': Quorum(3), 'commit': Quorum(3), 'prepare': Quorum(2), 'reply': Quorum(2), 'consistency_proof': Quorum(2), 'propagate': Quorum(2), 'ledger_status': Quorum(2)} 2017-11-17 13:57:32,587 | INFO | plugin_loader.py ( 117) | _load | plugin FirebaseStatsConsumer successfully loaded from module plugin_firebase_stats_consumer 2017-11-17 13:57:32,588 | DISPLAY | replicas.py ( 40) | grow | Node3 added replica Node3:0 to instance 0 (master) 2017-11-17 13:57:32,588 | DISPLAY | replicas.py ( 40) | grow | Node3 added replica Node3:1 to instance 1 (backup) 2017-11-17 13:57:32,639 | INFO | upgrader.py ( 140) | check_upgrade_succeeded | Node 'Node3' successfully upgraded to version 1.2.45 2017-11-17 13:57:32,639 | INFO | node.py (2503) | initStateFromLedger | Node3 found state to be empty, recreating from ledger 2017-11-17 13:57:32,640 | INFO | stacks.py ( 84) | start | CONNECTION: Node3 listening for other nodes at 0.0.0.0:9705 2017-11-17 13:57:32,641 | INFO | node.py ( 680) | start | Node3 first time running... 2017-11-17 13:57:32,642 | INFO | zstack.py ( 585) | connect | CONNECTION: Node3 looking for Node2 at 10.0.0.3:9703 2017-11-17 13:57:32,643 | INFO | zstack.py ( 585) | connect | CONNECTION: Node3 looking for Node4 at 10.0.0.5:9707 2017-11-17 13:57:32,644 | INFO | zstack.py ( 585) | connect | CONNECTION: Node3 looking for Node1 at 10.0.0.2:9701 2017-11-17 13:57:32,777 | INFO | node.py (2031) | do_view_change_if_possible | VIEW CHANGE: Node3 initiating a view change to 3 from 0 2017-11-17 13:57:32,778 | INFO | node.py (2241) | startViewChange | VIEW CHANGE: Node3 changed to view 3, will start catchup now 2017-11-17 13:57:32,781 | INFO | propagator.py ( 179) | propagate | Node3 propagating request ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927047995990) from client Node2 2017-11-17 13:57:32,790 | INFO | propagator.py ( 179) | propagate | Node3 propagating request ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927048003928) from client Node1 2017-11-17 13:57:32,793 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node3 now connected to Node2 2017-11-17 13:57:32,793 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node3 now connected to Node1 2017-11-17 13:57:37,475 | INFO | node.py (1551) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 0 2017-11-17 13:57:37,476 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 0, caught up 0 in total 2017-11-17 13:57:37,476 | INFO | node.py (1551) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 2 2017-11-17 13:57:37,476 | INFO | upgrader.py ( 212) | processLedger | DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya found upgrade START txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)]) 2017-11-17 13:57:37,477 | INFO | upgrader.py ( 290) | handleUpgradeTxn | Node 'Node3' handles upgrade txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)]) 2017-11-17 13:57:37,477 | INFO | upgrader.py ( 310) | handleUpgradeTxn | Node 'Node3' has already performed an upgrade with upgrade_id 15109263663135171. Last recorded event is ('succeeded', datetime.datetime(2017, 11, 17, 13, 57, 0, 258870, tzinfo=tzlocal()), '1.2.45', '15109263663135171') 2017-11-17 13:57:37,477 | INFO | node.py (1551) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 1 2017-11-17 13:57:37,477 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 1, caught up 0 in total 2017-11-17 13:57:37,478 | INFO | node.py (1588) | allLedgersCaughtUp | CATCH-UP: Node3 caught up till (2, 1) 2017-11-17 13:57:37,478 | INFO | node.py (1598) | allLedgersCaughtUp | CATCH-UP: Node3 does not need any more catchups 2017-11-17 13:57:37,479 | DISPLAY | primary_selector.py ( 327) | _start_selection | PRIMARY SELECTION: Node3:0 selected primary Node4:0 for instance 0 (view 3) 2017-11-17 13:57:37,479 | INFO | node.py ( 532) | start_participating | Node3 started participating 2017-11-17 13:57:37,480 | DISPLAY | primary_selector.py ( 351) | _start_selection | VIEW CHANGE: Node3:0 declares view change 3 as completed for instance 0, new primary is Node4:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 4, '3aErFDAegiipoxjKyq8YBzyNpDPHaoNhZ8BqVu4e3uCg')] 2017-11-17 13:57:37,484 | DISPLAY | primary_selector.py ( 327) | _start_selection | PRIMARY SELECTION: Node3:1 selected primary Node1:1 for instance 1 (view 3) 2017-11-17 13:57:37,484 | INFO | replica.py (2025) | enqueue_prepare | Node3:1 queueing prepare due to unavailability of PRE-PREPARE. Prepare PREPARE{'ppTime': 1510927048, 'viewNo': 3, 'stateRootHash': None, 'txnRootHash': None, 'ppSeqNo': 1, 'digest': '35f37aa655aec5ad6e4da1c6df55a1f1e5607382ed0a3f3ab14e6ca9696c66df', 'instId': 1} from Node2:1 2017-11-17 13:57:37,485 | DISPLAY | primary_selector.py ( 351) | _start_selection | VIEW CHANGE: Node3:1 declares view change 3 as completed for instance 1, new primary is Node1:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 4, '3aErFDAegiipoxjKyq8YBzyNpDPHaoNhZ8BqVu4e3uCg')] 2017-11-17 13:57:37,485 | INFO | upgrader.py ( 149) | should_notify_about_upgrade_result | Node's 'Node3' last upgrade txn is OrderedDict([('data', OrderedDict([('action', 'in_progress'), ('version', '1.2.45')])), ('identifier', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya'), ('reqId', 1510927020097708), ('signature', '36bUMwhQhaVvVo1xHJqoG8WKH1uSiq2Sj5zN4Sr5BeNS1uxkGMiS1xcUJHkkE9QJuqYk3r4sq58kZLg7HVLvbpkp'), ('txnTime', 1510927020), ('type', '110'), ('seqNo', 4)]) 2017-11-17 13:57:37,487 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 2, caught up 0 in total 2017-11-17 13:57:37,487 | INFO | node.py (1598) | allLedgersCaughtUp | CATCH-UP: Node3 does not need any more catchups 2017-11-17 13:57:37,488 | INFO | node.py (1685) | no_more_catchups_needed | Node3 starting to participate since catchup is done, primaries are selected but mode was not set to participating 2017-11-17 13:57:37,488 | INFO | node.py ( 532) | start_participating | Node3 started participating 2017-11-17 13:57:37,520 | INFO | propagator.py ( 179) | propagate | Node3 propagating request ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926752873422) from client Node2 2017-11-17 13:57:37,520 | INFO | propagator.py ( 179) | propagate | Node3 propagating request ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926748095846) from client Node1 2017-11-17 13:57:37,521 | INFO | propagator.py ( 179) | propagate | Node3 propagating request ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927057486576) from client Node3 2017-11-17 13:57:37,540 | INFO | replica.py (1619) | order_3pc_key | Node3:1 ordered batch request, view no 3, ppSeqNo 1, ledger 2, state root None, txn root None, requests ordered [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926748095846), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926752873422), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927048003928), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927047995990)], discarded [] 2017-11-17 13:57:37,542 | INFO | replica.py (1619) | order_3pc_key | Node3:1 ordered batch request, view no 3, ppSeqNo 2, ledger 2, state root None, txn root None, requests ordered [('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927057486576)], discarded [] 2017-11-17 13:57:42,599 | INFO | monitor.py ( 343) | isMasterThroughputTooLow | MONITORING: Node3 master throughput ratio 0.0 is lower than Delta 0.4. 2017-11-17 13:57:42,599 | INFO | node.py (2128) | sendInstanceChange | VIEW CHANGE: Node3 sending an instance change with view_no 4 since Primary of master protocol instance degraded the performance 2017-11-17 13:57:42,600 | INFO | node.py (2131) | sendInstanceChange | MONITORING: Node3 metrics for monitor: Node3 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [1381900.797609953, 1381900.797965252] ordered request counts: {0: 0, 1: 3} ordered request durations: {0: 0, 1: 0.06145448633469641} master request latencies: {} client avg request latencies: [{}, {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': (1, 0.020191582152619958), 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': (1, 0.02162105403840542), 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': (1, 0.019641850143671036)}] throughput: {0: 0, 1: 48.816615009378715} master throughput: 0 total requests: 0 avg backup throughput: 48.816615009378715 master throughput ratio: 0.0 2017-11-17 13:57:42,600 | INFO | node.py (2031) | do_view_change_if_possible | VIEW CHANGE: Node3 initiating a view change to 4 from 3 2017-11-17 13:57:42,601 | INFO | node.py (2241) | startViewChange | VIEW CHANGE: Node3 changed to view 4, will start catchup now 2017-11-17 13:57:42,602 | INFO | node.py (2075) | checkPerformance | Node3 sent view change since performance degraded of master instance 2017-11-17 13:57:42,631 | INFO | node.py (1551) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 0 2017-11-17 13:57:42,631 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 0, caught up 0 in total 2017-11-17 13:57:42,649 | INFO | node.py (1551) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 2 2017-11-17 13:57:42,650 | INFO | upgrader.py ( 212) | processLedger | DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya found upgrade START txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)]) 2017-11-17 13:57:42,650 | INFO | upgrader.py ( 290) | handleUpgradeTxn | Node 'Node3' handles upgrade txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)]) 2017-11-17 13:57:42,650 | INFO | upgrader.py ( 310) | handleUpgradeTxn | Node 'Node3' has already performed an upgrade with upgrade_id 15109263663135171. Last recorded event is ('succeeded', datetime.datetime(2017, 11, 17, 13, 57, 0, 258870, tzinfo=tzlocal()), '1.2.45', '15109263663135171') 2017-11-17 13:57:42,650 | INFO | upgrader.py ( 149) | should_notify_about_upgrade_result | Node's 'Node3' last upgrade txn is OrderedDict([('data', OrderedDict([('action', 'in_progress'), ('version', '1.2.45')])), ('identifier', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya'), ('reqId', 1510927020097708), ('signature', '36bUMwhQhaVvVo1xHJqoG8WKH1uSiq2Sj5zN4Sr5BeNS1uxkGMiS1xcUJHkkE9QJuqYk3r4sq58kZLg7HVLvbpkp'), ('txnTime', 1510927020), ('type', '110'), ('seqNo', 4)]) 2017-11-17 13:57:42,651 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 2, caught up 0 in total 2017-11-17 13:57:42,667 | INFO | propagator.py ( 179) | propagate | Node3 propagating request ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927062654152) from client Node1 2017-11-17 13:57:42,668 | INFO | propagator.py ( 179) | propagate | Node3 propagating request ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927062651136) from client Node3 2017-11-17 13:57:42,676 | INFO | node.py (1551) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 1 2017-11-17 13:57:42,676 | INFO | ledger_manager.py ( 865) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 1, caught up 0 in total 2017-11-17 13:57:42,676 | INFO | node.py (1598) | allLedgersCaughtUp | CATCH-UP: Node3 does not need any more catchups 2017-11-17 13:57:42,678 | INFO | propagator.py ( 179) | propagate | Node3 propagating request ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927062662523) from client Node2 2017-11-17 13:57:42,697 | DISPLAY | primary_selector.py ( 327) | _start_selection | PRIMARY SELECTION: Node3:0 selected primary Node1:0 for instance 0 (view 4) 2017-11-17 13:57:42,697 | INFO | node.py ( 532) | start_participating | Node3 started participating 2017-11-17 13:57:42,698 | DISPLAY | primary_selector.py ( 351) | _start_selection | VIEW CHANGE: Node3:0 declares view change 4 as completed for instance 0, new primary is Node1:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 4, '3aErFDAegiipoxjKyq8YBzyNpDPHaoNhZ8BqVu4e3uCg')] 2017-11-17 13:57:42,698 | DISPLAY | primary_selector.py ( 327) | _start_selection | PRIMARY SELECTION: Node3:1 selected primary Node2:1 for instance 1 (view 4) 2017-11-17 13:57:42,699 | DISPLAY | primary_selector.py ( 351) | _start_selection | VIEW CHANGE: Node3:1 declares view change 4 as completed for instance 1, new primary is Node2:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 4, '3aErFDAegiipoxjKyq8YBzyNpDPHaoNhZ8BqVu4e3uCg')] 2017-11-17 13:57:42,726 | INFO | replica.py (1619) | order_3pc_key | Node3:0 ordered batch request, view no 4, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 8PHonJAkhGQDvCHhZ336fu17U8UV3wYpJGCjV7gdLdBd, requests ordered [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926748095846), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926752873422), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927048003928), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927047995990), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927057486576), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927062651136), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927062654152), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927062662523)], discarded [] 2017-11-17 13:57:42,727 | INFO | replica.py (1619) | order_3pc_key | Node3:1 ordered batch request, view no 4, ppSeqNo 1, ledger 2, state root None, txn root None, requests ordered [('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927062651136), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927062654152), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927062662523)], discarded [] 2017-11-17 13:57:42,730 | INFO | node.py (2374) | executeBatch | Node3 committed batch request, view no 4, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 8PHonJAkhGQDvCHhZ336fu17U8UV3wYpJGCjV7gdLdBd, requests: [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926748095846), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926752873422), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927048003928), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927047995990), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927057486576), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927062651136), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927062654152), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927062662523)] 2017-11-17 13:57:44,766 | INFO | keep_in_touch.py ( 98) | _connsChanged | CONNECTION: Node3 now connected to Node4