2017-11-17 13:43:44,268 | INFO     | node.py              (2420) | initStateFromLedger | Node3 found state to be empty, recreating from ledger
2017-11-17 13:43:44,351 | INFO     | node.py              (2420) | initStateFromLedger | Node3 found state to be empty, recreating from ledger
2017-11-17 13:43:44,394 | INFO     | plugin_loader.py     ( 117) | _load | plugin FirebaseStatsConsumer successfully loaded from module plugin_firebase_stats_consumer
2017-11-17 13:43:44,395 | DISPLAY  | replicas.py          (  36) | grow | Node3 added replica Node3:0 to instance 0 (master)
2017-11-17 13:43:44,396 | DISPLAY  | replicas.py          (  36) | grow | Node3 added replica Node3:1 to instance 1 (backup)
2017-11-17 13:43:44,487 | INFO     | node.py              (2420) | initStateFromLedger | Node3 found state to be empty, recreating from ledger
2017-11-17 13:43:44,488 | INFO     | stacks.py            (  84) | start | CONNECTION: Node3 listening for other nodes at 0.0.0.0:9705
2017-11-17 13:43:44,488 | INFO     | node.py              ( 621) | start | Node3 first time running...
2017-11-17 13:43:44,490 | INFO     | zstack.py            ( 590) | connect | CONNECTION: Node3 looking for Node4 at 10.0.0.5:9707
2017-11-17 13:43:44,491 | INFO     | zstack.py            ( 590) | connect | CONNECTION: Node3 looking for Node2 at 10.0.0.3:9703
2017-11-17 13:43:44,491 | INFO     | zstack.py            ( 590) | connect | CONNECTION: Node3 looking for Node1 at 10.0.0.2:9701
2017-11-17 13:43:44,559 | INFO     | keep_in_touch.py     (  98) | _connsChanged | CONNECTION: Node3 now connected to Node2
2017-11-17 13:43:44,596 | INFO     | keep_in_touch.py     (  98) | _connsChanged | CONNECTION: Node3 now connected to Node1
2017-11-17 13:43:44,623 | INFO     | node.py              (1489) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 0
2017-11-17 13:43:44,624 | INFO     | ledger_manager.py    ( 848) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 0, caught up 0 in total
2017-11-17 13:43:44,651 | INFO     | node.py              (1489) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 2
2017-11-17 13:43:44,652 | INFO     | upgrader.py          ( 151) | should_notify_about_upgrade_result | Node's 'Node3' last upgrade txn is None
2017-11-17 13:43:44,652 | INFO     | ledger_manager.py    ( 848) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 2, caught up 0 in total
2017-11-17 13:43:44,667 | INFO     | node.py              (1489) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 1
2017-11-17 13:43:44,667 | INFO     | ledger_manager.py    ( 848) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 1, caught up 0 in total
2017-11-17 13:43:44,668 | INFO     | node.py              (1537) | allLedgersCaughtUp | CATCH-UP: Node3 does not need any more catchups
2017-11-17 13:43:44,671 | DISPLAY  | primary_selector.py  ( 291) | _startSelection | PRIMARY SELECTION: Node3:0 selected primary Node1:0 for instance 0 (view 0)
2017-11-17 13:43:44,671 | INFO     | node.py              ( 510) | start_participating | Node3 started participating
2017-11-17 13:43:44,672 | DISPLAY  | primary_selector.py  ( 315) | _startSelection | VIEW CHANGE: Node3:0 declares view change 0 as completed for instance 0, new primary is Node1:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')]
2017-11-17 13:43:44,672 | DISPLAY  | primary_selector.py  ( 291) | _startSelection | PRIMARY SELECTION: Node3:1 selected primary Node2:1 for instance 1 (view 0)
2017-11-17 13:43:44,672 | DISPLAY  | primary_selector.py  ( 315) | _startSelection | VIEW CHANGE: Node3:1 declares view change 0 as completed for instance 1, new primary is Node2:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')]
2017-11-17 13:43:45,136 | INFO     | keep_in_touch.py     (  98) | _connsChanged | CONNECTION: Node3 now connected to Node4
2017-11-17 13:45:38,702 | INFO     | message_processor.py (  29) | discard | Node3 discarding message ({'signature': 'KNpuf1rjcogYVHpD982n5b3tY527KEDAdKDbuVfSDTY95x1jUkULk3buoEzHuGj6KLjfHSNN6wcdkF9dfztPSHe', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'operation': {'reinstall': False, 'justification': None, 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'name': 'upgrade_to_the_latest_full', 'action': 'start', 'timeout': 10, 'type': '109', 'version': '1.2.45', 'schedule': {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-11-17T13:45:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-11-17T14:00:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-11-17T13:50:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-11-17T13:55:00.258870+00:00'}, 'force': False}, 'reqId': 1510926338687540}, b'=cQu#Y-Ir.lyHj/JuuLEhk!lc&(9BYFL7fT8TyD}') because InvalidClientRequest("{'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-11-17T13:45:00.258870+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-11-17T14:00:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-11-17T13:50:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-11-17T13:55:00.258870+00:00'} not a valid schedule since 2017-11-17 13:45:00.258870+00:00 is less than current time",)
2017-11-17 13:46:06,340 | INFO     | propagator.py        ( 148) | propagate | Node3 propagating request ('V4SGRU86Z58d6TV7PBUe6f', 1510926366313517) from client b'=cQu#Y-Ir.lyHj/JuuLEhk!lc&(9BYFL7fT8TyD}'
2017-11-17 13:46:06,396 | INFO     | replica.py           (1483) | order_3pc_key | Node3:1 ordered batch request, view no 0, ppSeqNo 1, ledger 2, state root None, txn root None, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1510926366313517)], discarded []
2017-11-17 13:46:06,427 | INFO     | replica.py           (1483) | order_3pc_key | Node3:0 ordered batch request, view no 0, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root XjH6SHaWYaxmk1gyUBigBTfZctf6HT46vK3WgUhceQG, requests ordered [('V4SGRU86Z58d6TV7PBUe6f', 1510926366313517)], discarded []
2017-11-17 13:46:06,428 | INFO     | upgrader.py          ( 292) | handleUpgradeTxn | Node 'Node3' handles upgrade txn {'justification': None, 'signature': 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj', 'timeout': 10, 'reinstall': False, 'version': '1.2.45', 'identifier': 'V4SGRU86Z58d6TV7PBUe6f', 'schedule': {'4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA': '2017-11-17T14:02:00.258870+00:00', 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': '2017-11-17T13:47:00.258870+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2017-11-17T13:52:00.258870+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': '2017-11-17T13:57:00.258870+00:00'}, 'rootHash': 'XjH6SHaWYaxmk1gyUBigBTfZctf6HT46vK3WgUhceQG', 'txnTime': 1510926366, 'reqId': 1510926366313517, 'auditPath': [], 'name': 'upgrade_to_the_latest_full', 'seqNo': 1, 'action': 'start', 'sha256': 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f', 'type': '109', 'force': False}
2017-11-17 13:46:06,428 | INFO     | upgrader.py          ( 321) | handleUpgradeTxn | Node 'Node3' schedules upgrade to 1.2.45
2017-11-17 13:46:06,428 | INFO     | upgrader.py          ( 359) | _scheduleUpgrade | DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya's upgrader processing upgrade for version 1.2.45
2017-11-17 13:46:06,429 | INFO     | node.py              (2290) | executeBatch | Node3 committed batch request, view no 0, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root XjH6SHaWYaxmk1gyUBigBTfZctf6HT46vK3WgUhceQG, requests: [('V4SGRU86Z58d6TV7PBUe6f', 1510926366313517)]
2017-11-17 13:46:59,447 | INFO     | propagator.py        ( 148) | propagate | Node3 propagating request ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926419422000) from client Node1
2017-11-17 13:46:59,468 | INFO     | replica.py           (1483) | order_3pc_key | Node3:0 ordered batch request, view no 0, ppSeqNo 2, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 2PDWCTTViFKEEVQvxGsg6s7VZm4rCmmBu8G1VLJhEdom, requests ordered [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926419422000)], discarded []
2017-11-17 13:46:59,469 | INFO     | replica.py           (1483) | order_3pc_key | Node3:1 ordered batch request, view no 0, ppSeqNo 2, ledger 2, state root None, txn root None, requests ordered [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926419422000)], discarded []
2017-11-17 13:46:59,470 | INFO     | node.py              (2290) | executeBatch | Node3 committed batch request, view no 0, ppSeqNo 2, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 2PDWCTTViFKEEVQvxGsg6s7VZm4rCmmBu8G1VLJhEdom, requests: [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926419422000)]
2017-11-17 13:47:30,414 | INFO     | keep_in_touch.py     (  93) | _connsChanged | CONNECTION: Node3 disconnected from Node1
2017-11-17 13:47:30,414 | INFO     | node.py              ( 823) | onConnsChanged | Node3 lost connection to primary of master
2017-11-17 13:47:32,419 | INFO     | node.py              (2048) | sendInstanceChange | VIEW CHANGE: Node3 sending an instance change with view_no 1 since Primary of master protocol instance disconnected
2017-11-17 13:47:32,419 | INFO     | node.py              (2051) | sendInstanceChange | MONITORING: Node3 metrics for monitor: Node3 Monitor metrics:: None
            Delta: 0.4
            Lambda: 60
            Omega: 5
            instances started: [1381072.604918195, 1381072.605166084]
            ordered request counts: {0: 2, 1: 2}
            ordered request durations: {0: 0.08513712720014155, 1: 0.05257508018985391}
            master request latencies: {}
            client avg request latencies: [{'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': (1, 0.021839564200490713), 'V4SGRU86Z58d6TV7PBUe6f': (1, 0.06329756299965084)}, {'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': (1, 0.021925772074609995), 'V4SGRU86Z58d6TV7PBUe6f': (1, 0.030649308115243912)}]
            throughput: {0: 23.49151381744855, 1: 38.04083593934234}
            master throughput: 23.49151381744855
            total requests: 2
            avg backup throughput: 38.04083593934234
            master throughput ratio: 0.6175341113667093
2017-11-17 13:47:32,419 | INFO     | node.py              (2108) | propose_view_change | Node3 sent view change since was disconnected from primary for too long
2017-11-17 13:47:32,432 | INFO     | node.py              (1954) | do_view_change_if_possible | VIEW CHANGE: Node3 initiating a view change to 1 from 0
2017-11-17 13:47:32,435 | INFO     | node.py              (2159) | startViewChange | VIEW CHANGE: Node3 changed to view 1, will start catchup now
2017-11-17 13:47:32,435 | INFO     | message_processor.py (  29) | discard | Node3 discarding message INSTANCE_CHANGE{'viewNo': 1, 'reason': 26} because Received instance change request with view no 1 which is not more than its view no 1
2017-11-17 13:47:32,465 | INFO     | node.py              (1489) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 0
2017-11-17 13:47:32,466 | INFO     | ledger_manager.py    ( 848) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 0, caught up 0 in total
2017-11-17 13:47:32,494 | INFO     | node.py              (1489) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 2
2017-11-17 13:47:32,494 | INFO     | upgrader.py          ( 214) | processLedger | DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya found upgrade START txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)])
2017-11-17 13:47:32,495 | INFO     | upgrader.py          ( 292) | handleUpgradeTxn | Node 'Node3' handles upgrade txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)])
2017-11-17 13:47:32,495 | INFO     | upgrader.py          ( 321) | handleUpgradeTxn | Node 'Node3' schedules upgrade to 1.2.45
2017-11-17 13:47:32,495 | INFO     | upgrader.py          ( 326) | handleUpgradeTxn | Node 'Node3' cancels previous upgrade and schedules a new one to 1.2.45
2017-11-17 13:47:32,495 | INFO     | upgrader.py          ( 390) | _cancelScheduledUpgrade | Cancelling upgrade of node 'Node3' to version 1.2.45 due to some reason
2017-11-17 13:47:32,495 | INFO     | upgrader.py          ( 359) | _scheduleUpgrade | DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya's upgrader processing upgrade for version 1.2.45
2017-11-17 13:47:32,496 | INFO     | upgrader.py          ( 151) | should_notify_about_upgrade_result | Node's 'Node3' last upgrade txn is None
2017-11-17 13:47:32,496 | INFO     | ledger_manager.py    ( 848) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 2, caught up 0 in total
2017-11-17 13:47:32,514 | INFO     | node.py              (1489) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 1
2017-11-17 13:47:32,514 | INFO     | ledger_manager.py    ( 848) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 1, caught up 0 in total
2017-11-17 13:47:32,515 | INFO     | node.py              (1537) | allLedgersCaughtUp | CATCH-UP: Node3 does not need any more catchups
2017-11-17 13:47:32,516 | DISPLAY  | primary_selector.py  ( 291) | _startSelection | PRIMARY SELECTION: Node3:0 selected primary Node2:0 for instance 0 (view 1)
2017-11-17 13:47:32,517 | INFO     | node.py              ( 510) | start_participating | Node3 started participating
2017-11-17 13:47:32,517 | DISPLAY  | primary_selector.py  ( 315) | _startSelection | VIEW CHANGE: Node3:0 declares view change 1 as completed for instance 0, new primary is Node2:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 2, '2PDWCTTViFKEEVQvxGsg6s7VZm4rCmmBu8G1VLJhEdom')]
2017-11-17 13:47:32,517 | DISPLAY  | primary_selector.py  ( 291) | _startSelection | PRIMARY SELECTION: Node3:1 selected primary Node3:1 for instance 1 (view 1)
2017-11-17 13:47:32,518 | DISPLAY  | primary_selector.py  ( 315) | _startSelection | VIEW CHANGE: Node3:1 declares view change 1 as completed for instance 1, new primary is Node3:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 2, '2PDWCTTViFKEEVQvxGsg6s7VZm4rCmmBu8G1VLJhEdom')]
2017-11-17 13:47:44,630 | INFO     | keep_in_touch.py     (  98) | _connsChanged | CONNECTION: Node3 now connected to Node1
2017-11-17 13:47:44,658 | INFO     | message_processor.py (  29) | discard | Node3 discarding message {'signature': '485QNuqefeLN4d7RrAyUYtK7egvRBsMhzADLYScsLmJivg8FKytaWwbPnj82pxd6FrSbch5znp8M5bn1gsRwraFn', 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'operation': {'signature': '5Rjt3r36mqM669ZoEcccqa6zJU5UEdxDLrfZD3UAJrWeN3hbvPCsVnzqmUootfcgMsfm43MzMkq1vXcXyveN6VjN', 'type': '110', 'data': {'action': 'complete', 'version': '1.2.45'}}, 'protocolVersion': 1, 'reqId': 1510926464650411} because 'SafeRequest' object has no attribute 'schema_is_strict'
2017-11-17 13:51:59,520 | INFO     | propagator.py        ( 148) | propagate | Node3 propagating request ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926719502400) from client Node2
2017-11-17 13:51:59,542 | INFO     | replica.py           (1483) | order_3pc_key | Node3:1 ordered batch request, view no 1, ppSeqNo 1, ledger 2, state root None, txn root None, requests ordered [('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926719502400)], discarded []
2017-11-17 13:51:59,545 | INFO     | replica.py           (1483) | order_3pc_key | Node3:0 ordered batch request, view no 1, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root BvBE3KNL7XqXP3gkNvaS9mSi1veDkzTj32rY5mpWBQbk, requests ordered [('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926719502400)], discarded []
2017-11-17 13:51:59,546 | INFO     | node.py              (2290) | executeBatch | Node3 committed batch request, view no 1, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root BvBE3KNL7XqXP3gkNvaS9mSi1veDkzTj32rY5mpWBQbk, requests: [('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926719502400)]
2017-11-17 13:52:26,037 | INFO     | keep_in_touch.py     (  93) | _connsChanged | CONNECTION: Node3 disconnected from Node2
2017-11-17 13:52:26,037 | INFO     | node.py              ( 823) | onConnsChanged | Node3 lost connection to primary of master
2017-11-17 13:52:28,048 | INFO     | node.py              (2048) | sendInstanceChange | VIEW CHANGE: Node3 sending an instance change with view_no 2 since Primary of master protocol instance disconnected
2017-11-17 13:52:28,049 | INFO     | node.py              (2051) | sendInstanceChange | MONITORING: Node3 metrics for monitor: Node3 Monitor metrics:: None
            Delta: 0.4
            Lambda: 60
            Omega: 5
            instances started: [1381072.604918195, 1381072.605166084]
            ordered request counts: {0: 1, 1: 1}
            ordered request durations: {0: 0.02551817218773067, 1: 0.02203888911753893}
            master request latencies: {}
            client avg request latencies: [{'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': (1, 0.02551817218773067)}, {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': (1, 0.02203888911753893)}]
            throughput: {0: 39.187759712696334, 1: 45.374337820148234}
            master throughput: 39.187759712696334
            total requests: 3
            avg backup throughput: 45.374337820148234
            master throughput ratio: 0.8636546910728737
2017-11-17 13:52:28,049 | INFO     | node.py              (2108) | propose_view_change | Node3 sent view change since was disconnected from primary for too long
2017-11-17 13:52:28,051 | INFO     | node.py              (1954) | do_view_change_if_possible | VIEW CHANGE: Node3 initiating a view change to 2 from 1
2017-11-17 13:52:28,052 | INFO     | node.py              (2159) | startViewChange | VIEW CHANGE: Node3 changed to view 2, will start catchup now
2017-11-17 13:52:28,070 | INFO     | node.py              (1489) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 0
2017-11-17 13:52:28,070 | INFO     | ledger_manager.py    ( 848) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 0, caught up 0 in total
2017-11-17 13:52:28,086 | INFO     | node.py              (1489) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 2
2017-11-17 13:52:28,086 | INFO     | upgrader.py          ( 214) | processLedger | DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya found upgrade START txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)])
2017-11-17 13:52:28,086 | INFO     | upgrader.py          ( 292) | handleUpgradeTxn | Node 'Node3' handles upgrade txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)])
2017-11-17 13:52:28,087 | INFO     | upgrader.py          ( 321) | handleUpgradeTxn | Node 'Node3' schedules upgrade to 1.2.45
2017-11-17 13:52:28,087 | INFO     | upgrader.py          ( 326) | handleUpgradeTxn | Node 'Node3' cancels previous upgrade and schedules a new one to 1.2.45
2017-11-17 13:52:28,087 | INFO     | upgrader.py          ( 390) | _cancelScheduledUpgrade | Cancelling upgrade of node 'Node3' to version 1.2.45 due to some reason
2017-11-17 13:52:28,087 | INFO     | upgrader.py          ( 359) | _scheduleUpgrade | DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya's upgrader processing upgrade for version 1.2.45
2017-11-17 13:52:28,088 | INFO     | upgrader.py          ( 151) | should_notify_about_upgrade_result | Node's 'Node3' last upgrade txn is None
2017-11-17 13:52:28,088 | INFO     | ledger_manager.py    ( 848) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 2, caught up 0 in total
2017-11-17 13:52:28,102 | INFO     | message_processor.py (  29) | discard | Node3 discarding message {'signature': '5uX7Q8zCtLBdMp77gNeszGewZ4YqZo77WRK9dCbjso3EYqUL1F2GQSd8qMbjPiK49JT7k1HJATotnTN5paEvBeYM', 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'operation': {'signature': '5Rjt3r36mqM669ZoEcccqa6zJU5UEdxDLrfZD3UAJrWeN3hbvPCsVnzqmUootfcgMsfm43MzMkq1vXcXyveN6VjN', 'type': '110', 'data': {'action': 'complete', 'version': '1.2.45'}}, 'protocolVersion': 1, 'reqId': 1510926748095846} because 'SafeRequest' object has no attribute 'schema_is_strict'
2017-11-17 13:52:28,103 | INFO     | node.py              (1489) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 1
2017-11-17 13:52:28,104 | INFO     | ledger_manager.py    ( 848) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 1, caught up 0 in total
2017-11-17 13:52:28,104 | INFO     | node.py              (1537) | allLedgersCaughtUp | CATCH-UP: Node3 does not need any more catchups
2017-11-17 13:52:28,119 | DISPLAY  | primary_selector.py  ( 291) | _startSelection | PRIMARY SELECTION: Node3:0 selected primary Node3:0 for instance 0 (view 2)
2017-11-17 13:52:28,119 | INFO     | node.py              ( 510) | start_participating | Node3 started participating
2017-11-17 13:52:28,119 | DISPLAY  | primary_selector.py  ( 315) | _startSelection | VIEW CHANGE: Node3:0 declares view change 2 as completed for instance 0, new primary is Node3:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 3, 'BvBE3KNL7XqXP3gkNvaS9mSi1veDkzTj32rY5mpWBQbk')]
2017-11-17 13:52:28,120 | DISPLAY  | primary_selector.py  ( 291) | _startSelection | PRIMARY SELECTION: Node3:1 selected primary Node4:1 for instance 1 (view 2)
2017-11-17 13:52:28,120 | DISPLAY  | primary_selector.py  ( 315) | _startSelection | VIEW CHANGE: Node3:1 declares view change 2 as completed for instance 1, new primary is Node4:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 3, 'BvBE3KNL7XqXP3gkNvaS9mSi1veDkzTj32rY5mpWBQbk')]
2017-11-17 13:52:32,827 | INFO     | keep_in_touch.py     (  98) | _connsChanged | CONNECTION: Node3 now connected to Node2
2017-11-17 13:52:32,892 | INFO     | message_processor.py (  29) | discard | Node3 discarding message {'signature': '2dvC3j8jPcADXgKBygm4FzmvmeRToTDNs2Tu4LFHkvRLpEm1EbyAtFXNwW1VNDFnYMcNwNWFtK67A1cyy5vZhgEs', 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'operation': {'signature': 'aXn8Rhnak15iQAyvXCt5Vz4Qm4XXhHBxg2CnNidCjAGWCbrxBfQEjhv98Wm67aLMuamt3KrpbTELweqMzyxswqP', 'type': '110', 'data': {'action': 'complete', 'version': '1.2.45'}}, 'protocolVersion': 1, 'reqId': 1510926752873422} because 'SafeRequest' object has no attribute 'schema_is_strict'
2017-11-17 13:52:32,894 | INFO     | message_processor.py (  29) | discard | Node3 discarding message {'senderClient': 'Node2', 'request': {'signature': '2dvC3j8jPcADXgKBygm4FzmvmeRToTDNs2Tu4LFHkvRLpEm1EbyAtFXNwW1VNDFnYMcNwNWFtK67A1cyy5vZhgEs', 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'operation': {'signature': 'aXn8Rhnak15iQAyvXCt5Vz4Qm4XXhHBxg2CnNidCjAGWCbrxBfQEjhv98Wm67aLMuamt3KrpbTELweqMzyxswqP', 'type': '110', 'data': {'action': 'complete', 'version': '1.2.45'}}, 'protocolVersion': 1, 'reqId': 1510926752873422}, 'op': 'PROPAGATE'} because InvalidNodeMsg('validation error [ClientMessageValidator]: unknown field - protocolVersion=1',)
2017-11-17 13:52:32,936 | INFO     | message_processor.py (  29) | discard | Node3 discarding message {'senderClient': 'Node1', 'request': {'signature': '5uX7Q8zCtLBdMp77gNeszGewZ4YqZo77WRK9dCbjso3EYqUL1F2GQSd8qMbjPiK49JT7k1HJATotnTN5paEvBeYM', 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'operation': {'signature': '5Rjt3r36mqM669ZoEcccqa6zJU5UEdxDLrfZD3UAJrWeN3hbvPCsVnzqmUootfcgMsfm43MzMkq1vXcXyveN6VjN', 'type': '110', 'data': {'action': 'complete', 'version': '1.2.45'}}, 'protocolVersion': 1, 'reqId': 1510926748095846}, 'op': 'PROPAGATE'} because InvalidNodeMsg('validation error [ClientMessageValidator]: unknown field - protocolVersion=1',)
2017-11-17 13:52:32,936 | INFO     | message_processor.py (  29) | discard | Node3 discarding message {'senderClient': 'Node2', 'request': {'signature': '2dvC3j8jPcADXgKBygm4FzmvmeRToTDNs2Tu4LFHkvRLpEm1EbyAtFXNwW1VNDFnYMcNwNWFtK67A1cyy5vZhgEs', 'identifier': '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 'operation': {'signature': 'aXn8Rhnak15iQAyvXCt5Vz4Qm4XXhHBxg2CnNidCjAGWCbrxBfQEjhv98Wm67aLMuamt3KrpbTELweqMzyxswqP', 'type': '110', 'data': {'action': 'complete', 'version': '1.2.45'}}, 'protocolVersion': 1, 'reqId': 1510926752873422}, 'op': 'PROPAGATE'} because InvalidNodeMsg('validation error [ClientMessageValidator]: unknown field - protocolVersion=1',)
2017-11-17 13:52:32,948 | INFO     | message_processor.py (  29) | discard | Node3 discarding message {'senderClient': 'Node1', 'request': {'signature': '5uX7Q8zCtLBdMp77gNeszGewZ4YqZo77WRK9dCbjso3EYqUL1F2GQSd8qMbjPiK49JT7k1HJATotnTN5paEvBeYM', 'identifier': 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 'operation': {'signature': '5Rjt3r36mqM669ZoEcccqa6zJU5UEdxDLrfZD3UAJrWeN3hbvPCsVnzqmUootfcgMsfm43MzMkq1vXcXyveN6VjN', 'type': '110', 'data': {'action': 'complete', 'version': '1.2.45'}}, 'protocolVersion': 1, 'reqId': 1510926748095846}, 'op': 'PROPAGATE'} because InvalidNodeMsg('validation error [ClientMessageValidator]: unknown field - protocolVersion=1',)
2017-11-17 13:57:00,096 | INFO     | upgrader.py          ( 421) | _callUpgradeAgent | DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya's upgrader calling agent for upgrade
2017-11-17 13:57:00,097 | INFO     | node.py              ( 254) | notify_upgrade_start | Node3 is about to be upgraded, sending NODE_UPGRADE
2017-11-17 13:57:00,099 | INFO     | upgrader.py          ( 433) | _sendUpdateRequest | Sending message to control tool: {"version": "1.2.45"}
2017-11-17 13:57:00,100 | INFO     | upgrader.py          ( 452) | _sendUpdateRequest | Waiting 10 minutes for upgrade to be performed
2017-11-17 13:57:00,117 | INFO     | propagator.py        ( 148) | propagate | Node3 propagating request ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927020097708) from client Node3
2017-11-17 13:57:00,145 | INFO     | replica.py           (1483) | order_3pc_key | Node3:1 ordered batch request, view no 2, ppSeqNo 1, ledger 2, state root None, txn root None, requests ordered [('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927020097708)], discarded []
2017-11-17 13:57:00,160 | INFO     | replica.py           (1483) | order_3pc_key | Node3:0 ordered batch request, view no 2, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 3aErFDAegiipoxjKyq8YBzyNpDPHaoNhZ8BqVu4e3uCg, requests ordered [('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927020097708)], discarded []
2017-11-17 13:57:00,161 | INFO     | node.py              (2290) | executeBatch | Node3 committed batch request, view no 2, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 3aErFDAegiipoxjKyq8YBzyNpDPHaoNhZ8BqVu4e3uCg, requests: [('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927020097708)]
2017-11-17 13:57:32,525 | INFO     | node.py              (2503) | initStateFromLedger | Node3 found state to be empty, recreating from ledger
2017-11-17 13:57:32,527 | INFO     | pool_manager.py      ( 409) | _order_node | Node3 node Node1 ordered, NYM Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv
2017-11-17 13:57:32,527 | INFO     | pool_manager.py      ( 409) | _order_node | Node3 node Node2 ordered, NYM 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb
2017-11-17 13:57:32,527 | INFO     | pool_manager.py      ( 409) | _order_node | Node3 node Node3 ordered, NYM DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya
2017-11-17 13:57:32,527 | INFO     | pool_manager.py      ( 409) | _order_node | Node3 node Node4 ordered, NYM 4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA
2017-11-17 13:57:32,540 | WARNING  | node.py              ( 623) | _create_bls_bft | BLS: Transactions will not be BLS signed by this Node, since BLS keys were not found. Please make sure that a script to init BLS keys was called (init_bls_keys), and NODE txn was sent with BLS public keys.
2017-11-17 13:57:32,543 | INFO     | node.py              (2503) | initStateFromLedger | Node3 found state to be empty, recreating from ledger
2017-11-17 13:57:32,554 | INFO     | node.py              ( 479) | setPoolParams | Node3 updated its pool parameters: f 1, totalNodes 4, allNodeNames {'Node3', 'Node2', 'Node4', 'Node1'}, requiredNumberOfInstances 2, minimumNodes 3, quorums {'election': Quorum(3), 'timestamp': Quorum(2), 'f': 1, 'view_change': Quorum(3), 'bls_signatures': Quorum(3), 'propagate_primary': Quorum(2), 'checkpoint': Quorum(2), 'same_consistency_proof': Quorum(2), 'view_change_done': Quorum(3), 'commit': Quorum(3), 'prepare': Quorum(2), 'reply': Quorum(2), 'consistency_proof': Quorum(2), 'propagate': Quorum(2), 'ledger_status': Quorum(2)}
2017-11-17 13:57:32,587 | INFO     | plugin_loader.py     ( 117) | _load | plugin FirebaseStatsConsumer successfully loaded from module plugin_firebase_stats_consumer
2017-11-17 13:57:32,588 | DISPLAY  | replicas.py          (  40) | grow | Node3 added replica Node3:0 to instance 0 (master)
2017-11-17 13:57:32,588 | DISPLAY  | replicas.py          (  40) | grow | Node3 added replica Node3:1 to instance 1 (backup)
2017-11-17 13:57:32,639 | INFO     | upgrader.py          ( 140) | check_upgrade_succeeded | Node 'Node3' successfully upgraded to version 1.2.45
2017-11-17 13:57:32,639 | INFO     | node.py              (2503) | initStateFromLedger | Node3 found state to be empty, recreating from ledger
2017-11-17 13:57:32,640 | INFO     | stacks.py            (  84) | start | CONNECTION: Node3 listening for other nodes at 0.0.0.0:9705
2017-11-17 13:57:32,641 | INFO     | node.py              ( 680) | start | Node3 first time running...
2017-11-17 13:57:32,642 | INFO     | zstack.py            ( 585) | connect | CONNECTION: Node3 looking for Node2 at 10.0.0.3:9703
2017-11-17 13:57:32,643 | INFO     | zstack.py            ( 585) | connect | CONNECTION: Node3 looking for Node4 at 10.0.0.5:9707
2017-11-17 13:57:32,644 | INFO     | zstack.py            ( 585) | connect | CONNECTION: Node3 looking for Node1 at 10.0.0.2:9701
2017-11-17 13:57:32,777 | INFO     | node.py              (2031) | do_view_change_if_possible | VIEW CHANGE: Node3 initiating a view change to 3 from 0
2017-11-17 13:57:32,778 | INFO     | node.py              (2241) | startViewChange | VIEW CHANGE: Node3 changed to view 3, will start catchup now
2017-11-17 13:57:32,781 | INFO     | propagator.py        ( 179) | propagate | Node3 propagating request ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927047995990) from client Node2
2017-11-17 13:57:32,790 | INFO     | propagator.py        ( 179) | propagate | Node3 propagating request ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927048003928) from client Node1
2017-11-17 13:57:32,793 | INFO     | keep_in_touch.py     (  98) | _connsChanged | CONNECTION: Node3 now connected to Node2
2017-11-17 13:57:32,793 | INFO     | keep_in_touch.py     (  98) | _connsChanged | CONNECTION: Node3 now connected to Node1
2017-11-17 13:57:37,475 | INFO     | node.py              (1551) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 0
2017-11-17 13:57:37,476 | INFO     | ledger_manager.py    ( 865) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 0, caught up 0 in total
2017-11-17 13:57:37,476 | INFO     | node.py              (1551) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 2
2017-11-17 13:57:37,476 | INFO     | upgrader.py          ( 212) | processLedger | DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya found upgrade START txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)])
2017-11-17 13:57:37,477 | INFO     | upgrader.py          ( 290) | handleUpgradeTxn | Node 'Node3' handles upgrade txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)])
2017-11-17 13:57:37,477 | INFO     | upgrader.py          ( 310) | handleUpgradeTxn | Node 'Node3' has already performed an upgrade with upgrade_id 15109263663135171. Last recorded event is ('succeeded', datetime.datetime(2017, 11, 17, 13, 57, 0, 258870, tzinfo=tzlocal()), '1.2.45', '15109263663135171')
2017-11-17 13:57:37,477 | INFO     | node.py              (1551) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 1
2017-11-17 13:57:37,477 | INFO     | ledger_manager.py    ( 865) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 1, caught up 0 in total
2017-11-17 13:57:37,478 | INFO     | node.py              (1588) | allLedgersCaughtUp | CATCH-UP: Node3 caught up till (2, 1)
2017-11-17 13:57:37,478 | INFO     | node.py              (1598) | allLedgersCaughtUp | CATCH-UP: Node3 does not need any more catchups
2017-11-17 13:57:37,479 | DISPLAY  | primary_selector.py  ( 327) | _start_selection | PRIMARY SELECTION: Node3:0 selected primary Node4:0 for instance 0 (view 3)
2017-11-17 13:57:37,479 | INFO     | node.py              ( 532) | start_participating | Node3 started participating
2017-11-17 13:57:37,480 | DISPLAY  | primary_selector.py  ( 351) | _start_selection | VIEW CHANGE: Node3:0 declares view change 3 as completed for instance 0, new primary is Node4:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 4, '3aErFDAegiipoxjKyq8YBzyNpDPHaoNhZ8BqVu4e3uCg')]
2017-11-17 13:57:37,484 | DISPLAY  | primary_selector.py  ( 327) | _start_selection | PRIMARY SELECTION: Node3:1 selected primary Node1:1 for instance 1 (view 3)
2017-11-17 13:57:37,484 | INFO     | replica.py           (2025) | enqueue_prepare | Node3:1 queueing prepare due to unavailability of PRE-PREPARE. Prepare PREPARE{'ppTime': 1510927048, 'viewNo': 3, 'stateRootHash': None, 'txnRootHash': None, 'ppSeqNo': 1, 'digest': '35f37aa655aec5ad6e4da1c6df55a1f1e5607382ed0a3f3ab14e6ca9696c66df', 'instId': 1} from Node2:1
2017-11-17 13:57:37,485 | DISPLAY  | primary_selector.py  ( 351) | _start_selection | VIEW CHANGE: Node3:1 declares view change 3 as completed for instance 1, new primary is Node1:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 4, '3aErFDAegiipoxjKyq8YBzyNpDPHaoNhZ8BqVu4e3uCg')]
2017-11-17 13:57:37,485 | INFO     | upgrader.py          ( 149) | should_notify_about_upgrade_result | Node's 'Node3' last upgrade txn is OrderedDict([('data', OrderedDict([('action', 'in_progress'), ('version', '1.2.45')])), ('identifier', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya'), ('reqId', 1510927020097708), ('signature', '36bUMwhQhaVvVo1xHJqoG8WKH1uSiq2Sj5zN4Sr5BeNS1uxkGMiS1xcUJHkkE9QJuqYk3r4sq58kZLg7HVLvbpkp'), ('txnTime', 1510927020), ('type', '110'), ('seqNo', 4)])
2017-11-17 13:57:37,487 | INFO     | ledger_manager.py    ( 865) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 2, caught up 0 in total
2017-11-17 13:57:37,487 | INFO     | node.py              (1598) | allLedgersCaughtUp | CATCH-UP: Node3 does not need any more catchups
2017-11-17 13:57:37,488 | INFO     | node.py              (1685) | no_more_catchups_needed | Node3 starting to participate since catchup is done, primaries are selected but mode was not set to participating
2017-11-17 13:57:37,488 | INFO     | node.py              ( 532) | start_participating | Node3 started participating
2017-11-17 13:57:37,520 | INFO     | propagator.py        ( 179) | propagate | Node3 propagating request ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926752873422) from client Node2
2017-11-17 13:57:37,520 | INFO     | propagator.py        ( 179) | propagate | Node3 propagating request ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926748095846) from client Node1
2017-11-17 13:57:37,521 | INFO     | propagator.py        ( 179) | propagate | Node3 propagating request ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927057486576) from client Node3
2017-11-17 13:57:37,540 | INFO     | replica.py           (1619) | order_3pc_key | Node3:1 ordered batch request, view no 3, ppSeqNo 1, ledger 2, state root None, txn root None, requests ordered [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926748095846), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926752873422), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927048003928), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927047995990)], discarded []
2017-11-17 13:57:37,542 | INFO     | replica.py           (1619) | order_3pc_key | Node3:1 ordered batch request, view no 3, ppSeqNo 2, ledger 2, state root None, txn root None, requests ordered [('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927057486576)], discarded []
2017-11-17 13:57:42,599 | INFO     | monitor.py           ( 343) | isMasterThroughputTooLow | MONITORING: Node3 master throughput ratio 0.0 is lower than Delta 0.4.
2017-11-17 13:57:42,599 | INFO     | node.py              (2128) | sendInstanceChange | VIEW CHANGE: Node3 sending an instance change with view_no 4 since Primary of master protocol instance degraded the performance
2017-11-17 13:57:42,600 | INFO     | node.py              (2131) | sendInstanceChange | MONITORING: Node3 metrics for monitor: Node3 Monitor metrics:: None
            Delta: 0.4
            Lambda: 60
            Omega: 5
            instances started: [1381900.797609953, 1381900.797965252]
            ordered request counts: {0: 0, 1: 3}
            ordered request durations: {0: 0, 1: 0.06145448633469641}
            master request latencies: {}
            client avg request latencies: [{}, {'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': (1, 0.020191582152619958), 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya': (1, 0.02162105403840542), 'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv': (1, 0.019641850143671036)}]
            throughput: {0: 0, 1: 48.816615009378715}
            master throughput: 0
            total requests: 0
            avg backup throughput: 48.816615009378715
            master throughput ratio: 0.0
2017-11-17 13:57:42,600 | INFO     | node.py              (2031) | do_view_change_if_possible | VIEW CHANGE: Node3 initiating a view change to 4 from 3
2017-11-17 13:57:42,601 | INFO     | node.py              (2241) | startViewChange | VIEW CHANGE: Node3 changed to view 4, will start catchup now
2017-11-17 13:57:42,602 | INFO     | node.py              (2075) | checkPerformance | Node3 sent view change since performance degraded of master instance
2017-11-17 13:57:42,631 | INFO     | node.py              (1551) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 0
2017-11-17 13:57:42,631 | INFO     | ledger_manager.py    ( 865) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 0, caught up 0 in total
2017-11-17 13:57:42,649 | INFO     | node.py              (1551) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 2
2017-11-17 13:57:42,650 | INFO     | upgrader.py          ( 212) | processLedger | DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya found upgrade START txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)])
2017-11-17 13:57:42,650 | INFO     | upgrader.py          ( 290) | handleUpgradeTxn | Node 'Node3' handles upgrade txn OrderedDict([('action', 'start'), ('force', False), ('identifier', 'V4SGRU86Z58d6TV7PBUe6f'), ('justification', None), ('name', 'upgrade_to_the_latest_full'), ('reinstall', False), ('reqId', 1510926366313517), ('schedule', OrderedDict([('4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA', '2017-11-17T14:02:00.258870+00:00'), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', '2017-11-17T13:52:00.258870+00:00'), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', '2017-11-17T13:57:00.258870+00:00'), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', '2017-11-17T13:47:00.258870+00:00')])), ('sha256', 'ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f'), ('signature', 'rhY7W5BGPFvBC3TB4cgbrPSn11AMs9oAzBbuYyoqQLY7Yz791rzvcQSJU9FmJF3pPfvHH3oNGTgMkLGqdKFZkDj'), ('timeout', 10), ('txnTime', 1510926366), ('type', '109'), ('version', '1.2.45'), ('seqNo', 1)])
2017-11-17 13:57:42,650 | INFO     | upgrader.py          ( 310) | handleUpgradeTxn | Node 'Node3' has already performed an upgrade with upgrade_id 15109263663135171. Last recorded event is ('succeeded', datetime.datetime(2017, 11, 17, 13, 57, 0, 258870, tzinfo=tzlocal()), '1.2.45', '15109263663135171')
2017-11-17 13:57:42,650 | INFO     | upgrader.py          ( 149) | should_notify_about_upgrade_result | Node's 'Node3' last upgrade txn is OrderedDict([('data', OrderedDict([('action', 'in_progress'), ('version', '1.2.45')])), ('identifier', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya'), ('reqId', 1510927020097708), ('signature', '36bUMwhQhaVvVo1xHJqoG8WKH1uSiq2Sj5zN4Sr5BeNS1uxkGMiS1xcUJHkkE9QJuqYk3r4sq58kZLg7HVLvbpkp'), ('txnTime', 1510927020), ('type', '110'), ('seqNo', 4)])
2017-11-17 13:57:42,651 | INFO     | ledger_manager.py    ( 865) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 2, caught up 0 in total
2017-11-17 13:57:42,667 | INFO     | propagator.py        ( 179) | propagate | Node3 propagating request ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927062654152) from client Node1
2017-11-17 13:57:42,668 | INFO     | propagator.py        ( 179) | propagate | Node3 propagating request ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927062651136) from client Node3
2017-11-17 13:57:42,676 | INFO     | node.py              (1551) | preLedgerCatchUp | Node3 reverted 0 batches before starting catch up for ledger 1
2017-11-17 13:57:42,676 | INFO     | ledger_manager.py    ( 865) | catchupCompleted | CATCH-UP: Node3 completed catching up ledger 1, caught up 0 in total
2017-11-17 13:57:42,676 | INFO     | node.py              (1598) | allLedgersCaughtUp | CATCH-UP: Node3 does not need any more catchups
2017-11-17 13:57:42,678 | INFO     | propagator.py        ( 179) | propagate | Node3 propagating request ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927062662523) from client Node2
2017-11-17 13:57:42,697 | DISPLAY  | primary_selector.py  ( 327) | _start_selection | PRIMARY SELECTION: Node3:0 selected primary Node1:0 for instance 0 (view 4)
2017-11-17 13:57:42,697 | INFO     | node.py              ( 532) | start_participating | Node3 started participating
2017-11-17 13:57:42,698 | DISPLAY  | primary_selector.py  ( 351) | _start_selection | VIEW CHANGE: Node3:0 declares view change 4 as completed for instance 0, new primary is Node1:0, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 4, '3aErFDAegiipoxjKyq8YBzyNpDPHaoNhZ8BqVu4e3uCg')]
2017-11-17 13:57:42,698 | DISPLAY  | primary_selector.py  ( 327) | _start_selection | PRIMARY SELECTION: Node3:1 selected primary Node2:1 for instance 1 (view 4)
2017-11-17 13:57:42,699 | DISPLAY  | primary_selector.py  ( 351) | _start_selection | VIEW CHANGE: Node3:1 declares view change 4 as completed for instance 1, new primary is Node2:1, ledger info is [(0, 4, '5xizCdcGJoYwSK5swMP4BDasTxDbULANZozqM2M2uRo3'), (1, 15, 'TswSHbkNk6QAQJuyuqosYLYjuYEYZrRyCwuY9rGvzXx'), (2, 4, '3aErFDAegiipoxjKyq8YBzyNpDPHaoNhZ8BqVu4e3uCg')]
2017-11-17 13:57:42,726 | INFO     | replica.py           (1619) | order_3pc_key | Node3:0 ordered batch request, view no 4, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 8PHonJAkhGQDvCHhZ336fu17U8UV3wYpJGCjV7gdLdBd, requests ordered [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926748095846), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926752873422), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927048003928), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927047995990), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927057486576), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927062651136), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927062654152), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927062662523)], discarded []
2017-11-17 13:57:42,727 | INFO     | replica.py           (1619) | order_3pc_key | Node3:1 ordered batch request, view no 4, ppSeqNo 1, ledger 2, state root None, txn root None, requests ordered [('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927062651136), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927062654152), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927062662523)], discarded []
2017-11-17 13:57:42,730 | INFO     | node.py              (2374) | executeBatch | Node3 committed batch request, view no 4, ppSeqNo 1, ledger 2, state root DfNLmH4DAHTKv63YPFJzuRdeEtVwF5RtVnvKYHd8iLEA, txn root 8PHonJAkhGQDvCHhZ336fu17U8UV3wYpJGCjV7gdLdBd, requests: [('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510926748095846), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510926752873422), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927048003928), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927047995990), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927057486576), ('DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya', 1510927062651136), ('Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv', 1510927062654152), ('8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb', 1510927062662523)]
2017-11-17 13:57:44,766 | INFO     | keep_in_touch.py     (  98) | _connsChanged | CONNECTION: Node3 now connected to Node4