2018-01-17 14:02:34,066 | DEBUG | selector_events.py (53) | __init__ | Using selector: EpollSelector 2018-01-17 14:02:34,085 | DEBUG | ledger.py (201) | start | Starting ledger... 2018-01-17 14:02:34,093 | DEBUG | ledger.py (67) | recoverTree | Recovering tree from transaction log 2018-01-17 14:02:34,111 | DEBUG | ledger.py (82) | recoverTree | Recovered tree in 0.018196233999333344 seconds 2018-01-17 14:02:34,147 | DEBUG | ledger.py (201) | start | Starting ledger... 2018-01-17 14:02:34,155 | DEBUG | ledger.py (67) | recoverTree | Recovering tree from transaction log 2018-01-17 14:02:34,172 | DEBUG | ledger.py (82) | recoverTree | Recovered tree in 0.016735046010580845 seconds 2018-01-17 14:02:34,182 | INFO | node.py (2565) | initStateFromLedger | Node5 found state to be empty, recreating from ledger 2018-01-17 14:02:34,186 | INFO | pool_manager.py (399) | _order_node | Node5 node Node1 ordered, NYM Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv 2018-01-17 14:02:34,186 | INFO | pool_manager.py (399) | _order_node | Node5 node Node2 ordered, NYM 8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb 2018-01-17 14:02:34,186 | INFO | pool_manager.py (399) | _order_node | Node5 node Node3 ordered, NYM DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya 2018-01-17 14:02:34,187 | INFO | pool_manager.py (399) | _order_node | Node5 node Node4 ordered, NYM 4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA 2018-01-17 14:02:34,200 | INFO | node.py (721) | _create_bls_bft | BLS: BLS Signatures will be used for Node Node5 2018-01-17 14:02:34,209 | INFO | node.py (2565) | initStateFromLedger | Node5 found state to be empty, recreating from ledger 2018-01-17 14:02:34,218 | INFO | node.py (558) | setPoolParams | Node5 updated its pool parameters: f 1, totalNodes 5, allNodeNames {'Node5', 'Node4', 'Node2', 'Node1', 'Node3'}, requiredNumberOfInstances 2, minimumNodes 3, quorums {'f': 1, 'timestamp': Quorum(2), 'propagate': Quorum(2), 'propagate_primary': Quorum(2), 'same_consistency_proof': Quorum(2), 'view_change_done': Quorum(4), 'checkpoint': Quorum(2), 'consistency_proof': Quorum(2), 'bls_signatures': Quorum(4), 'reply': Quorum(2), 'prepare': Quorum(3), 'view_change': Quorum(4), 'election': Quorum(4), 'observer_data': Quorum(2), 'ledger_status': Quorum(3), 'commit': Quorum(4)} 2018-01-17 14:02:34,261 | INFO | plugin_loader.py (118) | _load | plugin FirebaseStatsConsumer successfully loaded from module plugin_firebase_stats_consumer 2018-01-17 14:02:34,263 | DISPLAY | replicas.py (41) | grow | Node5 added replica Node5:0 to instance 0 (master) 2018-01-17 14:02:34,263 | DISPLAY | replicas.py (41) | grow | Node5 added replica Node5:1 to instance 1 (backup) 2018-01-17 14:02:34,264 | DEBUG | plugin_helper.py (23) | loadPlugins | Plugin loading started to load plugins from plugins_dir: /var/lib/indy/plugins 2018-01-17 14:02:34,264 | DEBUG | plugin_helper.py (28) | loadPlugins | Plugin directory created at: /var/lib/indy/plugins 2018-01-17 14:02:34,264 | DEBUG | plugin_helper.py (63) | loadPlugins | Total plugins loaded from plugins_dir /var/lib/indy/plugins are : 0 2018-01-17 14:02:34,289 | DEBUG | ledger.py (201) | start | Starting ledger... 2018-01-17 14:02:34,298 | DEBUG | ledger.py (67) | recoverTree | Recovering tree from transaction log 2018-01-17 14:02:34,314 | DEBUG | ledger.py (82) | recoverTree | Recovered tree in 0.016385285009164363 seconds 2018-01-17 14:02:34,355 | INFO | node.py (2565) | initStateFromLedger | Node5 found state to be empty, recreating from ledger 2018-01-17 14:02:34,356 | DEBUG | ledger.py (199) | start | Ledger already started. 2018-01-17 14:02:34,356 | DEBUG | ledger.py (199) | start | Ledger already started. 2018-01-17 14:02:34,356 | DEBUG | ledger.py (199) | start | Ledger already started. 2018-01-17 14:02:34,357 | DEBUG | authenticator.py (31) | start | Starting ZAP at inproc://zeromq.zap.1 2018-01-17 14:02:34,357 | DEBUG | base.py (72) | allow | Allowing 0.0.0.0 2018-01-17 14:02:34,358 | DEBUG | base.py (112) | configure_curve | Configure curve: *[/var/lib/indy/sandbox/keys/Node5/public_keys] 2018-01-17 14:02:34,359 | INFO | stacks.py (84) | start | CONNECTION: Node5 listening for other nodes at 0.0.0.0:9701 2018-01-17 14:02:34,359 | DEBUG | authenticator.py (31) | start | Starting ZAP at inproc://zeromq.zap.2 2018-01-17 14:02:34,359 | DEBUG | base.py (72) | allow | Allowing 0.0.0.0 2018-01-17 14:02:34,360 | DEBUG | base.py (112) | configure_curve | Configure curve: *[*] 2018-01-17 14:02:34,360 | INFO | node.py (786) | start | Node5 first time running... 2018-01-17 14:02:34,362 | INFO | zstack.py (584) | connect | CONNECTION: Node5 looking for Node3 at 10.0.0.103:9705 2018-01-17 14:02:34,364 | INFO | zstack.py (584) | connect | CONNECTION: Node5 looking for Node4 at 10.0.0.104:9707 2018-01-17 14:02:34,368 | INFO | zstack.py (584) | connect | CONNECTION: Node5 looking for Node1 at 10.0.0.101:9701 2018-01-17 14:02:34,369 | INFO | zstack.py (584) | connect | CONNECTION: Node5 looking for Node2 at 10.0.0.102:9703 2018-01-17 14:03:34,366 | INFO | node.py (2258) | propose_view_change | Node5 primary has been disconnected for too long 2018-01-17 14:03:34,367 | INFO | view_changer.py (206) | on_primary_loss | Node5 sending instance with view_no = 1 and trying to start view change since primary was lost 2018-01-17 14:03:34,367 | INFO | view_changer.py (380) | sendInstanceChange | VIEW CHANGE: Node5 sending an instance change with view_no 1 since Primary of master protocol instance disconnected 2018-01-17 14:03:34,368 | INFO | view_changer.py (383) | sendInstanceChange | MONITORING: Node5 metrics for monitor: Node5 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [97968.134585899, 97968.135100477] ordered request counts: {0: 0, 1: 0} ordered request durations: {0: 0, 1: 0} master request latencies: {} client avg request latencies: [{}, {}] throughput: {0: 0, 1: 0} master throughput: None total requests: 0 avg backup throughput: None master throughput ratio: None 2018-01-17 14:03:54,408 | INFO | view_changer.py (430) | _start_view_change_if_possible | VIEW CHANGE: Node5 starting view change for 2 after 2 view change indications from other nodes 2018-01-17 14:03:54,409 | INFO | node.py (485) | on_view_change_start | VIEW CHANGE: Node5 changed to view 2, will start catchup now 2018-01-17 14:03:54,427 | INFO | node.py (1696) | preLedgerCatchUp | Node5 reverted 0 batches before starting catch up for ledger 0 2018-01-17 14:03:54,442 | INFO | pool_manager.py (399) | _order_node | Node5 node Node5 ordered, NYM 4Tn3wZMNCvhSTXPcLinQDnHyj56DTLQtL61ki4jo2Loc 2018-01-17 14:03:54,444 | INFO | node.py (753) | update_bls_key | BLS: BLS key is rotated/set for Node Node5. BLS Signatures will be used for Node. 2018-01-17 14:03:54,447 | INFO | node.py (1696) | preLedgerCatchUp | Node5 reverted 0 batches before starting catch up for ledger 2 2018-01-17 14:03:54,449 | INFO | ledger_manager.py (871) | mark_ledger_synced | CATCH-UP: Node5 completed catching up ledger 2, caught up 0 in total 2018-01-17 14:03:54,450 | INFO | ledger_manager.py (871) | mark_ledger_synced | CATCH-UP: Node5 completed catching up ledger 0, caught up 1 in total 2018-01-17 14:03:54,477 | INFO | ledger_manager.py (871) | mark_ledger_synced | CATCH-UP: Node5 completed catching up ledger 2, caught up 0 in total 2018-01-17 14:03:54,479 | INFO | node.py (1696) | preLedgerCatchUp | Node5 reverted 0 batches before starting catch up for ledger 1 2018-01-17 14:08:20,150 | INFO | ledger_manager.py (871) | mark_ledger_synced | CATCH-UP: Node5 completed catching up ledger 1, caught up 80003 in total 2018-01-17 14:08:20,151 | INFO | node.py (1733) | allLedgersCaughtUp | CATCH-UP: Node5 caught up till (2, 999) 2018-01-17 14:08:20,152 | INFO | node.py (1738) | allLedgersCaughtUp | Node5 needs to catchup again 2018-01-17 14:08:20,184 | INFO | view_changer.py (232) | on_view_change_not_completed_in_time | Node5 sending instance with view_no = 3 since view change to view 2 is not completed in time 2018-01-17 14:08:20,185 | INFO | view_changer.py (380) | sendInstanceChange | VIEW CHANGE: Node5 sending an instance change with view_no 3 since View change could not complete in time 2018-01-17 14:08:20,185 | INFO | view_changer.py (383) | sendInstanceChange | MONITORING: Node5 metrics for monitor: Node5 Monitor metrics:: None Delta: 0.4 Lambda: 60 Omega: 5 instances started: [97968.134585899, 97968.135100477] ordered request counts: {0: 0, 1: 0} ordered request durations: {0: 0, 1: 0} master request latencies: {} client avg request latencies: [{}, {}] throughput: {0: 0, 1: 0} master throughput: None total requests: 0 avg backup throughput: None master throughput ratio: None 2018-01-17 14:08:20,209 | INFO | node.py (1696) | preLedgerCatchUp | Node5 reverted 0 batches before starting catch up for ledger 0 2018-01-17 14:08:20,209 | INFO | ledger_manager.py (871) | mark_ledger_synced | CATCH-UP: Node5 completed catching up ledger 0, caught up 0 in total 2018-01-17 14:08:20,233 | INFO | keep_in_touch.py (98) | _connsChanged | CONNECTION: Node5 now connected to Node2 2018-01-17 14:08:20,247 | INFO | node.py (1696) | preLedgerCatchUp | Node5 reverted 0 batches before starting catch up for ledger 2 2018-01-17 14:08:20,248 | INFO | ledger_manager.py (871) | mark_ledger_synced | CATCH-UP: Node5 completed catching up ledger 2, caught up 0 in total 2018-01-17 14:08:20,250 | INFO | keep_in_touch.py (98) | _connsChanged | CONNECTION: Node5 now connected to Node4 2018-01-17 14:08:20,250 | INFO | keep_in_touch.py (98) | _connsChanged | CONNECTION: Node5 now connected to Node1 2018-01-17 14:08:20,250 | INFO | keep_in_touch.py (98) | _connsChanged | CONNECTION: Node5 now connected to Node3 2018-01-17 14:08:20,267 | INFO | node.py (1696) | preLedgerCatchUp | Node5 reverted 0 batches before starting catch up for ledger 1 2018-01-17 14:08:20,267 | INFO | ledger_manager.py (871) | mark_ledger_synced | CATCH-UP: Node5 completed catching up ledger 1, caught up 0 in total 2018-01-17 14:08:20,269 | INFO | node.py (1743) | allLedgersCaughtUp | CATCH-UP: Node5 does not need any more catchups 2018-01-17 14:08:20,270 | DISPLAY | node.py (2286) | select_primaries | PRIMARY SELECTION: Node5:0 selected primary Node3:0 for instance 0 (view 2) 2018-01-17 14:08:20,270 | INFO | node.py (611) | start_participating | Node5 started participating 2018-01-17 14:08:20,270 | DISPLAY | node.py (2308) | select_primaries | VIEW CHANGE: Node5:0 declares view change 2 as completed for instance 0, new primary is Node3:0, ledger info is [(0, 5, '6dFcz2WurSBDtEfrNQeAPwE7U8dwNoiRvMPcm6hGLntP'), (1, 80012, '848xTKh6sSLUnneBfNzKGzT4PPVhUfKgQ5512bkdHZv8'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2018-01-17 14:08:20,271 | DISPLAY | node.py (2286) | select_primaries | PRIMARY SELECTION: Node5:1 selected primary Node4:1 for instance 1 (view 2) 2018-01-17 14:08:20,271 | DISPLAY | node.py (2308) | select_primaries | VIEW CHANGE: Node5:1 declares view change 2 as completed for instance 1, new primary is Node4:1, ledger info is [(0, 5, '6dFcz2WurSBDtEfrNQeAPwE7U8dwNoiRvMPcm6hGLntP'), (1, 80012, '848xTKh6sSLUnneBfNzKGzT4PPVhUfKgQ5512bkdHZv8'), (2, 0, 'GKot5hBsd81kMupNCXHaqbhv3huEbxAFMLnpcX2hniwn')] 2018-01-17 14:09:43,652 | INFO | propagator.py (179) | propagate | Node5 propagating request ('V4SGRU86Z58d6TV7PBUe6f', 1516198183626847) from client b'ek%(qbP+Eu