indy-node 1.6.753, 1.6.759 (w/o freshness)
Steps to Reproduce:
1. Setup the pool.
2. In case of indy-node 1.6.759 set UPDATE_STATE_FRESHNESS = False in config.
3. Restart pool.
4. Start production load.
- Two nodes were lagged on ~3K txns in domain ledger.
- On both of lagged nodes there are no logs and metrics and validator-info updates after lagging:
- Uptime of indy-node services according to systemctl status is the same as on another nodes.
- On not lagged nodes following warnings constantly appear:
- The last message from lagged node in alive node logs:
- There are no any errors or warnings in journalctl.
- Node should not lag.
- Even if node lags, it should continue writing logs and metrics.
Logs and metrics:
1) s3://qanodelogs/indy-1876/prod_load_14_01_2019 (Node13 is lagged here)
To get logs, run following command on log processor machine:
aws s3 cp --recursive s3://qanodelogs/indy-1876/prod_load_14_01_2019/ /home/ev/logs/indy-1876/prod_load_14_01_2019/
At the end of this test following nodes were lagged: