Details
-
Bug
-
Status: Complete
-
Highest
-
Resolution: Done
-
None
-
None
-
None
-
EV 18.13 Benchmark hardening, EV 18.14 Monitoring/Stability, EV 18.15 Stability/Availabilit
Description
I have 7 nodes spread globally.
I had upgraded them from 1.3.57 to 1.3.62 successfully.
Some nodes have an upgrade.log file showing the successful upgrade, but most do not.
Issue
I upgraded yesterday from 1.3.62 to 1.4.66
I sent the upgrade txn at noon and scheduled the upgrade for 1:30pm
Schedule
ledger pool-upgrade name=upgrade1366 version=1.4.66 action=start sha256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 schedule={"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv":"2018-07-02T13:30:00.258870-06:00","8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb":"2018-07-02T13:30:00.258870-06:00","DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya":"2018-07-02T13:30:00.258870-06:00","4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA":"2018-07-02T13:30:00.258870-06:00","4SWokCJWJc69Tn74VvLS6t2G2ucvXqM9FDMsWJjmsUxe":"2018-07-02T13:30:00.258870-06:00","Cv1Ehj43DDM5ttNBmC6VPpEfwXWwfGktHwjDJsTV5Fz8":"2018-07-02T13:30:00.258870-06:00","BM8dTooz5uykCbYSAAFwKNkYfT4koomBHsSWHTDtkjhW":"2018-07-02T13:30:00.258870-06:00"} timeout=15 force=true
I checked on the upgrade the next day and found 3 of the 4 nodes successfully upgraded. The other 4 nodes did not and do not have anything in the journalctl file showing they even tried to perform an upgrade.
I could not see any errors in any logs and most don't have an upgrade.log file even from the last upgrade.
The node log files show a view change was being requested before and after the upgrade.
I am attaching the node logs.
The machines are located in AWS and are listed in the AWS QA Group - Regional Usagespreadsheet with evernym. Go to the tab "Pool-QA RC"
The machines have the indy-node service shut off on all of them to be checked before attempting a manual upgrade.
To gain access talk with VladimirWork or krw910 on how to login