Uploaded image for project: 'Indy Node'
  1. Indy Node
  2. INDY-1447

Upgrade failed on pool from 1.3.62 to 1.4.66

    Details

    • Type: Bug
    • Status: Complete
    • Priority: Highest
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: 1.5
    • Component/s: None
    • Labels:
      None
    • Sprint:
      EV 18.13 Benchmark hardening, EV 18.14 Monitoring/Stability, EV 18.15 Stability/Availabilit

      Description

      I have 7 nodes spread globally.
      I had upgraded them from 1.3.57 to 1.3.62 successfully.
      Some nodes have an upgrade.log file showing the successful upgrade, but most do not.

      Issue
      I upgraded yesterday from 1.3.62 to 1.4.66
      I sent the upgrade txn at noon and scheduled the upgrade for 1:30pm

      Schedule

      ledger pool-upgrade name=upgrade1366 version=1.4.66 action=start sha256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 schedule={"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv":"2018-07-02T13:30:00.258870-06:00","8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb":"2018-07-02T13:30:00.258870-06:00","DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya":"2018-07-02T13:30:00.258870-06:00","4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA":"2018-07-02T13:30:00.258870-06:00","4SWokCJWJc69Tn74VvLS6t2G2ucvXqM9FDMsWJjmsUxe":"2018-07-02T13:30:00.258870-06:00","Cv1Ehj43DDM5ttNBmC6VPpEfwXWwfGktHwjDJsTV5Fz8":"2018-07-02T13:30:00.258870-06:00","BM8dTooz5uykCbYSAAFwKNkYfT4koomBHsSWHTDtkjhW":"2018-07-02T13:30:00.258870-06:00"} timeout=15 force=true
      

      I checked on the upgrade the next day and found 3 of the 4 nodes successfully upgraded. The other 4 nodes did not and do not have anything in the journalctl file showing they even tried to perform an upgrade.
      I could not see any errors in any logs and most don't have an upgrade.log file even from the last upgrade.

      The node log files show a view change was being requested before and after the upgrade.

      I am attaching the node logs.
      The machines are located in AWS and are listed in the AWS QA Group - Regional Usagespreadsheet with evernym. Go to the tab "Pool-QA RC"

      The machines have the indy-node service shut off on all of them to be checked before attempting a manual upgrade.
      To gain access talk with Vladimir Shishkin or Kelly Wilson on how to login

        Attachments

        1. 1447_without_drop.tar.gz
          1.04 MB
        2. 1447.tar.gz
          339 kB
        3. config_ledger.txt
          11 kB
        4. Node1_control.log
          1 kB
        5. Node1_upgrade_log
          0.6 kB
        6. Node1.log
          1.43 MB
        7. Node2_control.log
          1 kB
        8. Node2_upgrade_log
          0.6 kB
        9. Node2.log
          1.42 MB
        10. Node3_control.log
          1 kB
        11. Node3_upgrade_log
          0.6 kB
        12. Node3.log
          4.67 MB
        13. Node4_control.log
          19 kB
        14. Node4.log
          2.53 MB
        15. Node5_control.log
          1 kB
        16. Node5_upgrade_log
          0.6 kB
        17. Node5.log
          1.42 MB
        18. Node6_control.log
          19 kB
        19. Node6.log
          3.93 MB
        20. Node7_control.log
          19 kB
        21. Node7.log
          3.61 MB
        22. pool_ledger.txt
          3 kB

          Issue Links

            Activity

              People

              • Assignee:
                VladimirWork Vladimir Shishkin
                Reporter:
                krw910 Kelly Wilson
                Watchers:
                Alexander Shcherbakov, Kelly Wilson, Nikita Spivachuk, Vladimir Shishkin
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: