Uploaded image for project: 'Indy Node'
  1. Indy Node
  2. INDY-1447

Upgrade failed on pool from 1.3.62 to 1.4.66

    XMLWordPrintable

Details

    • Bug
    • Status: Complete
    • Highest
    • Resolution: Done
    • None
    • 1.5
    • None
    • None
    • EV 18.13 Benchmark hardening, EV 18.14 Monitoring/Stability, EV 18.15 Stability/Availabilit

    Description

      I have 7 nodes spread globally.
      I had upgraded them from 1.3.57 to 1.3.62 successfully.
      Some nodes have an upgrade.log file showing the successful upgrade, but most do not.

      Issue
      I upgraded yesterday from 1.3.62 to 1.4.66
      I sent the upgrade txn at noon and scheduled the upgrade for 1:30pm

      Schedule

      ledger pool-upgrade name=upgrade1366 version=1.4.66 action=start sha256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 schedule={"Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv":"2018-07-02T13:30:00.258870-06:00","8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb":"2018-07-02T13:30:00.258870-06:00","DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya":"2018-07-02T13:30:00.258870-06:00","4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA":"2018-07-02T13:30:00.258870-06:00","4SWokCJWJc69Tn74VvLS6t2G2ucvXqM9FDMsWJjmsUxe":"2018-07-02T13:30:00.258870-06:00","Cv1Ehj43DDM5ttNBmC6VPpEfwXWwfGktHwjDJsTV5Fz8":"2018-07-02T13:30:00.258870-06:00","BM8dTooz5uykCbYSAAFwKNkYfT4koomBHsSWHTDtkjhW":"2018-07-02T13:30:00.258870-06:00"} timeout=15 force=true
      

      I checked on the upgrade the next day and found 3 of the 4 nodes successfully upgraded. The other 4 nodes did not and do not have anything in the journalctl file showing they even tried to perform an upgrade.
      I could not see any errors in any logs and most don't have an upgrade.log file even from the last upgrade.

      The node log files show a view change was being requested before and after the upgrade.

      I am attaching the node logs.
      The machines are located in AWS and are listed in the AWS QA Group - Regional Usagespreadsheet with evernym. Go to the tab "Pool-QA RC"

      The machines have the indy-node service shut off on all of them to be checked before attempting a manual upgrade.
      To gain access talk with VladimirWork or krw910 on how to login

      Attachments

        1. pool_ledger.txt
          3 kB
        2. Node7.log
          3.61 MB
        3. Node7_control.log
          19 kB
        4. Node6.log
          3.93 MB
        5. Node6_control.log
          19 kB
        6. Node5.log
          1.42 MB
        7. Node5_upgrade_log
          0.6 kB
        8. Node5_control.log
          1 kB
        9. Node4.log
          2.53 MB
        10. Node4_control.log
          19 kB
        11. Node3.log
          4.67 MB
        12. Node3_upgrade_log
          0.6 kB
        13. Node3_control.log
          1 kB
        14. Node2.log
          1.42 MB
        15. Node2_upgrade_log
          0.6 kB
        16. Node2_control.log
          1 kB
        17. Node1.log
          1.43 MB
        18. Node1_upgrade_log
          0.6 kB
        19. Node1_control.log
          1 kB
        20. config_ledger.txt
          11 kB
        21. 1447.tar.gz
          339 kB
        22. 1447_without_drop.tar.gz
          1.04 MB

        Issue Links

          Activity

            People

              VladimirWork Vladimir Shishkin
              krw910 Kelly Wilson
              Alexander Shcherbakov, Kelly Wilson, Nikita Spivachuk, Vladimir Shishkin
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: