Uploaded image for project: 'Indy Node'
  1. Indy Node
  2. INDY-1150

Pool upgrade was scheduled but was not happened on part of nodes

    XMLWordPrintable

Details

    • Bug
    • Status: Complete
    • High
    • Resolution: Done
    • None
    • None
    • None
    • None
    • Environment:
      QA live pool (7+18 nodes)
      indy-node 1.2.214 -> 1.2.294

    • Sprint 18.03 Stability, DKMS

    Description

      Case 1:

      Steps to Reproduce:
      1. Setup the pool of 7 nodes.
      2. Add 18 nodes manually.
      3. Send some count of transactions (in my cases first time it was 300 txns, second time - 170).
      4. Schedule upgrade:

      send POOL_UPGRADE name=upgrade12294 version=1.2.294 sha256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 action=start schedule={'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv':'2018-02-05T05:00:00.000000+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2018-02-05T05:05:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya':'2018-02-05T05:10:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA':'2018-02-05T05:15:00.000000+00:00', '4SWokCJWJc69Tn74VvLS6t2G2ucvXqM9FDMsWJjmsUxe':'2018-02-05T05:20:00.000000+00:00', 'Cv1Ehj43DDM5ttNBmC6VPpEfwXWwfGktHwjDJsTV5Fz8':'2018-02-05T05:25:00.000000+00:00', 'BM8dTooz5uykCbYSAAFwKNkYfT4koomBHsSWHTDtkjhW':'2018-02-05T05:30:00.000000+00:00', '4EG9n9ErTVqcZ7xSnhHH8PQVtYDfugw5SwqcezvYVrGg':'2018-02-05T05:35:00.000000+00:00', 'FR454t4km7JJvaZc6ttzFU9Lor5ib9YB3wao1kJeoEZb':'2018-02-05T05:40:00.000000+00:00', 'F9abmLwUQy5svguBPNi2GQTdKsiVGDAyBfaFpwcpLJDm':'2018-02-05T05:45:00.000000+00:00', '72vCSKVDbcbpnCXekEd9vhagse75uXnjJDjDGYDxeXPG':'2018-02-05T05:50:00.000000+00:00', '8oE3bGvZTzGxVQRHxfLZHD8Uyx2cbX6wnF1aQAiJe5sm':'2018-02-05T05:55:00.000000+00:00', 'A9hZwUqe62MNQXoswU47271CyRf7g1G5Gat3aqeQ6DeN':'2018-02-05T06:00:00.000000+00:00', '2LssMb56SBibrQYFhRnnMvZHhEiaorMuNkmW8QsZfVwA':'2018-02-05T06:05:00.000000+00:00', '8A5NzusREw844wQmttwBqhhWkTNby5o4UvYq6T1FsByb':'2018-02-05T06:10:00.000000+00:00', 'Bh492xBFGYKS7Z57EQkP5cQtJp6jDvypHoSpXHh259q5':'2018-02-05T06:15:00.000000+00:00', '2MbQjn7ij9DFKT7rt425SaeDvgzyjycv72FEiPVdacEb':'2018-02-05T06:20:00.000000+00:00', 'EwZyzG8HBvjWvxmWVgreTVWYQJScpWMVKeqUa7Rk1Pr5':'2018-02-05T06:25:00.000000+00:00', '58b3Fy45qjcBfVtEt2Zi1MgiRzX9PPmj68FwD143SuWQ':'2018-02-05T06:30:00.000000+00:00', '2FGgKVcp2heyWiGTDLVEyF6AJrfaQBvrhkUzhYTjiHA6':'2018-02-05T06:35:00.000000+00:00', '6CRQcKzeRMCstErDT2Pso4he3rWWu1m16CRyp1fjYCFx':'2018-02-05T06:40:00.000000+00:00', '53skV1LWLCbcxxdvoxY3pKDx2MAvszA27hA6cBZxLbnf':'2018-02-05T06:45:00.000000+00:00', 'CbW92yCBgTMKquvsSRzDn5aA5uHzWZfP85bcW6RUK4hk':'2018-02-05T06:50:00.000000+00:00', 'H5cW9eWhcBSEHfaAVkqP5QNa11m6kZ9zDyRXQZDBoSpq':'2018-02-05T06:55:00.000000+00:00', 'DE8JMTgA7DaieF9iGKAyy5yvsZovroHr3SMEoDnbgFcp':'2018-02-05T07:00:00.000000+00:00'} timeout=10 force=False

      Actual Results:
      After 300 txns: 6 of 25 nodes were not upgraded (12, 15, 19, 22, 23, 24).
      After 170 txns: 5 of 25 nodes were not upgraded (14, 16, 23, 24, 25). After restart of node14, it was upgraded. 

      Expected Results:
      Pool should be successfully upgraded.

      Logs and data:
      After 300 txns: https://drive.google.com/open?id=1xUok5VxjQShQP29XD85h16BWk6bRSLH4
      Unfortunately, data of not upgraded nodes was not copied for not upgraded nodes.

      After 170 txns: https://drive.google.com/file/d/1RcIam23g7xCCSWD_Av4J493LbgSq-7Bd/view?usp=sharing
      journalctl: https://drive.google.com/file/d/1uG0AGwxb8Tk5KUaGO67HBjt5b-ZzBTYU/view?usp=sharing
      Config ledger: https://drive.google.com/file/d/17AXPe6sOYaXIKV0Hk6WbiaxDEwHg7lZN/view?usp=sharing

      Case 2 (in progress):
      Need to try the same case for upgrade with force=true.
      UPD: POOL_UPGRADE with force=True at the same time works without any problems. All nodes were upgraded and worked after upgrade.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              ozheregelya Olga Zheregelya
              Andrew Nikitin, Olga Zheregelya
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: