Uploaded image for project: 'Indy Node'
  1. Indy Node
  2. INDY-1150

Pool upgrade was scheduled but was not happened on part of nodes

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Complete
    • Priority: High
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      Environment:
      QA live pool (7+18 nodes)
      indy-node 1.2.214 -> 1.2.294

    • Epic Link:
    • Sprint:
      Sprint 18.03 Stability, DKMS

      Description

      Case 1:

      Steps to Reproduce:
      1. Setup the pool of 7 nodes.
      2. Add 18 nodes manually.
      3. Send some count of transactions (in my cases first time it was 300 txns, second time - 170).
      4. Schedule upgrade:

      send POOL_UPGRADE name=upgrade12294 version=1.2.294 sha256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 action=start schedule={'Gw6pDLhcBcoQesN72qfotTgFa7cbuqZpkX3Xo6pLhPhv':'2018-02-05T05:00:00.000000+00:00', '8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb': '2018-02-05T05:05:00.000000+00:00', 'DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya':'2018-02-05T05:10:00.000000+00:00', '4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA':'2018-02-05T05:15:00.000000+00:00', '4SWokCJWJc69Tn74VvLS6t2G2ucvXqM9FDMsWJjmsUxe':'2018-02-05T05:20:00.000000+00:00', 'Cv1Ehj43DDM5ttNBmC6VPpEfwXWwfGktHwjDJsTV5Fz8':'2018-02-05T05:25:00.000000+00:00', 'BM8dTooz5uykCbYSAAFwKNkYfT4koomBHsSWHTDtkjhW':'2018-02-05T05:30:00.000000+00:00', '4EG9n9ErTVqcZ7xSnhHH8PQVtYDfugw5SwqcezvYVrGg':'2018-02-05T05:35:00.000000+00:00', 'FR454t4km7JJvaZc6ttzFU9Lor5ib9YB3wao1kJeoEZb':'2018-02-05T05:40:00.000000+00:00', 'F9abmLwUQy5svguBPNi2GQTdKsiVGDAyBfaFpwcpLJDm':'2018-02-05T05:45:00.000000+00:00', '72vCSKVDbcbpnCXekEd9vhagse75uXnjJDjDGYDxeXPG':'2018-02-05T05:50:00.000000+00:00', '8oE3bGvZTzGxVQRHxfLZHD8Uyx2cbX6wnF1aQAiJe5sm':'2018-02-05T05:55:00.000000+00:00', 'A9hZwUqe62MNQXoswU47271CyRf7g1G5Gat3aqeQ6DeN':'2018-02-05T06:00:00.000000+00:00', '2LssMb56SBibrQYFhRnnMvZHhEiaorMuNkmW8QsZfVwA':'2018-02-05T06:05:00.000000+00:00', '8A5NzusREw844wQmttwBqhhWkTNby5o4UvYq6T1FsByb':'2018-02-05T06:10:00.000000+00:00', 'Bh492xBFGYKS7Z57EQkP5cQtJp6jDvypHoSpXHh259q5':'2018-02-05T06:15:00.000000+00:00', '2MbQjn7ij9DFKT7rt425SaeDvgzyjycv72FEiPVdacEb':'2018-02-05T06:20:00.000000+00:00', 'EwZyzG8HBvjWvxmWVgreTVWYQJScpWMVKeqUa7Rk1Pr5':'2018-02-05T06:25:00.000000+00:00', '58b3Fy45qjcBfVtEt2Zi1MgiRzX9PPmj68FwD143SuWQ':'2018-02-05T06:30:00.000000+00:00', '2FGgKVcp2heyWiGTDLVEyF6AJrfaQBvrhkUzhYTjiHA6':'2018-02-05T06:35:00.000000+00:00', '6CRQcKzeRMCstErDT2Pso4he3rWWu1m16CRyp1fjYCFx':'2018-02-05T06:40:00.000000+00:00', '53skV1LWLCbcxxdvoxY3pKDx2MAvszA27hA6cBZxLbnf':'2018-02-05T06:45:00.000000+00:00', 'CbW92yCBgTMKquvsSRzDn5aA5uHzWZfP85bcW6RUK4hk':'2018-02-05T06:50:00.000000+00:00', 'H5cW9eWhcBSEHfaAVkqP5QNa11m6kZ9zDyRXQZDBoSpq':'2018-02-05T06:55:00.000000+00:00', 'DE8JMTgA7DaieF9iGKAyy5yvsZovroHr3SMEoDnbgFcp':'2018-02-05T07:00:00.000000+00:00'} timeout=10 force=False

      Actual Results:
      After 300 txns: 6 of 25 nodes were not upgraded (12, 15, 19, 22, 23, 24).
      After 170 txns: 5 of 25 nodes were not upgraded (14, 16, 23, 24, 25). After restart of node14, it was upgraded. 

      Expected Results:
      Pool should be successfully upgraded.

      Logs and data:
      After 300 txns: https://drive.google.com/open?id=1xUok5VxjQShQP29XD85h16BWk6bRSLH4
      Unfortunately, data of not upgraded nodes was not copied for not upgraded nodes.

      After 170 txns: https://drive.google.com/file/d/1RcIam23g7xCCSWD_Av4J493LbgSq-7Bd/view?usp=sharing
      journalctl: https://drive.google.com/file/d/1uG0AGwxb8Tk5KUaGO67HBjt5b-ZzBTYU/view?usp=sharing
      Config ledger: https://drive.google.com/file/d/17AXPe6sOYaXIKV0Hk6WbiaxDEwHg7lZN/view?usp=sharing

      Case 2 (in progress):
      Need to try the same case for upgrade with force=true.
      UPD: POOL_UPGRADE with force=True at the same time works without any problems. All nodes were upgraded and worked after upgrade.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned
              Reporter:
              ozheregelya Olga Zheregelya
              Watchers:
              Andrew Nikitin, Olga Zheregelya
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: