Uploaded image for project: 'Indy Node'
  1. Indy Node
  2. INDY-1519

1.3.62 -> 1.5.67 forced upgrade without one node in schedule was failed

    Details

    • Type: Bug
    • Status: Complete
    • Priority: Highest
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: 1.5
    • Component/s: None
    • Labels:
      None
    • Documentation Impact:
      Unset
    • Sprint:
      EV 18.15 Stability/Availabilit

      Description

      Build Info:
      indy-node 1.3.62 -> 1.5.67

      Steps to Reproduce:
      1. Install pool of 7 nodes.
      2. Schedule forced upgrade with the 1st node missing via old CLI on future time:

      send POOL_UPGRADE name=upgrade67new version=1.5.67 sha256=ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f action=start schedule={'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb':'2018-07-24T15:00:00.258870+00:00','DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya':'2018-07-24T15:00:00.258870+00:00','4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA':'2018-07-24T15:00:00.258870+00:00','4SWokCJWJc69Tn74VvLS6t2G2ucvXqM9FDMsWJjmsUxe':'2018-07-24T15:00:00.258870+00:00','Cv1Ehj43DDM5ttNBmC6VPpEfwXWwfGktHwjDJsTV5Fz8':'2018-07-24T15:00:00.258870+00:00','BM8dTooz5uykCbYSAAFwKNkYfT4koomBHsSWHTDtkjhW':'2018-07-24T15:00:00.258870+00:00'} timeout=10 force=True
      

      Actual Results:
      1. Upgrade was triggered right after scheduling (not in time).
      2. 2nd..7th nodes were upgraded but node service was failed periodically with this error:

      Jul 24 14:53:20 e7c7073a1401 env[7206]: Traceback (most recent call last):
      Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/bin/start_indy_node", line 19, in <module>
      Jul 24 14:53:20 e7c7073a1401 env[7206]:     client_ip=sys.argv[4], client_port=int(sys.argv[5]))
      Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/indy_node/utils/node_runner.py", line 54, in run_node
      Jul 24 14:53:20 e7c7073a1401 env[7206]:     looper.run()
      Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 263, in run
      Jul 24 14:53:20 e7c7073a1401 env[7206]:     return self.loop.run_until_complete(what)
      Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
      Jul 24 14:53:20 e7c7073a1401 env[7206]:     return future.result()
      Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
      Jul 24 14:53:20 e7c7073a1401 env[7206]:     raise self._exception
      Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
      Jul 24 14:53:20 e7c7073a1401 env[7206]:     result = coro.send(None)
      Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 227, in runForever
      Jul 24 14:53:20 e7c7073a1401 env[7206]:     await self.runOnceNicely()
      Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 210, in runOnceNicely
      Jul 24 14:53:20 e7c7073a1401 env[7206]:     msgsProcessed = await self.prodAllOnce()
      Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 152, in prodAllOnce
      Jul 24 14:53:20 e7c7073a1401 env[7206]:     s += await n.prod(limit)
      Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/indy_node/server/node.py", line 298, in prod
      Jul 24 14:53:20 e7c7073a1401 env[7206]:     c = await super().prod(limit)
      Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/plenum/server/node.py", line 1063, in prod
      Jul 24 14:53:20 e7c7073a1401 env[7206]:     c += await self.serviceNodeMsgs(limit)
      Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/plenum/server/node.py", line 1097, in serviceNodeMsgs
      Jul 24 14:53:20 e7c7073a1401 env[7206]:     await self.processNodeInBox()
      Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/plenum/server/node.py", line 1658, in processNodeInBox
      Jul 24 14:53:20 e7c7073a1401 env[7206]:     await self.nodeMsgRouter.handle(m)
      Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/plenum/server/router.py", line 81, in handle
      Jul 24 14:53:20 e7c7073a1401 env[7206]:     res = self.handleSync(msg)
      Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/plenum/server/router.py", line 70, in handleSync
      Jul 24 14:53:20 e7c7073a1401 env[7206]:     return self.getFunc(msg[0])(*msg)
      Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/plenum/server/message_req_processor.py", line 51, in process_message_rep
      Jul 24 14:53:20 e7c7073a1401 env[7206]:     return handler.process(msg, frm)
      Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/plenum/server/message_handlers.py", line 61, in process
      Jul 24 14:53:20 e7c7073a1401 env[7206]:     valid_msg = self.create(msg.msg, **params)
      Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/plenum/server/message_handlers.py", line 76, in create
      Jul 24 14:53:20 e7c7073a1401 env[7206]:     return LedgerStatus(**msg)
      Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/plenum/common/messages/message_base.py", line 101, in __init__
      Jul 24 14:53:20 e7c7073a1401 env[7206]:     self.validate(input_as_dict)
      Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/plenum/common/messages/message_base.py", line 24, in validate
      Jul 24 14:53:20 e7c7073a1401 env[7206]:     self._validate_fields_with_schema(dct, self.schema)
      Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/plenum/common/messages/message_base.py", line 35, in _validate_fields_with_schema
      Jul 24 14:53:20 e7c7073a1401 env[7206]:     self._raise_missed_fields(*missed_required_fields)
      Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/plenum/common/messages/message_base.py", line 57, in _raise_missed_fields
      Jul 24 14:53:20 e7c7073a1401 env[7206]:     raise MissingProtocolVersionError(msg)
      Jul 24 14:53:20 e7c7073a1401 env[7206]: plenum.common.exceptions.MissingProtocolVersionError: validation error [LedgerStatus]: missed fields - protocolVersion. Make sure that the latest LibIndy is used an
      Jul 24 14:53:20 e7c7073a1401 systemd[1]: indy-node.service: Main process exited, code=exited, status=1/FAILURE
      Jul 24 14:53:20 e7c7073a1401 systemd[1]: indy-node.service: Unit entered failed state.
      Jul 24 14:53:20 e7c7073a1401 systemd[1]: indy-node.service: Failed with result 'exit-code'.
      Jul 24 14:53:30 e7c7073a1401 systemd[1]: indy-node.service: Service hold-off time over, scheduling restart.
      Jul 24 14:53:30 e7c7073a1401 systemd[1]: Stopped Indy Node.
      -- Subject: Unit indy-node.service has finished shutting down
      

        Attachments

          Activity

            People

            • Assignee:
              ozheregelya Olga Zheregelya
              Reporter:
              VladimirWork Vladimir Shishkin
              Watchers:
              Nikita Spivachuk, Olga Zheregelya, Vladimir Shishkin
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: