Uploaded image for project: 'Indy Node'
  1. Indy Node
  2. INDY-1519

1.3.62 -> 1.5.67 forced upgrade without one node in schedule was failed

    XMLWordPrintable

Details

    • Bug
    • Status: Complete
    • Highest
    • Resolution: Done
    • None
    • 1.5
    • None
    • None
    • Unset
    • EV 18.15 Stability/Availabilit

    Description

      Build Info:
      indy-node 1.3.62 -> 1.5.67

      Steps to Reproduce:
      1. Install pool of 7 nodes.
      2. Schedule forced upgrade with the 1st node missing via old CLI on future time:

      send POOL_UPGRADE name=upgrade67new version=1.5.67 sha256=ed0a366b4ef36d40c055672a8b83679e99246fec71a706b4ae4cb7958feace3f action=start schedule={'8ECVSk179mjsjKRLWiQtssMLgp6EPhWXtaYyStWPSGAb':'2018-07-24T15:00:00.258870+00:00','DKVxG2fXXTU8yT5N7hGEbXB3dfdAnYv1JczDUHpmDxya':'2018-07-24T15:00:00.258870+00:00','4PS3EDQ3dW1tci1Bp6543CfuuebjFrg36kLAUcskGfaA':'2018-07-24T15:00:00.258870+00:00','4SWokCJWJc69Tn74VvLS6t2G2ucvXqM9FDMsWJjmsUxe':'2018-07-24T15:00:00.258870+00:00','Cv1Ehj43DDM5ttNBmC6VPpEfwXWwfGktHwjDJsTV5Fz8':'2018-07-24T15:00:00.258870+00:00','BM8dTooz5uykCbYSAAFwKNkYfT4koomBHsSWHTDtkjhW':'2018-07-24T15:00:00.258870+00:00'} timeout=10 force=True

      Actual Results:
      1. Upgrade was triggered right after scheduling (not in time).
      2. 2nd..7th nodes were upgraded but node service was failed periodically with this error:

      Jul 24 14:53:20 e7c7073a1401 env[7206]: Traceback (most recent call last):
Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/bin/start_indy_node", line 19, in <module>
Jul 24 14:53:20 e7c7073a1401 env[7206]:     client_ip=sys.argv[4], client_port=int(sys.argv[5]))
Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/indy_node/utils/node_runner.py", line 54, in run_node
Jul 24 14:53:20 e7c7073a1401 env[7206]:     looper.run()
Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 263, in run
Jul 24 14:53:20 e7c7073a1401 env[7206]:     return self.loop.run_until_complete(what)
Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
Jul 24 14:53:20 e7c7073a1401 env[7206]:     return future.result()
Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
Jul 24 14:53:20 e7c7073a1401 env[7206]:     raise self._exception
Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
Jul 24 14:53:20 e7c7073a1401 env[7206]:     result = coro.send(None)
Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 227, in runForever
Jul 24 14:53:20 e7c7073a1401 env[7206]:     await self.runOnceNicely()
Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 210, in runOnceNicely
Jul 24 14:53:20 e7c7073a1401 env[7206]:     msgsProcessed = await self.prodAllOnce()
Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/stp_core/loop/looper.py", line 152, in prodAllOnce
Jul 24 14:53:20 e7c7073a1401 env[7206]:     s += await n.prod(limit)
Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/indy_node/server/node.py", line 298, in prod
Jul 24 14:53:20 e7c7073a1401 env[7206]:     c = await super().prod(limit)
Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/plenum/server/node.py", line 1063, in prod
Jul 24 14:53:20 e7c7073a1401 env[7206]:     c += await self.serviceNodeMsgs(limit)
Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/plenum/server/node.py", line 1097, in serviceNodeMsgs
Jul 24 14:53:20 e7c7073a1401 env[7206]:     await self.processNodeInBox()
Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/plenum/server/node.py", line 1658, in processNodeInBox
Jul 24 14:53:20 e7c7073a1401 env[7206]:     await self.nodeMsgRouter.handle(m)
Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/plenum/server/router.py", line 81, in handle
Jul 24 14:53:20 e7c7073a1401 env[7206]:     res = self.handleSync(msg)
Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/plenum/server/router.py", line 70, in handleSync
Jul 24 14:53:20 e7c7073a1401 env[7206]:     return self.getFunc(msg[0])(*msg)
Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/plenum/server/message_req_processor.py", line 51, in process_message_rep
Jul 24 14:53:20 e7c7073a1401 env[7206]:     return handler.process(msg, frm)
Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/plenum/server/message_handlers.py", line 61, in process
Jul 24 14:53:20 e7c7073a1401 env[7206]:     valid_msg = self.create(msg.msg, **params)
Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/plenum/server/message_handlers.py", line 76, in create
Jul 24 14:53:20 e7c7073a1401 env[7206]:     return LedgerStatus(**msg)
Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/plenum/common/messages/message_base.py", line 101, in __init__
Jul 24 14:53:20 e7c7073a1401 env[7206]:     self.validate(input_as_dict)
Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/plenum/common/messages/message_base.py", line 24, in validate
Jul 24 14:53:20 e7c7073a1401 env[7206]:     self._validate_fields_with_schema(dct, self.schema)
Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/plenum/common/messages/message_base.py", line 35, in _validate_fields_with_schema
Jul 24 14:53:20 e7c7073a1401 env[7206]:     self._raise_missed_fields(*missed_required_fields)
Jul 24 14:53:20 e7c7073a1401 env[7206]:   File "/usr/local/lib/python3.5/dist-packages/plenum/common/messages/message_base.py", line 57, in _raise_missed_fields
Jul 24 14:53:20 e7c7073a1401 env[7206]:     raise MissingProtocolVersionError(msg)
Jul 24 14:53:20 e7c7073a1401 env[7206]: plenum.common.exceptions.MissingProtocolVersionError: validation error [LedgerStatus]: missed fields - protocolVersion. Make sure that the latest LibIndy is used an
Jul 24 14:53:20 e7c7073a1401 systemd[1]: indy-node.service: Main process exited, code=exited, status=1/FAILURE
Jul 24 14:53:20 e7c7073a1401 systemd[1]: indy-node.service: Unit entered failed state.
Jul 24 14:53:20 e7c7073a1401 systemd[1]: indy-node.service: Failed with result 'exit-code'.
Jul 24 14:53:30 e7c7073a1401 systemd[1]: indy-node.service: Service hold-off time over, scheduling restart.
Jul 24 14:53:30 e7c7073a1401 systemd[1]: Stopped Indy Node.
-- Subject: Unit indy-node.service has finished shutting down

      Attachments

        Activity

          People

            ozheregelya Olga Zheregelya
            VladimirWork Vladimir Shishkin
            Nikita Spivachuk, Olga Zheregelya, Vladimir Shishkin
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: