Medium Description
Environment:
indy-node 1.6.631 -> 1.6.661 -> 1.6.662
docker pool of 10 nodes
Steps to Reproduce:
1. Setup the pool with indy-node 1.6.631.
2. Upgrade it to indy-node 1.6.661.
=> Pool successfully upgraded, following information is shown in journalctl:
Nov 07 16:01:33 69b5c1da28c9 env[83]: WARNING: apt does not have a stable CLI interface. Use with caution in scripts. Nov 07 16:05:03 69b5c1da28c9 env[83]: WARNING: apt does not have a stable CLI interface. Use with caution in scripts. Nov 07 16:06:59 69b5c1da28c9 env[83]: + deps='indy-anoncreds=1.0.32 python3-indy-crypto=0.4.5 python3-dateutil=2.6.1 indy-plenum=1.6.582 python3-timeout-decorator=0.4.0 indy-node=1.6.661' Nov 07 16:06:59 69b5c1da28c9 env[83]: + '[' -z 'indy-anoncreds=1.0.32 python3-indy-crypto=0.4.5 python3-dateutil=2.6.1 indy-plenum=1.6.582 python3-timeout-decorator=0.4.0 indy-node=1.6.661' ']' Nov 07 16:06:59 69b5c1da28c9 env[83]: + echo 'Try to donwload indy version indy-anoncreds=1.0.32 python3-indy-crypto=0.4.5 python3-dateutil=2.6.1 indy-plenum=1.6.582 python3-timeout-decorator=0.4.0 indy-node=1.6.661' Nov 07 16:06:59 69b5c1da28c9 env[83]: Try to donwload indy version indy-anoncreds=1.0.32 python3-indy-crypto=0.4.5 python3-dateutil=2.6.1 indy-plenum=1.6.582 python3-timeout-decorator=0.4.0 indy-node=1.6.661 Nov 07 16:06:59 69b5c1da28c9 env[83]: + apt-get -y update Nov 07 16:07:00 69b5c1da28c9 env[83]: Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease Nov 07 16:07:00 69b5c1da28c9 env[83]: Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease Nov 07 16:07:00 69b5c1da28c9 env[83]: Hit:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease Nov 07 16:07:00 69b5c1da28c9 env[83]: Get:4 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [107 kB] Nov 07 16:07:02 69b5c1da28c9 env[83]: Hit:5 https://repo.sovrin.org/deb xenial InRelease Nov 07 16:07:02 69b5c1da28c9 env[83]: Hit:6 https://repo.sovrin.org/sdk/deb xenial InRelease Nov 07 16:07:07 69b5c1da28c9 env[83]: Fetched 107 kB in 7s (14.9 kB/s) Nov 07 16:07:17 69b5c1da28c9 env[83]: Reading package lists... Nov 07 16:07:18 69b5c1da28c9 env[83]: + apt-get --download-only -y --allow-downgrades --allow-change-held-packages install indy-anoncreds=1.0.32 python3-indy-crypto=0.4.5 python3-dateutil=2.6.1 indy-plenum=1.6.582 python3-timeout-decorator=0.4.0 indy-node=1.6.661 Nov 07 16:07:28 69b5c1da28c9 env[83]: Reading package lists... Nov 07 16:07:30 69b5c1da28c9 env[83]: Building dependency tree... Nov 07 16:07:30 69b5c1da28c9 env[83]: Reading state information... Nov 07 16:07:32 69b5c1da28c9 env[83]: indy-anoncreds is already the newest version (1.0.32). Nov 07 16:07:32 69b5c1da28c9 env[83]: python3-dateutil is already the newest version (2.6.1). Nov 07 16:07:32 69b5c1da28c9 env[83]: python3-indy-crypto is already the newest version (0.4.5). Nov 07 16:07:32 69b5c1da28c9 env[83]: python3-timeout-decorator is already the newest version (0.4.0). Nov 07 16:07:32 69b5c1da28c9 env[83]: The following held packages will be changed: Nov 07 16:07:32 69b5c1da28c9 env[83]: indy-node indy-plenum Nov 07 16:07:32 69b5c1da28c9 env[83]: The following packages will be upgraded: Nov 07 16:07:32 69b5c1da28c9 env[83]: indy-node indy-plenum Nov 07 16:07:35 69b5c1da28c9 env[83]: 2 upgraded, 0 newly installed, 0 to remove and 15 not upgraded. Nov 07 16:07:35 69b5c1da28c9 env[83]: Need to get 1385 kB of archives. Nov 07 16:07:35 69b5c1da28c9 env[83]: After this operation, 101 kB of additional disk space will be used. Nov 07 16:07:35 69b5c1da28c9 env[83]: Get:1 https://repo.sovrin.org/deb xenial/master amd64 indy-node amd64 1.6.661 [592 kB] Nov 07 16:07:36 69b5c1da28c9 env[83]: Get:2 https://repo.sovrin.org/deb xenial/master amd64 indy-plenum amd64 1.6.582 [793 kB] Nov 07 16:07:36 69b5c1da28c9 env[83]: Fetched 1385 kB in 3s (438 kB/s) Nov 07 16:07:36 69b5c1da28c9 env[83]: Download complete and in download only mode Nov 07 16:07:36 69b5c1da28c9 env[83]: + ret=0 Nov 07 16:07:36 69b5c1da28c9 env[83]: + '[' 0 -ne 0 ']' Nov 07 16:07:36 69b5c1da28c9 env[83]: + '[' '' = '' ']' Nov 07 16:07:36 69b5c1da28c9 env[83]: + echo 'Stop indy-node' Nov 07 16:07:36 69b5c1da28c9 env[83]: Stop indy-node Nov 07 16:07:36 69b5c1da28c9 env[83]: + systemctl stop indy-node Nov 07 16:07:36 69b5c1da28c9 systemd[1]: Stopping Indy Node... Nov 07 16:07:37 69b5c1da28c9 systemd[1]: Stopped Indy Node. Nov 07 16:07:37 69b5c1da28c9 env[83]: + echo 'Run indy upgrade to indy-anoncreds=1.0.32 python3-indy-crypto=0.4.5 python3-dateutil=2.6.1 indy-plenum=1.6.582 python3-timeout-decorator=0.4.0 indy-node=1.6.661' Nov 07 16:07:37 69b5c1da28c9 env[83]: Run indy upgrade to indy-anoncreds=1.0.32 python3-indy-crypto=0.4.5 python3-dateutil=2.6.1 indy-plenum=1.6.582 python3-timeout-decorator=0.4.0 indy-node=1.6.661 Nov 07 16:07:37 69b5c1da28c9 env[83]: + apt-get -y --allow-downgrades --allow-change-held-packages --reinstall install indy-anoncreds=1.0.32 python3-indy-crypto=0.4.5 python3-dateutil=2.6.1 indy-plenum=1.6.582 python3-timeout-decorator=0.4.0 indy-node=1.6.661 Nov 07 16:07:44 69b5c1da28c9 env[83]: Reading package lists... Nov 07 16:07:46 69b5c1da28c9 env[83]: Building dependency tree... Nov 07 16:07:46 69b5c1da28c9 env[83]: Reading state information... Nov 07 16:07:47 69b5c1da28c9 env[83]: The following held packages will be changed: Nov 07 16:07:47 69b5c1da28c9 env[83]: indy-node indy-plenum Nov 07 16:07:47 69b5c1da28c9 env[83]: The following packages will be upgraded: Nov 07 16:07:47 69b5c1da28c9 env[83]: indy-node indy-plenum Nov 07 16:07:49 69b5c1da28c9 env[83]: 2 upgraded, 0 newly installed, 4 reinstalled, 0 to remove and 15 not upgraded. Nov 07 16:07:49 69b5c1da28c9 env[83]: Need to get 248 kB/1634 kB of archives. Nov 07 16:07:49 69b5c1da28c9 env[83]: After this operation, 101 kB of additional disk space will be used. Nov 07 16:07:49 69b5c1da28c9 env[83]: Get:1 https://repo.sovrin.org/deb xenial/master amd64 indy-anoncreds amd64 1.0.32 [40.9 kB] Nov 07 16:07:49 69b5c1da28c9 env[83]: Get:2 https://repo.sovrin.org/deb xenial/master amd64 python3-dateutil amd64 2.6.1 [195 kB] Nov 07 16:07:50 69b5c1da28c9 env[83]: Get:3 https://repo.sovrin.org/deb xenial/master amd64 python3-indy-crypto amd64 0.4.5 [6012 B] Nov 07 16:07:50 69b5c1da28c9 env[83]: Get:4 https://repo.sovrin.org/deb xenial/master amd64 python3-timeout-decorator amd64 0.4.0 [6698 B] Nov 07 16:07:50 69b5c1da28c9 env[83]: debconf: delaying package configuration, since apt-utils is not installed Nov 07 16:07:50 69b5c1da28c9 env[83]: Fetched 248 kB in 2s (114 kB/s) Nov 07 16:07:51 69b5c1da28c9 env[83]: [614B blob data] Nov 07 16:07:51 69b5c1da28c9 env[83]: Preparing to unpack .../indy-anoncreds_1.0.32_amd64.deb ... Nov 07 16:07:51 69b5c1da28c9 env[83]: Unpacking indy-anoncreds (1.0.32) over (1.0.32) ... Nov 07 16:07:51 69b5c1da28c9 env[83]: Preparing to unpack .../indy-node_1.6.661_amd64.deb ... Nov 07 16:07:52 69b5c1da28c9 env[83]: Unpacking indy-node (1.6.661) over (1.6.631) ... Nov 07 16:07:53 69b5c1da28c9 env[83]: Preparing to unpack .../indy-plenum_1.6.582_amd64.deb ... Nov 07 16:07:54 69b5c1da28c9 env[83]: Unpacking indy-plenum (1.6.582) over (1.6.564) ... Nov 07 16:07:57 69b5c1da28c9 env[83]: Preparing to unpack .../python3-dateutil_2.6.1_amd64.deb ... Nov 07 16:07:57 69b5c1da28c9 env[83]: Unpacking python3-dateutil (2.6.1) over (2.6.1) ... Nov 07 16:07:58 69b5c1da28c9 env[83]: Preparing to unpack .../python3-indy-crypto_0.4.5_amd64.deb ... Nov 07 16:07:58 69b5c1da28c9 env[83]: Unpacking python3-indy-crypto (0.4.5) over (0.4.5) ... Nov 07 16:07:58 69b5c1da28c9 env[83]: Preparing to unpack .../python3-timeout-decorator_0.4.0_amd64.deb ... Nov 07 16:07:58 69b5c1da28c9 env[83]: Unpacking python3-timeout-decorator (0.4.0) over (0.4.0) ... Nov 07 16:07:59 69b5c1da28c9 env[83]: Setting up indy-anoncreds (1.0.32) ... Nov 07 16:08:01 69b5c1da28c9 env[83]: Setting up python3-indy-crypto (0.4.5) ... Nov 07 16:08:01 69b5c1da28c9 env[83]: Setting up python3-dateutil (2.6.1) ... Nov 07 16:08:03 69b5c1da28c9 env[83]: Setting up indy-plenum (1.6.582) ... Nov 07 16:08:15 69b5c1da28c9 env[83]: Setting up python3-timeout-decorator (0.4.0) ... Nov 07 16:08:15 69b5c1da28c9 env[83]: Setting up indy-node (1.6.661) ... Nov 07 16:08:16 69b5c1da28c9 env[83]: /var/lib/dpkg/info/indy-node.postinst: line 132: /etc/supervisor/indy-node.conf: No such file or directory Nov 07 16:08:22 69b5c1da28c9 env[83]: + ret=0 Nov 07 16:08:22 69b5c1da28c9 env[83]: + '[' 0 -ne 0 ']' Nov 07 16:08:23 69b5c1da28c9 env[83]: + '[' '' = '' ']' Nov 07 16:08:23 69b5c1da28c9 env[83]: + systemctl daemon-reload Nov 07 16:08:23 69b5c1da28c9 systemd[1]: Reloading. Nov 07 16:08:24 69b5c1da28c9 env[83]: + systemctl reset-failed Nov 07 16:08:24 69b5c1da28c9 env[83]: + echo 'Starting indy-node' Nov 07 16:08:24 69b5c1da28c9 env[83]: Starting indy-node Nov 07 16:08:24 69b5c1da28c9 env[83]: + systemctl start indy-node Nov 07 16:08:24 69b5c1da28c9 systemd[1]: Started Indy Node. Nov 07 16:08:24 69b5c1da28c9 env[83]: + echo 'Restarting agent' Nov 07 16:08:24 69b5c1da28c9 env[83]: Restarting agent Nov 07 16:08:24 69b5c1da28c9 env[83]: + systemctl restart indy-node-control Nov 07 16:08:24 69b5c1da28c9 systemd[1]: Stopping Indy Node... Nov 07 16:08:24 69b5c1da28c9 systemd[1]: Stopping Service for upgrade of existing Indy Node and other operations... Nov 07 16:08:24 69b5c1da28c9 systemd[1]: Stopped Service for upgrade of existing Indy Node and other operations. Nov 07 16:08:24 69b5c1da28c9 systemd[1]: Stopped Indy Node. Nov 07 16:08:24 69b5c1da28c9 systemd[1]: Started Indy Node. Nov 07 16:08:24 69b5c1da28c9 systemd[1]: Started Service for upgrade of existing Indy Node and other operations.
3. On part of the nodes change sovrin repo from master to RC to get next upgrade failed.
4. Upgrade indy-node to 1.6.662 version.
Actual Results:
Upgrade completed on part of the nodes.
In journalctl of successfully upgraded nodes:
Nov 08 13:22:08 21a0845c5ffc systemd[1]: Stopping Indy Node... Nov 08 13:22:09 21a0845c5ffc systemd[1]: Stopped Indy Node. Nov 08 13:22:28 21a0845c5ffc systemd[1]: Reloading. Nov 08 13:22:42 21a0845c5ffc systemd[1]: Reloading. Nov 08 13:22:44 21a0845c5ffc systemd[1]: Reloading. Nov 08 13:23:06 21a0845c5ffc systemd[1]: Reloading. Nov 08 13:23:07 21a0845c5ffc systemd[1]: Started Indy Node. Nov 08 13:23:07 21a0845c5ffc systemd[1]: Stopping Service for upgrade of existing Indy Node and other operations... Nov 08 13:23:07 21a0845c5ffc systemd[1]: Stopping Indy Node... Nov 08 13:23:07 21a0845c5ffc systemd[1]: Stopped Indy Node. Nov 08 13:23:07 21a0845c5ffc systemd[1]: Stopped Service for upgrade of existing Indy Node and other operations. Nov 08 13:23:07 21a0845c5ffc systemd[1]: Started Service for upgrade of existing Indy Node and other operations. Nov 08 13:23:07 21a0845c5ffc systemd[1]: Started Indy Node.
In journalctl of failed nodes:
Nov 08 13:21:31 5088f23c2c74 env[1606]: Upgrade from 1.6.661 to 1.6.662 failed: command upgrade_indy_node "indy-node=1.6.662" returned 1 Nov 08 13:21:31 5088f23c2c74 env[1606]: Trying to rollback to the previous version command upgrade_indy_node "indy-node=1.6.662" returned 1 Nov 08 13:22:51 5088f23c2c74 systemd[1]: Stopping Indy Node... Nov 08 13:22:52 5088f23c2c74 systemd[1]: Stopped Indy Node. Nov 08 13:23:16 5088f23c2c74 systemd[1]: Reloading. Nov 08 13:23:19 5088f23c2c74 systemd[1]: Reloading. Nov 08 13:23:21 5088f23c2c74 systemd[1]: Reloading. Nov 08 13:23:22 5088f23c2c74 systemd[1]: Reloading. Nov 08 13:23:23 5088f23c2c74 systemd[1]: Started Indy Node. Nov 08 13:23:23 5088f23c2c74 systemd[1]: Stopping Indy Node... Nov 08 13:23:23 5088f23c2c74 systemd[1]: Stopping Service for upgrade of existing Indy Node and other operations... Nov 08 13:23:23 5088f23c2c74 systemd[1]: Stopped Indy Node. Nov 08 13:23:23 5088f23c2c74 systemd[1]: Stopped Service for upgrade of existing Indy Node and other operations. Nov 08 13:23:23 5088f23c2c74 systemd[1]: Started Service for upgrade of existing Indy Node and other operations. Nov 08 13:23:23 5088f23c2c74 systemd[1]: Started Indy Node.
Expected Results:
Need to specify reason of upgrade failure in journalctl in case of failed upgrade and to put more information about successful upgrade.