[Bug 1859656] Re: [2.6] Unable to reboot s390x KVM machine after initial deploy

[Frank Heimes]

In between I found the time to setup an env. build upon older releases:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.1 LTS
Release:        18.04
Codename:       bionic

$ dpkg -l | grep -i qemu
ii  qemu-block-extra:s390x                1:2.11+dfsg-1ubuntu7                  
 s390x        extra block backend modules for qemu-system and qemu-utils
ii  qemu-kvm                              1:2.11+dfsg-1ubuntu7                  
 s390x        QEMU Full virtualization on x86 hardware
ii  qemu-system-common                    1:2.11+dfsg-1ubuntu7                  
 s390x        QEMU full system emulation binaries (common files)
ii  qemu-system-s390x                     1:2.11+dfsg-1ubuntu7                  
 s390x        QEMU full system emulation binaries (s390x)
ii  qemu-utils                            1:2.11+dfsg-1ubuntu7                  
 s390x        QEMU utilities

$ apt-cache policy maas
maas:
  Installed: 2.6.0-7803-g6fc5f26eb-0ubuntu1~18.04.1
  Candidate: 2.6.0-7803-g6fc5f26eb-0ubuntu1~18.04.1
  Version table:
 *** 2.6.0-7803-g6fc5f26eb-0ubuntu1~18.04.1 500
        500 http://ppa.launchpad.net/maas-maintainers/testing/ubuntu 
bionic/main s390x Packages
        100 /var/lib/dpkg/status
     2.4.2-7034-g2f5deb8b8-0ubuntu1 500
        500 http://us.ports.ubuntu.com/ubuntu-ports bionic-updates/main s390x 
Packages
        500 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main s390x 
Packages
        500 http://aus.ports.ubuntu.com/ubuntu-ports bionic-updates/main s390x 
Packages
     2.4.0~beta2-6865-gec43e47e6-0ubuntu1 500
        500 http://us.ports.ubuntu.com/ubuntu-ports bionic/main s390x Packages
        500 http://ports.ubuntu.com/ubuntu-ports bionic/main s390x Packages

In this environment MAAS is not able to Commission ("Failed commissioning").
Trying to start the vm manually with virsh ends up with:

$ virsh start vm1 --console
Domain vm1 started
Connected to domain vm1
Escape character is ^]
done
  Using IPv4 address: 192.168.122.102
  Requesting file "boots390x.bin" via TFTP from 192.168.122.1
  Receiving data:  0 KBytesfile not found: boots390x.bin
Failed to load OS from network

So that is expecting, since the qemu packages version 1:2.11+dfsg-
1ubuntu7 were initially used - the GA version, that's not known to work
- the needed patch came later.

The first qemu packages that should be good are the ones with version 
1:2.11+dfsg-1ubuntu7.7.
But (after discussing with cpaelzer) the qemu packages didn't really changed 
since 1:2.11+dfsg-1ubuntu7.7, I thought that I now upgrade to the latest ones 
(1:2.11+dfsg-1ubuntu7.22):

$ sudo apt install qemu-block-extra qemu-kvm qemu-system-common 
qemu-system-s390x qemu-utils 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Suggested packages:
  debootstrap
Recommended packages:
  sharutils
The following packages will be upgraded:
  qemu-block-extra qemu-kvm qemu-system-common qemu-system-s390x qemu-utils
5 upgraded, 0 newly installed, 0 to remove and 167 not upgraded.
Need to get 3,249 kB of archives.
After this operation, 32.8 kB of additional disk space will be used.
Get:1 http://us.ports.ubuntu.com/ubuntu-ports bionic-proposed/main s390x 
qemu-utils s390x 1:2.11+dfsg-1ubuntu7.22 [811 kB]
Get:2 http://us.ports.ubuntu.com/ubuntu-ports bionic-proposed/main s390x 
qemu-system-common s390x 1:2.11+dfsg-1ubuntu7.22 [671 kB]
Get:3 http://us.ports.ubuntu.com/ubuntu-ports bionic-proposed/main s390x 
qemu-block-extra s390x 1:2.11+dfsg-1ubuntu7.22 [37.3 kB]
Get:4 http://us.ports.ubuntu.com/ubuntu-ports bionic-proposed/main s390x 
qemu-kvm s390x 1:2.11+dfsg-1ubuntu7.22 [12.5 kB]
Get:5 http://us.ports.ubuntu.com/ubuntu-ports bionic-proposed/main s390x 
qemu-system-s390x s390x 1:2.11+dfsg-1ubuntu7.22 [1,717 kB]
Fetched 3,249 kB in 0s (8,360 kB/s)        
(Reading database ... 67530 files and directories currently installed.)
Preparing to unpack .../qemu-utils_1%3a2.11+dfsg-1ubuntu7.22_s390x.deb ...
Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.22) over (1:2.11+dfsg-1ubuntu7) ...
Preparing to unpack .../qemu-system-common_1%3a2.11+dfsg-1ubuntu7.22_s390x.deb .
..
Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.22) over (1:2.11+dfsg-1ubuntu
7) ...
Preparing to unpack .../qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.22_s390x.deb ...
Unpacking qemu-block-extra:s390x (1:2.11+dfsg-1ubuntu7.22) over (1:2.11+dfsg-1ub
untu7) ...
Preparing to unpack .../qemu-kvm_1%3a2.11+dfsg-1ubuntu7.22_s390x.deb ...
Unpacking qemu-kvm (1:2.11+dfsg-1ubuntu7.22) over (1:2.11+dfsg-1ubuntu7) ...
Preparing to unpack .../qemu-system-s390x_1%3a2.11+dfsg-1ubuntu7.22_s390x.deb ..
.
Unpacking qemu-system-s390x (1:2.11+dfsg-1ubuntu7.22) over (1:2.11+dfsg-1ubuntu7
) ...
Setting up qemu-block-extra:s390x (1:2.11+dfsg-1ubuntu7.22) ...
Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.22) ...
Processing triggers for man-db (2.8.3-2) ...
Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.22) ...
Setting up qemu-system-s390x (1:2.11+dfsg-1ubuntu7.22) ...
Setting up qemu-kvm (1:2.11+dfsg-1ubuntu7.22) ...

And with that the Commissioning worked and ended correctly,
and I was also able to complete a Deployment afterwards.

I deployed 19.04 (disco, since I think that Sean faced the issue while
he tried to deploy disco, too) and it worked. The system (vm1) came up
and I was able to login:

$ ssh ubuntu@192.168.122.201 
The authenticity of host '192.168.122.201 (192.168.122.201)' can't be 
established.
ECDSA key fingerprint is SHA256:WMeXfn4hIAc38WUnXqPuhASMLjiig+uzdhqfkjzR7mI.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.122.201' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 19.04 (GNU/Linux 5.0.0-38-generic s390x)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Mon Feb 10 10:44:33 UTC 2020

  System load:  0.19              Processes:           95
  Usage of /:   42.4% of 7.27GB   Users logged in:     0
  Memory usage: 15%               IP address for enc1: 192.168.122.201
  Swap usage:   0%

4 updates can be installed immediately.
4 of these updates are security updates.


The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

ubuntu@vm1:~$

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1859656

Title:
  [2.6] Unable to reboot s390x KVM machine after initial deploy

Status in MAAS:
  New
Status in QEMU:
  Incomplete
Status in Ubuntu on IBM z Systems:
  Triaged

Bug description:
  MAAS version: 2.6.1 (7832-g17912cdc9-0ubuntu1~18.04.1)
  Arch: S390x

  Appears that MAAS can not find the s390x bootloader to boot from the
  disk, not sure how maas determines this.  However this was working in
  the past. I had originally thought that if the maas machine was
  deployed then it defaulted to boot from disk.

  If I force the VM to book from disk, the VM starts up as expected.

  Reproduce:

  - Deploy Disco on S390x KVM instance
  - Reboot it

  on the KVM console...

  Connected to domain s2lp6g001
  Escape character is ^]
  done
    Using IPv4 address: 10.246.75.160
    Using TFTP server: 10.246.72.3
    Bootfile name: 'boots390x.bin'
    Receiving data:  0 KBytes
    TFTP error: file not found: boots390x.bin
  Trying pxelinux.cfg files...
    Receiving data:  0 KBytes
    Receiving data:  0 KBytes
  Failed to load OS from network

  ==> /var/log/maas/rackd.log <==
  2020-01-14 18:21:24 provisioningserver.rackdservices.tftp: [info] 
boots390x.bin requested by 10.246.75.160
  2020-01-14 18:21:24 provisioningserver.rackdservices.tftp: [info] 
s390x/65a9ca43-9541-49be-b315-e2ca85936ea2 requested by 10.246.75.160
  2020-01-14 18:21:24 provisioningserver.rackdservices.tftp: [info] 
s390x/01-52-54-00-e5-d7-bb requested by 10.246.75.160
  2020-01-14 18:21:24 provisioningserver.rackdservices.tftp: [info] 
s390x/0AF64BA0 requested by 10.246.75.160
  2020-01-14 18:21:24 provisioningserver.rackdservices.tftp: [info] 
s390x/0AF64BA requested by 10.246.75.160
  2020-01-14 18:21:24 provisioningserver.rackdservices.tftp: [info] 
s390x/0AF64B requested by 10.246.75.160
  2020-01-14 18:21:24 provisioningserver.rackdservices.tftp: [info] s390x/0AF64 
requested by 10.246.75.160
  2020-01-14 18:21:24 provisioningserver.rackdservices.tftp: [info] s390x/0AF6 
requested by 10.246.75.160
  2020-01-14 18:21:24 provisioningserver.rackdservices.tftp: [info] s390x/0AF 
requested by 10.246.75.160
  2020-01-14 18:21:24 provisioningserver.rackdservices.tftp: [info] s390x/0A 
requested by 10.246.75.160
  2020-01-14 18:21:24 provisioningserver.rackdservices.tftp: [info] s390x/0 
requested by 10.246.75.160
  2020-01-14 18:21:24 provisioningserver.rackdservices.tftp: [info] 
s390x/default requested by 10.246.75.160

To manage notifications about this bug go to:
https://bugs.launchpad.net/maas/+bug/1859656/+subscriptions