Bug 857271

This is the situation for my box A (the one running xen) after having performed a complete update (instead of merely patching everything) - however, updating didn't make any difference: The result remains the same.

(In reply to comment #0)
> Steps to Reproduce:
> Prerequisites: Run virt-manager over a TCP connection.

You can run it over an ssh connection too.

> Box A (the xen host) has to have xend/libvirtd running on a xen-aware kernel.
> Networking has to be taken away from xend so that it doesn't interfere with
> libvirtd, then set the network type to routed.

Why must xend be running?  In openSUSE13.1, we've moved to the new libxl toolstack.  The old xm/xend toolstack is deprecated upstream and slated for removal.  I recommend disabling/removing xend and restarting libvirtd so that it loads and uses the new libxl driver.

> Register the hd of the system as a storage pool (physical drive) - this
> requires that a free partition exists on that device!

Do you mean a libvirt disk storage pool (http://libvirt.org/storage.html#StorageBackendDisk) ?  Yes, that requires a phyiscal disk.

> Register the CD or DVD ROM as a storage pool (pre-formatted fs).

This isn't really necessary.  You should be able to select an ISO or physical cdrom in the install tool (vm-install or virt-install).

> Box B (without any virtualization) only has virt-manager running as a remote
> control for xend/libvirtd on box A (connection type xen).
> 
> 1. Fire up the VM installation wizard and follow all steps.
> 2. Choose the physical optical drive as installation media.
> 3. Select one of the empty hd partitions for the VM's mass storage

FYI, as an option you can have vm-install (or virt-install) create an image file to serve as a disk for your virtual machines.  If you do need to use libvirt storage pools, the directory or filesystem pool may be a better choice, since you won't consume hard drive partitions with each VM.

> Error message returned (from virt-manager):
> Installation konnte nicht fertiggestellt werden: «POST-Operation schlug fehl:
> xend_post: Fehler von xen-Daemon: (xend.err 'Error creating domain: Block
> device type "qemu" is invalid.')»

Can you disable xend, restart libvirtd, and try another installation?

Forgot to set needinfo for my question in #2...

(In reply to comment #2)
> [...]
> Why must xend be running?  In openSUSE13.1, we've moved to the new libxl
> toolstack.  The old xm/xend toolstack is deprecated upstream and slated for
> removal.  I recommend disabling/removing xend and restarting libvirtd so that
> it loads and uses the new libxl driver.
That somehow breaks libvirtd. See below for reasons...
>
> [...]
>
> Can you disable xend, restart libvirtd, and try another installation?
Yes - but the entire process fails when I do.

Unfortunately I cannot do anything when I take away xend - libvirtd may be starting, but it seems as if it still requires xend to be running (whenever I attempt to establish a connection to libvirtd with xend disabled, virt-manager, although it connects to the remote machine, doesn't display anything for Domain-0, and when I attempt to create a machine, this is met with two errors:

Error I:
Installation konnte nicht fertiggestellt werden: «Ende der Datei beim Lesen von Daten: Eingabe-/Ausgabefehler»

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 96, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/create.py", line 2022, in do_install
    guest.start_install(False, meter=meter)
  File "/usr/lib/python2.7/site-packages/virtinst/Guest.py", line 1251, in start_install
    noboot)
  File "/usr/lib/python2.7/site-packages/virtinst/Guest.py", line 1319, in _create_guest
    dom = self.conn.createLinux(start_xml or final_xml, 0)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2886, in createLinux
    if ret is None:raise libvirtError('virDomainCreateLinux() failed', conn=self)
libvirtError: Ende der Datei beim Lesen von Daten: Eingabe-/Ausgabefehler


***

Error II (following a moment later):
Fehler beim Abfragen der Verbindung «xen+tcp://thorin@192.168.1.1/»: Interner Fehler: Client Socket ist geschlossen

None


- Attempting to install from an unregistered optical drive causes libvirtd to complain about unknown media.
- The use of a physical partition for storage is intended. However, switching to a partition image (I have tried that for verification) would be met with success.
- Taking xend out of the equation and starting libvirtd on its own causes some things to break (looks as if that still relies on xen to work, although I have taken away libvirt's xen plugin - also uninstalling the xen driver for libvirt would cause a degradation from x86_64 to i586).
Switching to SSH yields the exact same results as TCP/SASL (when xend runs, Domain-0 is displayed, when not, Domain-0 is not to be seen and any installation attempt fails). The other drawback of SSH: When pubkey authentication is used, the mechanism fails.
Unfortunately all this does not solve the problem caused by virt-manager (i. e. qemu: is prepended to the path to the installation media)...

What libvirt and Xen packages do you have installed?  E.g. output of

# rpm -qa | egrep 'xen|libvirt'

libvirt-daemon-driver-xen-1.1.2-2.14.2.x86_64
libvirt-daemon-driver-libxl-1.1.2-2.14.2.x86_64
libvirt-daemon-xen-1.1.2-2.14.2.x86_64
xen-libs-4.3.1_02-4.4.x86_64
xen-xend-tools-4.3.1_02-4.4.x86_64
ipset-kmp-xen-6.19_k3.11.6_4-2.1.9.x86_64
libvirt-daemon-driver-nwfilter-1.1.2-2.14.2.x86_64
xen-doc-html-4.3.1_02-4.4.x86_64
xen-tools-4.3.1_02-4.4.x86_64
xen-kmp-default-4.3.0_14_k3.11.6_4-1.3.x86_64
kernel-xen-3.11.6-4.1.x86_64
libvirt-client-1.1.2-2.14.2.x86_64
libvirt-daemon-1.1.2-2.14.2.x86_64
libvirt-daemon-driver-uml-1.1.2-2.14.2.x86_64
libvirt-daemon-driver-nodedev-1.1.2-2.14.2.x86_64
libvirt-daemon-config-nwfilter-1.1.2-2.14.2.x86_64
libvirt-daemon-qemu-1.1.2-2.14.2.x86_64
xen-kmp-default-4.3.1_02_k3.11.6_4-4.4.x86_64
libvirt-doc-1.1.2-2.14.2.x86_64
libvirt-python-1.1.2-2.14.2.x86_64
libvirt-daemon-driver-storage-1.1.2-2.14.2.x86_64
libvirt-1.1.2-2.14.2.x86_64
libvirt-daemon-driver-network-1.1.2-2.14.2.x86_64
libvirt-daemon-driver-qemu-1.1.2-2.14.2.x86_64
cloop-kmp-xen-2.639_k3.11.6_4-11.1.9.x86_64
libvirt-daemon-driver-secret-1.1.2-2.14.2.x86_64
libvirt-daemon-driver-lxc-1.1.2-2.14.2.x86_64
xen-4.3.1_02-4.4.x86_64
libvirt-daemon-driver-vbox-1.1.2-2.14.2.x86_64
libvirt-daemon-driver-interface-1.1.2-2.14.2.x86_64

Created attachment 573620 [details]
XML dump fom virt-install (generated with option --print-step all)

I finally managed to generate an XML dump of the desired VM configuration, and it looks as if the definition in line 27 isn't interpreted correctly (I have tried to generate the VM by use of virsh, but it complained the same way virt-manager did).

I have tried to change the type to xen, but it got rejected with an error as well (invalid block device type "xen").

UPDATE: Eliminating the offending line from the XML file allows the device to get installed properly without throwing any exception.

I'd still like to know why your setup doesn't work without xend.  What do you mean by "Taking xend out of the equation and starting libvirtd on its own causes some things to break"?  Does libvirtd fail to start if xend is not running?

Again, xend is on the way out and probably won't even be included in 13.2, so I'm loath to debug problems with the latest virt-manager, virt-install, libvirt, etc. on top of an old xen toolstack.  We need to get to the bottom of any unknown problems in the new toolstack.  Thanks!

When I shut down xend, I am still able to fire up libvirtd (or keep an already running libvirtd up), but for some reason I am unable to deal with anything concerning virtualization (libvirtd simply refuses to create virtual machines, nor does it display any stats on existing VMs, not even dom0).

Unfortunately I haven't found a way yet to tell libvirtd to completely ignore xend. Somehow libvirtd seems to still require a connection to that daemon...

(In reply to comment #10)
> When I shut down xend, I am still able to fire up libvirtd (or keep an already
> running libvirtd up), but for some reason I am unable to deal with anything
> concerning virtualization (libvirtd simply refuses to create virtual machines,
> nor does it display any stats on existing VMs, not even dom0).

Does virsh work?  E.g. 'virsh version'?

Some background which might help:
When you start libvirtd, it will attempt load any of the installed libvirt-daemon-driver-<hypervisor> modules.  In the case of Xen, it will try libvirt-daemon-driver-xen first, which will only load if xend is running.   It then attempts to load libvirt-daemon-driver-libxl, which will only load if xend is not running.  So in theory, when libvirtd is started, only one of the Xen drivers will be loaded - all based on whether or not xend is running.

One difference between the xend and libxl toolstacks is that the former supports the notion of managed or persistent domains, i.e. xend is stateful.  On the other hand, libxl is stateless.  By necessity, this difference is propogated to libvirt, where libvirt-daemon-driver-xen is stateless (xend maintains the state) and libvirt-daemon-driver-libxl is stateful.

This might shed some light on why e.g. you would no longer see any of your non-running domains when moving from xend to libxl.  The libvirt libxl driver must now do the job of xend and maintain domain state.  But it must know about the domains first.  One way to do that is export the domain XML using the old xend stack (xend+libvirt-daemon-driver-xen) and then import the domains into libvirt after moving to the libxl stack.  E.g. with xend running

for each dom
  virsh dumpxml > dom-name.xml
stop xend
restart libvirtd
for each dom
  virsh define dom-name.xml

I am concerned about your comment of not being able to create VMs, so we'll need to get to the bottom of that.  Not seeing existing domains should be explained above.  As for dom0, it is (currently) viewed as the host from libvirt's perspective, so not shown in the domain list.  This behavior is similar to the qemu/kvm driver, along with some of the other hypervisor drivers like vmware and hyper-v.

Created attachment 573639 [details]
Debug output of libvirtd/libxenlight

Now I have tossed out anything related to xend that could possibly be uninstalled and retried generating a VM - but the attempt fails again (virsh just displays an error message whereas virt-manager loses the TCP connection).

Grepping through /var/log/messages I noticed that libxenlight throws a segmentation fault (I have included the log messages of said attempt as an attachment).

Can you install the libvirt-*-debuginfo packages, run libvirtd in gdb, and then get a backtrace when the segfault occurs?  E.g.

# gdb /usr/sbin/libvirtd
<gdb> run -l
...
segfault occurs
...
<gdb> thread apply all bt

If you don't feel comfortable with this, I can try to reproduce locally.  Does the segfault occur when trying to create a domain with configuration like you posted in #7?

Created attachment 573740 [details]
libvirtd/libxenlight backtrace

No, the data provided in #7 is not an exact match to the data I used in my creation attempts. I used the following parameters in virt-manager to do the setup:

Step 1/5:
Machine name: Arbeitsplatz
Connection type: TCP/SASL (xen)
Installation media: Local media (CD ROM or ISO image)

Step 2/5:
Installation path: CD ROM or DVD (/dev/sr0)
OS type and version: Linux / openSuSE 13.1

Step 3/5:
Available memory: 2048 MByte
vCPUs: 2

Step 4/5:
DASD storage for VM: Managed or other memory
Location: /dev/sda4 (empty partition)

Step 5/5:
Extended options:
Network type: virtual routed network (dnsmasq is not used for handling DHCP requests, instead those are directed to a local dhcpd)
Use fixed MAC address: yes
MAC address: 00:16:3e:18:93:dd
Virtualization method: Xen (fullvirt)
Architecture: x86_64

Supplemental:
- Identifier for virtual network device: veth0
- Architecture of the box running the hypervisor: x86_64

One more thing that I noticed:
Attempting to edit any mass storage produces the following error (in case "Edit configuration before installation is checked) when you click on the hw entry:

Fehler beim Neuladen der Hardwareseite: unsupported operand type(s) for /: 'NoneType' and 'int'

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/details.py", line 1307, in hw_selected
    self.refresh_disk_page()
  File "/usr/share/virt-manager/virtManager/details.py", line 2921, in refresh_disk_page
    iotune_read_bytes_sec = disk.iotune_read_bytes_sec / 1024
TypeError: unsupported operand type(s) for /: 'NoneType' and 'int'

This issue seems to converge on bug 845648 now. Could anyone please double-check that?

Thanks for the backtrace.  It appears domain create failed, which upon cleanup triggered the segfault.  The segfault has already been fixed upstream and I've backported the patch to the 13.1 libvirt package and have it queued in our 13.1 maintenance project

https://build.opensuse.org/package/show/Virtualization:openSUSE13.1/libvirt

You can install the updated packages from the following repo?

http://download.opensuse.org/repositories/Virtualization:/openSUSE13.1/openSUSE_13.1/

That should get you around the segfault so we can start looking into the failed VM creation.  Perhaps you can try using vm-install and virt-install directly instead of launching them through virt-manager.  By default, virt-manager uses virt-install.  I'd be interested in knowing if vm-install works for you.

Unfortunately there don't seem to be any patches and/or updates so far.
YaST doesn't display any updates, and zypper update complains:
Daten des Repositories laden ...
Installierte Pakete lesen ...

The following 21 package updates will NOT be installed:
  libvirt libvirt-client libvirt-daemon libvirt-daemon-config-nwfilter 
  libvirt-daemon-driver-interface libvirt-daemon-driver-libxl 
  libvirt-daemon-driver-lxc libvirt-daemon-driver-network 
  libvirt-daemon-driver-nodedev libvirt-daemon-driver-nwfilter 
  libvirt-daemon-driver-qemu libvirt-daemon-driver-secret 
  libvirt-daemon-driver-storage libvirt-daemon-driver-uml 
  libvirt-daemon-driver-vbox libvirt-daemon-driver-xen libvirt-daemon-lxc 
  libvirt-daemon-qemu libvirt-daemon-xen libvirt-doc libvirt-python 

Keine auszuführenden Aktionen.

I have added the Virtualization repository with a priority of 50 so it should take precedence over the default repos.

Currently libvirt et al. of version 1.1.2-2.14.2.x86_64 is installed.

However, once these packages become available, I'm immediately going to run the tests.

The updated packages are in

http://download.opensuse.org/repositories/Virtualization:/openSUSE13.1/openSUSE_13.1/x86_64/

but the version is lower (1.1.2-10.1.x86_64.rpm) even though they contain everything in 1.1.2-2.14.2.x86_64 + the segfault fix.  You can compare the changelogs to verify that.

AFAIK, I don't have much control over the release part of the version, i.e. the 10.1 or 2.14.2 parts in these examples.  But when I submit those packages to the openSUSE maintenance process, the version will be set correctly before the updates are released in the official update repo.

You should be able to install the packages by explicitly stating the version you want, e.g. 'zypper in libvirt-daemon-driver-libxl-1.1.2-10.1.x86_64'.

BTW, once you get beyond the segfault, it would be really helpful to identify the XML configuration which causes VM creation to fail.  Thanks.

Created attachment 573886 [details]
Tracebacks provided by virt-manager

My first attempt has been with virt-manager again (this time after switching to the fixed libvirt packages - the segfault hasn't shown up again).
On the first installation attempt it hiccups when preparing the NIC and complains that libxenlight wouldn't support network entity type "network" (unfortunately I could not find any log entries for this problem).
On the second installation attempt (log follows in another attachment) with the NIC removed the process gets past the problem with the NIC, but eventually aborts when attempting to install a mass storage device (stumbles over a type definition of "NoneType" which is what you obtain when you click on a mass storage entry in virt-manager - in that moment it complains that it couldn't load the info, because the associated type would be of "NoneType").
Attempting to install with vm-install I managed to bypass the problem with the NIC (I used a paravirtualized NIC), but again aborts when it attempts to install the mass storage.

It doesn't matter if the VM is fully virtualized or paravirtualized - the entire process stops at the exact same spot.

Created attachment 573887 [details]
Log of failed VM installation attempt (w/o NIC installation)

This is what happened from an install via virt-manager after I have removed the virtual NIC from the list of devices.

FYI, I've added some fixes for emulated NICs to the 13.1 maintenance staging project (same repo as noted in #18), but I don't think they will solve your issue.

In #13, step 5/5, you mention selecting "virtual routed network".  Was this network created via libvirt?  E.g.

  http://libvirt.org/formatnetwork.html#examplesRoute

Also, I'd like to see the domXML created by the install tool.  It should be dumped to /root/.virt-manager/virt-manager.log

Created attachment 573951 [details]
XML dump of virt-manager (domain spec only)

As far as the network definition is concerned, I used libvirt's default network config and edited it. I set the network from bridged to routed and also tossed out the <dhcp> ... </dhcp> section, because I already have a stand-alone DHCP daemon running on dom0 and dnsmasq therefore would complain (on top of that I want all DHCP activity to be handled by one entity alone).

Here's what the network definition looks like (it is intended that the data traffic can be routed to any of the physical NICs present in the box):
<network>
  <name>default</name>
  <uuid>8d9d9625-f7a8-4228-a403-629aa49595ea</uuid>
  <forward mode='route'/>
  <bridge name='veth0' stp='on' delay='0'/>
  <mac address='52:54:00:f2:98:68'/>
  <ip address='192.168.127.1' netmask='255.255.255.0'>
  </ip>
</network>

As far as the XML is concerned, I have attached it.

The SWAMPID for this issue is 56039.
This issue was rated as moderate.
Please submit fixed packages until 2014-02-11.
When done, please reassign the bug to security-team@suse.de.
Patchinfo will be handled by security team.

UPDATE:
This workaround for the networking issue worked for me:
When working with a domain XML dump (taken from a virt-manager log) I set the interface type from network to bridge (the network is set up as a bridge in the host network definition for libvirtd) and made the source configuration point to the bridge created by libvirtd.

When attempting to create the domain with virsh the network is set up just fine with these modifications - what's remaining now is the problem with the DASD.

UPDATE II:

Added a missing <driver> tag to the domain's hd definition - and all the sudden the domain was generated!

openSUSE-SU-2014:0268-1: An update that solves four vulnerabilities and has three fixes is now available.

Category: security (moderate)
Bug References: 817407,857271,857492,858817,858824,859041,859051
CVE References: CVE-2013-6457,CVE-2013-6458,CVE-2014-0028,CVE-2014-1447
Sources used:
openSUSE 13.1 (src):    libvirt-1.1.2-2.18.3

(In reply to comment #27)
> Added a missing <driver> tag to the domain's hd definition - and all the sudden
> the domain was generated!

Did you add any name or type attributes on the <driver> element?  E.g.

 <driver name='qemu' type='raw'/>

That's exactly what I have plugged into the XML definition to make things work.

I must admit, I'm confused about the status of this bug.  Are things working well for you with latest openSUSE13.1 updates?

Yes - and I think pursuing this bug has become obsolete by moving away from xend (which has caused all the problems that has been the base of its reporting).

Ok, thanks for the info.  Let's handle any new issues via new bugs :-).  Thanks!