Bug 612006

I have tried Traditional Method with ifup.

Removing vif from the config file has resulted in another complaint... this is similar to other threads I've read circa 2006 on xen configuration on opensuse.

Eg:
Error: Device 768 (vbd) could not be connected. Hotplug scripts not working.

There seems to be some chatter over the past few years about these problems and the MAKEDEV of various /dev/ mount points.  I have no idea what these are presently nor how they are used.

(http://mail-index.netbsd.org/port-xen/2009/06/12/msg005064.html
and
http://old.nabble.com/second-domU:-Error:-Device-768-%28vbd%29-could-not-be-connected.-Hotplug--scripts-not-working-td25175485.html
)

Additional info/debugging here:
http://lists.xensource.com/archives/html/xen-users/2010-06/msg00240.html

Also found this thread which may be similar but not sure:
http://lists.xensource.com/archives/html/xen-users/2010-06/msg00000.html

Looking at comments in posts under Comment 3 above again I found once installing MAKEDEV it gives these errors:

linux-oit2:/dev # /sbin/MAKEDEV vnd4
/sbin/MAKEDEV: line 170: major_/dev/vc/0=4: No such file or directory
/sbin/MAKEDEV: line 170: major_/dev/tty=5: No such file or directory
/sbin/MAKEDEV: line 170: major_/dev/console=5: No such file or directory
/sbin/MAKEDEV: line 170: major_/dev/ptmx=5: No such file or directory
/sbin/MAKEDEV: line 170: major_sound/mixer=14: No such file or directory
/sbin/MAKEDEV: line 170: major_sound/dsp=14: No such file or directory
/sbin/MAKEDEV: line 170: major_sound/audio=14: No such file or directory
/sbin/MAKEDEV: line 170: major_sound/mixer1=14: No such file or directory
/sbin/MAKEDEV: line 170: major_cpu/msr=202: No such file or directory
/sbin/MAKEDEV: line 170: major_device-mapper=253: command not found
/sbin/MAKEDEV: don't know how to make device "vnd4"

There are no vnd* devs on this opensuse installation.

I've noticed there is an Oops everytime I start the VM.

XEN starts up and xend-debug has:
linux-oit2:/var/log/xen # tail -f /var/log/xen/xend-debug.log 
Xend started at Tue Jun  8 09:18:39 2010.
/usr/lib64/python2.6/site-packages/xen/xend/XendAPI.py:551: DeprecationWarning: object.__new__() takes no parameters
  return object.__new__(cls, *args, **kwds)





Then when I issue xm create <VM name> I see:

linux-oit2:/etc/xen/vm # xm create WinXP_sp3.hvm 
Using config file "./WinXP_sp3.hvm".

Message from syslogd@linux-oit2 at Jun  8 09:27:16 ...
 kernel:[  693.223052] Oops: 0000 [#1] SMP 

Message from syslogd@linux-oit2 at Jun  8 09:27:16 ...
 kernel:[  693.223061] last sysfs file: /sys/devices/xen-backend/vif-1-0/uevent

Message from syslogd@linux-oit2 at Jun  8 09:27:16 ...
 kernel:[  693.223413] Stack:

Message from syslogd@linux-oit2 at Jun  8 09:27:16 ...
 kernel:[  693.223454] Call Trace:

Message from syslogd@linux-oit2 at Jun  8 09:27:16 ...
 kernel:[  693.223620] Code: 48 3d 00 f0 ff ff 49 89 c5 0f 87 87 00 00 00 48 89 c2 48 c7 c6 e5 cf 88 a0 48 89 ef 31 c0 e8 45 41 99 df 4c 89 ef e8 4d c4 87 df <49> 8b 94 24 50 01 00 00 48 89 ef 48 c7 c6 ef cf 88 a0 31 c0 31 

Message from syslogd@linux-oit2 at Jun  8 09:27:16 ...
 kernel:[  693.223758] CR2: 0000000000000150
Started domain WinXP-SP3 (id=1)

Attached is the dmesg output with the OOPs.

[  693.222999] BUG: unable to handle kernel NULL pointer dereference at 0000000000000150
[  693.223015] IP: [<ffffffffa0889a63>] netback_uevent+0x73/0x100 [netbk]
[  693.223038] PGD 189cef067 PUD 189d30067 PMD 0 
[  693.223052] Oops: 0000 [#1] SMP 
[  693.223061] last sysfs file: /sys/devices/xen-backend/vif-1-0/uevent
[  693.223068] CPU 2 
...

Created attachment 367827 [details]
NULL ptr deref in XEND

NULL ptr work in progress: http://lists.xensource.com/archives/html/xen-devel/2010-05/msg01590.html

FWIW if I comment out the "vif =" line this oops goes away, but the hotscripts plugin still persists.  No dmesg/oops/xend-debug output this time ;o(

I have the very same problem. Running opensuse 11.3m7 as well. Everything you describe is exactly the same for me.

(In reply to comment #10)
> I have the very same problem. Running opensuse 11.3m7 as well. Everything you
> describe is exactly the same for me.

Thanks for the comment Thomas.  Unfortunately I've had very little input from xen-users.  Most revolves around networking troubleshooting but I dont think this is the problem.  I've not had time to inspect hotplugin errors.  Instead, I'm waiting on suse 11.3 RC1 in a few days.  I think others have had similar issues manifesting in different ways reading about XEN4.0 on the xen-users group.  I dont think anyone has read this bugzilla report from suse which is a shame given its the only distro that even came close to running a dom0 on W510.  I hope it magically fixes itself by July otherwise I'm going to have to look to other virtualisation solutions.  If you come across anything please update this thread. ;o)  I had contemplated trying XEN3.4 in opensuse but I'm not familiar with its package-manager to try this... I had tried different ones on ubuntu but there I had difficulty with driver support for W510 (graphics) and never got as far as launching a domU.

11.3 RC1 
- dom0 launching a domU has exactly the same behaviour with fresh install (hotplugin scripts not working).
- xm create still has a crash wrt vif configuration
- general networking manager seems more stable and automatic

Giving up.  Moving to some other virtualisation technology on Ubuntu.

I have this issue too on openSUSE factory with xen-4.0.0_21091_05-4.1

I can start network-enabled VM only once after reboot host system, next tryies to launch vm gives me 
"Error: Device 0 (vif) could not be connected. Hotplug scripts not working."

(In reply to comment #13)
> I have this issue too on openSUSE factory with xen-4.0.0_21091_05-4.1
> 
> I can start network-enabled VM only once after reboot host system, next tryies
> to launch vm gives me 
> "Error: Device 0 (vif) could not be connected. Hotplug scripts not working."

The vif hotplug scripts not working is the first error I encountered.  If you remove the VIF statement, which isn't very useful I agree, you end up with another hotplug not working error on the disk missing.  I may have the disk wrongly specified or options missing?

I also found this fix: http://xenbits.xen.org/xen-4.0-testing.hg?rev/72b9c78aef6b which is about spaces in the file-name to the disk image.  This isn't related to this problem but is related to hotplug scripts not working (in a different senario or maybe the same underlying issue).

The problem it's related to kernel version 2.6.34-12.3 (same problem on sles11sp1 there is a ptf now).

Try the latest kernel from kotd (kernel of the day) could help.

Is there a workaround for this issue? Or will have to downgrade to 11.2 ?:(

I found this x-ref http://lists.xensource.com/archives/html/xen-users/2010-07/msg00547.html to this bug report via my xen-users post http://lists.xensource.com/archives/html/xen-users/2010-06/msg00307.html.

I am not sure what relevance RHEL vs Ubuntu has in Fajar & Boris' conversation (perhaps Ubuntu == OpenSUSE) but it seems/implies the kernel should have statically linked in "backend Dom0 drivers" instead of loadable kernel modules.

Vittorio: do you know if the released version of opensuse 11.3 is any better.  I'm guessing not as I believe its using 2.6.34-x.

I can't downgrade to 11.2 without loosing driver/kernel support for W510 laptop.  I would really appreciate this working.  I have made some progress in getting XEN4.0 dom0 running in ubuntu environment.  I couldn't get the xend or xenfs to come up though.

This update from Boris seems to imply opensuse11.3 has a problem with hot-scripts-not-working due to a udev issue:

-----
Re: [Xen-users] How to patch 2.6.34-12-xen on Suse 11.3 ? ( udev 157 issue)
Thursday, July 29, 2010 9:41 AM
From:
"Boris Derzhavets" <bderzhavets@yahoo.com>
Add sender to Contacts
To:
xen-users@lists.xensource.com, JBeulich@novell.com

ftp://ftp.suse.com/pub/projects/kernel/kotd/openSUSE-11.3/x86_64/

Download kernel-xen-2.6.34.1-0.0.17.fdda44d.x86_64.rpm 
      34984 KB  07/28/2010   07:03:00 PM
and install provides kernel working with udev 157

Boris.

--- On Wed, 7/28/10, Boris Derzhavets <bderzhavets@yahoo.com> wrote:
    From: Boris Derzhavets <bderzhavets@yahoo.com>
    Subject: [Xen-users] How to patch 2.6.34-12-xen on Suse 11.3 ? ( udev 157 issue)
    To: xen-users@lists.xensource.com, JBeulich@novell.com
    Date: Wednesday, July 28, 2010, 4:06 PM

    > >>> On 02.07.10 at 13:14, PeanuTech <peanutdude <at> gmail.com> wrote:
    > > Hi,
    > >  I'm using OpenSuse 11.3 - with the latest updates from the repository. I
    > >  installed a guest PV OpenSuse 11.3 - RC1, and the virtual networking doesn't
    > >  work.
    > >  The virt-manager application isn't managing the virtual network so I can't
    > >  use that.
    > >  My router's log confirms that a dhcp address was given to the guest.
    > >  So I don't understand what's the problem.
    >
    > This likely is caused by an already fixed bug in netback caused by a
    > udev upgrade done very close before RC1.
    >
    > Jan
    >
    >
    I see this bug in 11.3 final with udev 157, e.g. attempt to install 11.3 PV DomU  at 11.3 Dom0 ("Hotplug scripts not working"). If patch exists and 2.6.34-12-xen may be patched, how to proceed ?

    Boris.
    P.S Downgrade to 151 works for PV Guests (for me), but not for HVM.
-----

I am not a Suse's expert, but Jan Beulich will be out of office until 08/05/2010. Looks like kernel at "kotd" page, i mentioned above ( ftp://ftp.suse.com/pub/projects/kernel/kotd/openSUSE-11.3/x86_64/ ) , has Jan's patch already applied and works with udev 157 on 11.3. This page seems to be updated on daily basis. I just  downloaded current xen-kernel. Installed  Xen host from DVD and installed new kernel :-
 # rpm -ivh kernel-xen-2.6.34.1-0.0.17.fdda44d.x86_64.rpm
Grub entry would be created automatically. Reloaded system works fine:-
  1. virt-install to create PV or HVM DomU
  2. virt-manager  to manage PV or HVM DomU
My limited experience is absolutly positive. No problems with HVM domus as i got
after udev downgrade 157->151.
However, I cannot issue any warranty until Jan will confirm , that kernel is OK with udev 157. In meantime i would test it in QA or testing environment.

Boris. 07/29/10

Thanks Boris for this update, I will try this out within the next week.  Sounds very promising.

I have opensuse 11.3 with kotd kernel now installed on my W510 laptop.  I am trying to recompile the xen4-testing.hg source to try out 4.0.1rc5.  

I've added the c/c++ patterns and dev86 packages.  Then "make xen" works.  Calling "make tools" bombs out in two places:
a) can't find: stub-32.h header file at /usr/src/gnu/
b) can't find: /usr/include/bits/predef.h

Can anyone point me to a set of instructions for opensuse 11.3 build of XEN source?  The XEN 4.0 source wiki instructions worked fine with Ubuntu.  Not the case with opensuse. ;o(

Continuing with XEN 4.0 in OpenSUSE 11.3 and kotd (Linux linux-z7hf.site 2.6.34.1-0.0.18.cbab77c-xen #1 SMP 2010-07-29 14:00:26 +0200 x86_64 x86_64 x86_64 GNU/Linux) I have found the following error when I attempt to connect to the system using the XEN VirtManager:

Unable to open connection to hypervisor URI 'xen:///':
unable to connect to 'localhost:8000': Connection refused
Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/connection.py", line 971, in _try_open
    None], flags)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 111, in openAuth
    if ret is None:raise libvirtError('virConnectOpenAuth() failed')
libvirtError: unable to connect to 'localhost:8000': Connection refused

The above message repeats many times on the console, then a box pops up.

When I attempt to create a VM I get a pop-up-box wiht "Error: None" in it and nothing more.

Forgot to add, I also see this when I attempt to start xend (manually):

mcook@linux-z7hf:/etc/xen> sudo /etc/init.d/xend start
Traceback (most recent call last):
  File "/usr/sbin/xend", line 36, in <module>
    from xen.xend.server import SrvDaemon
  File "/usr/lib64/python2.6/site-packages/xen/xend/server/SrvDaemon.py", line 20, in <module>
    import xen.lowlevel.xc
ImportError: libbz2.so.1.0: cannot open shared object file: No such file or directory
...
...

This is repeated several times for 20-30secs then stops displaying on cmd line.

Tried disabling "Firewall" under YAST and made no difference to connection error above.

(In reply to comment #23)
> Continuing with XEN 4.0 in OpenSUSE 11.3 and kotd (Linux linux-z7hf.site
> 2.6.34.1-0.0.18.cbab77c-xen #1 SMP 2010-07-29 14:00:26 +0200 x86_64 x86_64
> x86_64 GNU/Linux) I have found the following error when I attempt to connect to
> the system using the XEN VirtManager:
> 
> Unable to open connection to hypervisor URI 'xen:///':
> unable to connect to 'localhost:8000': Connection refused
> Traceback (most recent call last):
>   File "/usr/share/virt-manager/virtManager/connection.py", line 971, in
> _try_open
>     None], flags)
>   File "/usr/lib64/python2.6/site-packages/libvirt.py", line 111, in openAuth
>     if ret is None:raise libvirtError('virConnectOpenAuth() failed')
> libvirtError: unable to connect to 'localhost:8000': Connection refused
> 
> The above message repeats many times on the console, then a box pops up.
> 
> When I attempt to create a VM I get a pop-up-box wiht "Error: None" in it and
> nothing more.

Comment out (xend-unix-server yes) in /etc/xen/xend-config.sxp and export variable
export VIRSH_DEFAULT_CONNECT_URI="xen:///" in root’s .bashrc. Then reboot Xen Host.

I actually installed xen-server pattern and this problem went away.  Things seem to be working ok, I can boot HVM and PVs up and they will appear in the VirtManager.  Unlike earlier XEN3.x I dont see their defn's in VirtualManager panel until they are running.  I can successfully connect using the console/vnc session.  

I'm now onto the networking part which YAST, NetworkManager and XEN are all fighting over.

I removed Networkmanager from the picture and have YAST configuration (which I'm learning to use) to manually configure the NW interface behaviours.  I also 're-installed' XEN Tools which created some default bridges.

I have br0 and br1 however, under opensuse11.3 YAST configuration panel I consistently see that I have selected:

br0 -> eth0 and wlan0
br1 -> eth1

eth1 is DLINK DUB100 adapter with static IP address
eth0 is DHCP and wlan is connecting to the wifi.

However, using ifconfig I see:
 - br1 is mapped to eth0 with the IP addr I'd expect eth0 to get
 - no sign of br0

It seems to be a disconnect between YAST configuration (and /etc/sysconfig/network/ scripts compared to reality.

I have commented out the xend-config.sxp network-script lines and tried to read the OpenSUSE notes from XEN README.

YAST Network Panel keeps asking to install smpppd but will also uninstall xen-kernel.  So I keep ignoring this but maybe its related to my problems but I can't see why.

Anyone know why br0 is non-existent and br1 is mapped to eth0 despite what I ask YAST to do (which is what the README suggests I use)?

>I dont see their defn's in VirtualManager
> panel until they are running.  I can successfully connect using the console/vnc
> session.  

Perform guest install via virt-install.

What's wrong with default bridge "br0" installed along with Xen package ?

I couldn't find br0 despite the YAST/Opensuse Hypervisor tools install saying it was going to create it.  When I reference br0 in the guest config file it says it doesn't exist.  It seems quite happy with br1 but even with this specified I seem to have no connectivity from guest to the outside world.

So I uninstalled all xen-related packages booted into a non-dom0 opensuse kernel and reinstalled XEN hypervisor from YAST.  Then followed your blog steps stepping around the domU tools package problems.

I also found that smpppd package doesn't like hte xen kernel and haven't managed to install it (with deps still intact).

I found the hypervisor bridge-configuration step this time seems to have mapped eth0 to br1.  On starting my ubuntu HVM it presents eth4 which now has bridged access to the LAN.  Yay!  I am assuming br1 was chosen because the network configuration in hypervisor found several potential network interfaces.  It created a br2, although this is presumably for the wifi link.  I have not touched YAST Network configuration tool since but it seemed easily confused or just broken.  Perhaps XEN based nw configuration is required to link up guests with host interfaces.

A remaining puzzle is why the guest screen is 400x320 and where I may be able to configure this to something larger?  Or should I just run a vncserver within guest?

I will also continue to work wiht multiple nw interfaces as one of the guests needs two NIC interfaces.  Thanks for your suggestions and help though Boris!

(In reply to comment #30)
> So I uninstalled all xen-related packages booted into a non-dom0 opensuse
> kernel and reinstalled XEN hypervisor from YAST.  Then followed your blog steps
> stepping around the domU tools package problems.
You shouldn't do that . Just once installed Xen is OK for you, unless you
you want to add some new patch and in this case follow procedure like was
run for zfs24.patch.

I noticed that the networking in YAST Network manager panel now says br0 is mapped to wlan0, br1 to eth0 and br2 to eth1 (DLINK adapter).  I think this is 'as expected' and I have all HVM and PV guests up with network connectivity.  Will continue to monitor stability but this finally seems to be functional.  

Screen updates & networking is rather slow but hey, you'd think I'd be happy! ;o)

Remaining headache: setting the vnc screen size of XEN Ubuntu guest... it seems to default to 800x600.  Running a VNC server within the guest works but requires you to know the IP addr of the guest to connect.  Any ideas?

Related KOTD thread: https://bugzilla.novell.com/show_bug.cgi?id=623286#c12
And http://lists.xensource.com/archives/html/xen-users/2010-08/msg00478.html

I found a fresh install of 11.3 with KOTD solved the hot-scripts plugin and 'restored' by XEN networking headaches back to the default bridge.  YAST Network Services and XEN do not seem to play nicely together and manually attmepting to provision the necessary bridges etc seemed very unstable.

First of all, please attach yast logs from bridge setup

Hi Michael, we can only fix yast2 if we are provided by what happens on the machine - yast2 logs. You can gather them by using this document ( http://en.opensuse.org/openSUSE:Report_a_YaST_bug ). Thanks.

I close it because no response ,please reopen it if you can provide more information about this issue,thank you .