Bug 1152064

Summary: Editing or marking a libvirt vm as autostart on transactional server causes /etc to unmount
Product: [openSUSE] openSUSE Distribution Reporter: William Brown <william.brown>
Component: OtherAssignee: James Fehlig <jfehlig>
Status: RESOLVED INVALID QA Contact: E-mail List <qa-bugs>
Severity: Normal    
Priority: P5 - None CC: alynx.zhou, iforster, william.brown
Version: Leap 15.1   
Target Milestone: ---   
Hardware: Other   
OS: Other   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Attachments: machine xml
strace

Description William Brown 2019-09-26 04:50:43 UTC
Created attachment 819531 [details]
machine xml

On transactional server 15.1, any action that alters /etc/libvirt via virsh causes /etc to unmount and renders the system unstable.

For example:

virsh autostart <domain>
virsh edit <domain>
virt-create ....

This is 100% reproduce-able in my environment. 

Steps:

1. install transactional server
2. install libvirt
3. do anything with virsh

/etc will unmount and can not be recovered, forcing a reboot.

For the sake of completion I have attached an example machine xml that this affected, but it affects *every* vm I have regardless of the xml content, and affects new machines with virt-create.
Comment 1 William Brown 2019-09-26 05:12:51 UTC
Created attachment 819535 [details]
strace

strace of libvirtd when using "virsh autostart osuse..." which then triggers /etc to unmount.
Comment 2 Ignaz Forster 2019-09-26 15:12:36 UTC
I can't reproduce this out of the box. Is this a system using NetworkManager? If so, could you temporarily move /etc/NetworkManager/dispatcher.d/nfs out of the way and check whether the problem persists?
Comment 3 James Fehlig 2019-10-02 15:24:23 UTC
(In reply to William Brown from comment #0)
> 1. install transactional server

Is there anything special about /etc on a transactional server? Must services avoid writing there?
Comment 4 William Brown 2019-10-02 22:17:30 UTC
I'm not sure it's possible to move that file out the way, rm/mv causes /etc to unmount before the operation completes, and after the reboot it's still there ... I'm genuinely concerned now about if it's going to be possible to resolve this via an update :|
Comment 5 William Brown 2019-10-02 22:19:04 UTC
(In reply to William Brown from comment #4)
> I'm not sure it's possible to move that file out the way, rm/mv causes /etc
> to unmount before the operation completes, and after the reboot it's still
> there ... I'm genuinely concerned now about if it's going to be possible to
> resolve this via an update :|

Booting rescue + disable network manager, then the file can be removed. This then allows the machine to be autostarted. 

So yes, this is the network manager issue. Closing.
Comment 6 Ignaz Forster 2019-10-04 07:29:37 UTC
Thanks a lot for testing and confirmation!
For reference: The NetworkManager ticket is bug 1146935.

@James: Just to answer your question in case you are interested: A transactional server is using a read-only Btrfs root file system, with certain directories / subvolumes mounted read-write. /etc is one of those writable directories.
There are indeed a few things that have to be avoided on a transactional system, the most important one is that /var should not be written directly from the package - see https://en.opensuse.org/openSUSE:Packaging_for_transactional-updates for more information.
That said I haven't found any problems with the libvirt packages during testing.
Comment 7 James Fehlig 2019-10-04 12:44:21 UTC
(In reply to Ignaz Forster from comment #6)

> @James: Just to answer your question in case you are interested: A
> transactional server is using a read-only Btrfs root file system, with
> certain directories / subvolumes mounted read-write. /etc is one of those
> writable directories.
> There are indeed a few things that have to be avoided on a transactional
> system, the most important one is that /var should not be written directly
> from the package - see
> https://en.opensuse.org/openSUSE:Packaging_for_transactional-updates for
> more information.

Nice, thanks for the info!

> That said I haven't found any problems with the libvirt packages during
> testing.

Nod, I think it is fine in this regard. I'll keep it in mind for other packages I touch.