Bug 854413

Summary: systemd-udev-settle.service takes a long time to finish due to lvm2-activation-generator
Product: [openSUSE] openSUSE 13.1 Reporter: Robert Milasan <rmilasan>
Component: BasesystemAssignee: Dong Mao Zhang <dmzhang>
Status: RESOLVED FIXED QA Contact: E-mail List <qa-bugs>
Severity: Critical    
Priority: P5 - None CC: arvidjaar, aschnell, forgotten_sM9JzehKpy, hare, lwang, maintenance, manfred.h, systemd-maintainers, werner
Version: Final   
Target Milestone: ---   
Hardware: All   
OS: openSUSE 13.1   
Whiteboard:
Found By: L3 Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---

Description Robert Milasan 2013-12-09 09:04:18 UTC
In our latest openSUSE, we have something called lvm2-activation-generator which generated 2 service file: lvm2-activation-early.service and lvm2-activation.service.

This two service file are created on any system, no matter if LVM is used or not, which is incorrect.

There are two ways at the moment to disable this:

1. mask the two service files:

 systemctl mask lvm2-activation-early.service
 systemctl mask lvm2-activation.service

2. set use_lvmetad to 1 in /etc/lvm/lvm.conf

The lvm2-activation-generator should be made a bit more intelligent and maybe to check if we do have some kind of lvm setup on the system and if we don't just quit and don't generate any service files, will pull in udevadm settle all the time.

I'll try to have a quick look on how vgscan works and maybe based on it we could fix lvm2-activation-generator too.
Comment 1 Robert Milasan 2013-12-09 09:05:58 UTC
Here is an example of the issue:

rmilasan@coolcat:~> systemd-analyze blame|head  
          1.890s systemd-udev-settle.service
          1.804s NetworkManager.service
          1.447s ModemManager.service

and if we check if 'systemd-analyze dump' we will see the both lvm2-activation-*.service file pull in systemd-udev-settle.service
Comment 2 Hannes Reinecke 2013-12-09 09:29:59 UTC
vgscan is positively _EVIL_. vgscan scans for _each and every_ device in the system, to figure out whether LVM should be activated.

This is completely wrong when running under udev; there one should have explicit rules per LVM using 'collect' to stitch the LVM volumes together.

NEVER run vgscan from udev.

Also there is lvmetad for SLE12 (there's even a fate for it IIRC) which is designed to handle these situations. So vgscan shouln't be required at all.
Comment 3 Robert Milasan 2013-12-09 09:43:13 UTC
Hannes, I wasn't referring to use vgscan, I was refering to use some kind of code similar to vgscan and patch lvm2-activation-generator to check itself if we do have VGs or lvm setup on the system and only then generate the service files.

Also, there is no udev involved in this, only systemd.
Comment 4 Dr. Werner Fink 2013-12-09 09:53:23 UTC
 linux /suse/werner> rpm -qf /usr/lib/systemd/system-generators/lvm2-activation-generator
 lvm2-2.02.98-0.28.1.5.x86_64

... maybe it would be better to *use* udev rules for this job if possible?
Comment 5 Robert Milasan 2013-12-09 10:36:30 UTC
Maybe we should use what Archlinux does:

http://lists.freedesktop.org/archives/systemd-devel/2013-December/015181.html
Comment 6 Dr. Werner Fink 2013-12-09 11:07:53 UTC
@ Dong ? What do you think about this suggestion?
Comment 7 Robert Milasan 2013-12-09 14:44:02 UTC
Actually, after doing a test install of 13.1, I've found that we do have that setup has Archlinux has, but we also do have the lvm2-activation-generator.

So I guess we can drop or we should drop lvm2-activation-generator, and we are done.
Comment 8 Hannes Reinecke 2013-12-09 17:04:12 UTC
Yes, please.
Comment 9 Dong Mao Zhang 2013-12-10 11:17:08 UTC
(In reply to comment #7)
> Actually, after doing a test install of 13.1, I've found that we do have that
> setup has Archlinux has, but we also do have the lvm2-activation-generator.
> 
> So I guess we can drop or we should drop lvm2-activation-generator, and we are
> done.

If lvm user want to use lvmetad, I think they could drop lvm2-activation-generator. 
But if someone sets lvmetad=0 in lvm.conf, and he does not have the activation-generator, Does it mean he can not activate any LVM at boot time? I have not test this.

If so, I think we should add this to the docs.
Comment 10 Hannes Reinecke 2013-12-10 11:35:20 UTC
I would require lvmetad when LVMs should be activated at boot.
Having two paths to achieve the same end is always confusing, especially in an enterprise distribution.

So please update the docs to require lvmetad whenn booting from an LVM and remove the activation-generator.
Comment 11 Dong Mao Zhang 2013-12-10 12:00:30 UTC
I prefer setting lvmetad=1 as default for the normal users, but still keep activation-generator for cluster users.

looking at this(https://bugzilla.redhat.com/show_bug.cgi?id=814779), lvmetad could not work in cluster environment.

If a lvm user who has both local and cluster LV, the lock type will be 3. lvmetad can not work when lock type is 3. we still need the generator to activate the local LV.
Comment 12 Dr. Werner Fink 2013-12-10 12:07:46 UTC
(In reply to comment #11)

How difficult would it be to let lvm2-activation-generator scan the lvm.conf for lvmetad=0 that ist that the lvm2-activation-generator does its job if required and otherwise simply does nothing?
Comment 13 Dong Mao Zhang 2014-01-06 08:56:08 UTC
(In reply to comment #12)
> (In reply to comment #11)
> 
> How difficult would it be to let lvm2-activation-generator scan the lvm.conf
> for lvmetad=0 that ist that the lvm2-activation-generator does its job if
> required and otherwise simply does nothing?

lvm2-activation-generator is surely doing in this way. 
scan lvmetad = 1, if so, it just quit. If lvmetad = 0, it will generate a new service file which may cause the bootup slow.

So setting lvmetad = 1 could resolve this.
Comment 14 Dr. Werner Fink 2014-01-07 14:38:07 UTC
(In reply to comment #13)

Hmmm ... OK, then the question rises: How the decision / Who has done the  	decision not to set `lvmetad = 1' for the reported system?

Is this done by YaST2 storage module or has the system administrator do this at his own?
Comment 15 Dong Mao Zhang 2014-01-07 16:18:50 UTC
(In reply to comment #14)
> (In reply to comment #13)
> 
> Hmmm ... OK, then the question rises: How the decision / Who has done the     
> decision not to set `lvmetad = 1' for the reported system?
> 
> Is this done by YaST2 storage module or has the system administrator do this at
> his own?

YaST2 storage can not set lvmetad. But sysadmin could choose to set lvmetad=1.

I was me who decided to make 'lvmetad = 0' because I want to put this as a new feature of LVM. If sysadmin really needs this, he could open it by himself.

But it now appears "using lvmetad" is a better choice for normal users.
Comment 16 Bernhard Wiedemann 2014-01-16 11:00:29 UTC
This is an autogenerated message for OBS integration:
This bug (854413) was mentioned in
https://build.opensuse.org/request/show/214084 Factory / lvm2
Comment 17 Forgotten User sM9JzehKpy 2014-01-22 19:53:38 UTC
Be aware that the latest changes to LVM2 (as in Base:System) causes huge issues in conjunction with grub2 (os-prober) and systemd during shutdown. In both cases the system hangs and will not continue. This could lead to corruption for configuration files, etc. 

I am not sure if the change indicated in this bug is the root cause, as that there are also other changes. But at least the main people are assigned to this bug.  See also my email to the opensuse-factory mailinglist.
Comment 18 Benjamin Brunner 2014-01-23 12:08:01 UTC
Does this issue also affect the running openSUSE 13.1 update after it includes the same fix?
Comment 19 Benjamin Brunner 2014-01-27 16:07:40 UTC
Ping?
Comment 21 Robert Milasan 2014-01-28 07:57:55 UTC
I don't know if this is still in factory. I only tested the issue in 13.1 where I actually found it.
Comment 22 Swamp Workflow Management 2014-02-03 15:35:59 UTC
openSUSE-RU-2014:0186-1: An update that has 5 recommended fixes can now be installed.

Category: recommended (moderate)
Bug References: 779965,837954,851741,854076,854413
CVE References: 
Sources used:
openSUSE 13.1 (src):    lvm2-2.02.98-0.28.5.1
Comment 23 Andrei Borzenkov 2014-02-04 13:09:51 UTC
(In reply to comment #15)
> 
> But it now appears "using lvmetad" is a better choice for normal users.

Looks like normal users have problems with it ...

https://forums.opensuse.org/showthread.php/495141-boot-problem-after-lvm2-update
Comment 24 Robert Milasan 2014-02-04 13:13:18 UTC
Yes, it's a know issue, here is the bug report: bnc#862076
Comment 25 Andrei Borzenkov 2014-02-05 17:37:31 UTC
Having lvmetad activated by default results in one cosmetic issue (bnc#862405) and one serious regression due to probing non-exiting floppy by default (bnc#862403). At least the latter should better be resolved before next version of this update is issued.
Comment 26 Andrei Borzenkov 2014-02-05 17:55:40 UTC
One more consideration - if default lvm.conf was modified, it will not be replaced by new one, so use_lvmetad will remain 0. I.e., fix won't be effective.
Comment 27 Benjamin Brunner 2014-03-03 09:31:40 UTC
Update released for openSUSE 13.1. Resolved fixed.