Bug 997637

could you attach yast logs please?
https://en.opensuse.org/openSUSE:Bugreport_YaST

Created attachment 691400 [details]
y2log after installation, before first reboot

Created attachment 691403 [details]
y2log after boot (after fixing the issue)

I've been able to reproduce. The default lvm proposal does not work.

Arvin, could you check, please?

Not sure if this is systemd or lvm2 but clearly all volumes should be activated by default.

Assigning to lvm2 maintainer. Liuhua, feel free to pass it on to e.g. systemd if it's not an lvm issue.

This works fine in sle12-sp2, btw.

*** Bug 989871 has been marked as a duplicate of this bug. ***

Created attachment 692000 [details]
screenshots of the warning moment during intallation with lvm

I also encounter the similar problem with LVM both in refreshing installation and upgrading from 42.1.

It should be some problem with dracut or systemd. 

Please try the new build:

https://openqa.opensuse.org/tests/258708/asset/iso/openSUSE-Leap-42.2-DVD-x86_64-Build0175-Media.iso

openqa test shows the refresh installation with LVM2 scheme having no problem.

*** Bug 1000722 has been marked as a duplicate of this bug. ***

Liuhua, looks like Beta2 still hast the problem. Could you check please?

Maybe also good to know that (with beta2) I (bug 1000722) did not get any warning during install. Went through as if everything would be O.K. up to the moment when you hit the problem.

Created attachment 694535 [details]
system starts up to my surprise without intervention (after changing the order of disks)

this is the journal of the boot. The pertinent date is 27.09. 
I did want to apply the corrections for the start up, in the meanwhile I had changed the hardware to a three system disk and changed the sata with the beta from the first position to the second. 
I did choose to boot from the second disk and went away to get the info on how to change the setup. I came back....and found myself on the booted beta2 dektop. 
So the change of position makes the install work. 
If you need also or deem useful yast2 log, I will provide. 
No change but the hardware position (multiple disks present during boot instead of one) had taken place.

I'll report what I'm seeing on this bug.

I have 3 systems (call them system1, system2 and system3).  I have tested all alpha/beta release of 42.2 on systems 1 and 2.  The install on system3 is new for Beta2.

On system1 and system3, I am having LVM issues.  It seems to be intermittant.  Sometimes it boots okay and sometimes it goes into emergency mode.  (If I set "use_lvmetad = 0" then the system always boots okay).

On system2, I have never had a problem.

Here is what I suspect is the main difference.  On system1 and system3, the root, swap and home all come from the encrypted LVM.  On system2, the root file system is a separate (unencrypted) partition with only swap and home coming from the encrypted LVM.

On system1 and system3, when boot fails (goes to emergency mode), I look at "/dev/mapper".  Always the root volume and the swap volume have been mapped, but nothing else is mapped.  When it boot successfully, everything is mapped.

My best guess is that there is a timing issue.  And maybe that has something to do with the switchover from using the initrd to using the root file system.

let's CC systemd and dracut maintainers. Maybe they can help here.

(In reply to Neil Rickert from comment #16)
> My best guess is that there is a timing issue.  And maybe that has something
> to do with the switchover from using the initrd to using the root file
> system.

Yes, I think so too.

The activation of LVM for ROOT and NON-ROOT partions are different:

-For ROOT partitions on LVM : activated by Dracut 
  /usr/lib/dracut/modules.d/90lvm/64-lvm.rules
  ->calls lvm_scan.sh 
  -> `vgchange -ay --sysinit VGs`

-For NON-ROOT partitions on LVM, it depends whether using lvmetad to scan disks.
 o use_lvmetad = 0 
    generater -> lvm2-activation-generation

   That is why there is no problem with lvmetad disabled.

 o use_lvmetad = 1 (default value since sle12-sp1)
    udev: 69-dm-lvm-metad.rules instantiated 
    -> * "pvscan --cache $major $minor" to scan and update lvmetad about this PV
       *  "lvm2-pvscan@$major:$minor.service"
    -> `pvscan --cache --ay $major:$minor` to activate.

  In this case need the udev status keep persistent between initrd and the later root file system. I think on system1 and system3, the home/swap unmapped problem is because partition(NON-ROOT) not activated due some reasons.


Would you please check these services running?

  systemctl status lvm2-lvmetad.service
  systemctl status lvm2-pvscan@pv_major:pv_minor.service
  systemctl status systemd-cryptsetup@home.service 

Or add to the kernel option "systemd.log_level=debug" to provide systemd debug log?

Here's requested output (requested in comment 19):

After a successful boot:

 ----
# systemctl status lvm2-lvmetad.service
● lvm2-lvmetad.service - LVM2 metadata daemon
   Loaded: loaded (/usr/lib/systemd/system/lvm2-lvmetad.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2016-09-28 09:03:49 CDT; 1min 48s ago
     Docs: man:lvmetad(8)
 Main PID: 1196 (lvmetad)
    Tasks: 1 (limit: 512)
   CGroup: /system.slice/lvm2-lvmetad.service
           └─1196 /sbin/lvmetad -f

Sep 28 09:03:49 nwr8 systemd[1]: Started LVM2 metadata daemon.

# systemctl status lvm2-pvscan@pv_major:pv_minor.service
● lvm2-pvscan@pv_major:pv_minor.service - LVM2 PV scan on device pv_major:pv_minor
   Loaded: loaded (/usr/lib/systemd/system/lvm2-pvscan@.service; static; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:pvscan(8)

# systemctl status systemd-cryptsetup@home.service
● systemd-cryptsetup@home.service
   Loaded: not-found (Reason: No such file or directory)
   Active: inactive (dead)

# systemctl status systemd-cryptsetup@xhome.service
● systemd-cryptsetup@xhome.service
   Loaded: not-found (Reason: No such file or directory)
   Active: inactive (dead)
 ----
I repeated that last one with "xhome" because I am actually mounting the home volume at "/xhome" on that system (and I then use symlinks into it from home directory).  That's to avoid problems with using the same home volume on more than one system.  I'm not sure that's checking anything, since cryptsetup would be for the LVM, rather than for the home volume.

After an unsuccessful boot (I ran these from emergency mode before doing anything else:

 ----
# systemctl status lvm2-lvmetad.service
● lvm2-lvmetad.service - LVM2 metadata daemon
   Loaded: loaded (/usr/lib/systemd/system/lvm2-lvmetad.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2016-09-28 09:09:56 CDT; 2min 9s ago
     Docs: man:lvmetad(8)
 Main PID: 1142 (lvmetad)
    Tasks: 1 (limit: 512)
   CGroup: /system.slice/lvm2-lvmetad.service
           └─1142 /sbin/lvmetad -f

Sep 28 09:09:56 nwr8 systemd[1]: Started LVM2 metadata daemon.

# systemctl status lvm2-pvscan@pv_majorr:pv_minor.service
● lvm2-pvscan@pv_majorr:pv_minor.service - LVM2 PV scan on device pv_majorr:pv_minor
   Loaded: loaded (/usr/lib/systemd/system/lvm2-pvscan@.service; static; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:pvscan(8)
 ----

I'll add that I booted 3 times this morning.  The first two boots were successful.  The third took me to emergency mode with "/xhome" not mounted.  I mention this as an indicator that the problem is intermittent.

An additional comment.  I'm not sure if this is relevant, but I will mention it.

After a failed boot, I tried to manually access the LVM volumes.  And I ran into this problem:

 ----
# vgchange -a y
  Request to list VGs in lvmetad gave response token_mismatch. Reason: lvmetad cache is invalid due to a global_filter change or due to a running rescan
 ----
I did not see that on the alpha releases of 42.2.  I have seen that on both Beta releases.

Additional information: the computer (system1 as identified in an earlier comment) has two hard drives, with an encrypted LVM on each.  Install added "crypttab" entries for both of those encrypted LVMs.  If I comment out the crypttab entry that is not actually needed, then I do not run into error above.  If I leave both entries uncomments (thus active), then I seem to always run into that problem, which leaves no way of continuing.

(In reply to Neil Rickert from comment #20)
> ● lvm2-pvscan@pv_major:pv_minor.service - LVM2 PV scan on device
 pv_major:pv_minor should be replaced by PV's major and minor number
 
> ● systemd-cryptsetup@home.service
 Sorry, I was probably wrong I am not very clear with crypt service.

> I'll add that I booted 3 times this morning.  The first two boots were
> successful.  The third took me to emergency mode with "/xhome" not mounted. 
> I mention this as an indicator that the problem is intermittent.
I see. I will reproduce locally.
Thank you!

> pv_major:pv_minor should be replaced by PV's major and minor number

Oops.  Sorry about that.

Let's try again.

First, after a successful boot:
 ----
# systemctl status lvm2-lvmetad.service
● lvm2-lvmetad.service - LVM2 metadata daemon
   Loaded: loaded (/usr/lib/systemd/system/lvm2-lvmetad.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2016-09-28 22:30:34 CDT; 3min 38s ago
     Docs: man:lvmetad(8)
 Main PID: 1104 (lvmetad)
    Tasks: 1 (limit: 512)
   CGroup: /system.slice/lvm2-lvmetad.service
           └─1104 /sbin/lvmetad -f

Sep 28 22:30:34 nwr8 systemd[1]: Started LVM2 metadata daemon.

# systemctl status lvm2-pvscan@8:21.service
● lvm2-pvscan@8:21.service - LVM2 PV scan on device 8:21
   Loaded: loaded (/usr/lib/systemd/system/lvm2-pvscan@.service; static; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:pvscan(8)
 ----

And next, from emergency mode after a failed boot:
 ----
# systemctl status lvm2-lvmetad.service
● lvm2-lvmetad.service - LVM2 metadata daemon
   Loaded: loaded (/usr/lib/systemd/system/lvm2-lvmetad.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2016-09-28 22:46:12 CDT; 2min 34s ago
     Docs: man:lvmetad(8)
 Main PID: 1162 (lvmetad)
    Tasks: 1 (limit: 512)
   CGroup: /system.slice/lvm2-lvmetad.service
           └─1162 /sbin/lvmetad -f

Sep 28 22:46:12 nwr8 systemd[1]: Started LVM2 metadata daemon.

# systemctl status lvm2-pvscan@8:21.service
● lvm2-pvscan@8:21.service - LVM2 PV scan on device 8:21
   Loaded: loaded (/usr/lib/systemd/system/lvm2-pvscan@.service; static; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:pvscan(8)
 ----

The first two boots tonight were successful.  The third went into emergency mode over the LVM problem.

Please modify the following udev rule and have a try:

modify /usr/lib/udev/rules.d/69-dm-lvm-metad.rules
  
 from : 
 
    ENV{SYSTEMD_WANTS}="lvm2-pvscan@$major:$minor.service"
 
 to 

    ENV{SYSTEMD_WANTS}+="lvm2-pvscan@$major:$minor.service"

I don't know whether it can resolve the problem but it is not right to use "=" instead of "+=" which will override SYSTEMD_WANTS.

There we go. That's the problem. lvm2 doesn't buildrequire systemd. So those lines are completely missing in Leap. Try adding BuildRequires:  pkgconfig(systemd) like it's done in the Factory package.

I realized the mistake on that "pvscan" check.  The major:minor of 8:21 refers to the encrypted container.  I should, instead, have used 254:0.  That's what gives more interesting output on my 42.1 system (which does not use lvmetad).

So I tried that, but still got about the same output as before.

Hmm, shouldn't the scan be done by "lvmetad" rather than by "systemd"?  Maybe that "systemctl" command is checking something that doesn't matter.

On the udev rules:  I'm seeing what Ludwig mentioned.  That is, there is no mention of "SYSTEMD_WANTS" in that file.  There is a rule:
RUN+="/sbin/lvm pvscan --background --cache --activate ay --major $major --minor $minor", ENV{LVM_SCANNED}="1"

This is an autogenerated message for OBS integration:
This bug (997637) was mentioned in
https://build.opensuse.org/request/show/431532 42.2 / lvm2

(In reply to Bernhard Wiedemann from comment #27)
> This is an autogenerated message for OBS integration:
> This bug (997637) was mentioned in
> https://build.opensuse.org/request/show/431532 42.2 / lvm2

Should not this be submitted as maintenance update for SLE? And preferrably not from home project?

Encrypted LVM seems to be working correctly in 42.2 Beta3.

(In reply to Ludwig Nussel from comment #25)
> There we go. That's the problem. lvm2 doesn't buildrequire systemd. So those
> lines are completely missing in Leap. Try adding BuildRequires: 
> pkgconfig(systemd) like it's done in the Factory package.

I am confused, why there is no problem with SLE12-sp2 but there is problem with Leap since they have the same spec(both have no BuildRequres: pkgconfig(systemd))?

(In reply to Neil Rickert from comment #26)
> I realized the mistake on that "pvscan" check.  The major:minor of 8:21
> refers to the encrypted container.  I should, instead, have used 254:0. 
> That's what gives more interesting output on my 42.1 system (which does not
> use lvmetad).
> 
> So I tried that, but still got about the same output as before.
> 
> Hmm, shouldn't the scan be done by "lvmetad" rather than by "systemd"? 
> Maybe that "systemctl" command is checking something that doesn't matter.

LVM needs to cooperate with other components such as iscsi, md, multipath, encrypt,etc., which depends on udev events and systemd service starting sequence. Taking LVM on top of multipath as an example: multipath set the disk READY for LVM, then LVM use lvmetad to scan on top of mutipath partitons and use multipath partitons as its physical volumes. If multipath is not READY when LVM is scanning disks then it will fail to activate them.

> On the udev rules:  I'm seeing what Ludwig mentioned.  That is, there is no
> mention of "SYSTEMD_WANTS" in that file.  There is a rule:
> RUN+="/sbin/lvm pvscan --background --cache --activate ay --major $major
> --minor $minor", ENV{LVM_SCANNED}="1"

Yes, there is where NON-ROOT partitions are activated.

In case of use_lvmetad = 1 (by default), udev: 69-dm-lvm-metad.rules will scan disks and update lvmetad through
 
    "pvscan --cache $major $minor" 

and  instantiated systemd service
 
   "lvm2-pvscan@$major:$minor.service" (which will run `pvscan --cache --ay $major:$minor` to activate LVM partitions.)

(In reply to Liuhua Wang from comment #30)
> I am confused, why there is no problem with SLE12-sp2 but there is problem
> with Leap since they have the same spec(both have no BuildRequres:
> pkgconfig(systemd))?

for the record, we did see some possibly related problems
on SUSE OpenStack Cloud7 with SP2
where we use LVM only for non-root purposes.
The system booted fine, but always hung 90 seconds when shutting down in
LVM2 PV scan on device 7:0

which is supposed to do a lvm pvscan --cache 7:0

(In reply to Liuhua Wang from comment #30)
> (In reply to Ludwig Nussel from comment #25)
> > There we go. That's the problem. lvm2 doesn't buildrequire systemd. So those
> > lines are completely missing in Leap. Try adding BuildRequires: 
> > pkgconfig(systemd) like it's done in the Factory package.
> 
> I am confused, why there is no problem with SLE12-sp2 but there is problem
> with Leap since they have the same spec(both have no BuildRequres:
> pkgconfig(systemd))?

SLE build roots always have systemd preinstalled as side effect of
some special post build check there. That is not the case for Leap
or openSUSE in general.

(In reply to Bernhard Wiedemann from comment #33)
> for the record, we did see some possibly related problems
> on SUSE OpenStack Cloud7 with SP2
> where we use LVM only for non-root purposes.
> The system booted fine, but always hung 90 seconds when shutting down in
> LVM2 PV scan on device 7:0
> 
> which is supposed to do a lvm pvscan --cache 7:0

Hi Bernhard,

Would you please update to the latest lvm2 in sle12-sp2 IBS(applied the fixes mentioned in comment#24 and #25) and see whether still have the issue?

Thank you!

(In reply to Liuhua Wang from comment #38)
> Would you please update to the latest lvm2 in sle12-sp2 IBS(applied the
> fixes mentioned in comment#24 and #25) and see whether still have the issue?

Hi, I retested with the latest version and found
that our issue is actually unrelated to this,
but caused by a faulty service file that we add for openstack-cinder,
This is now tracked in bug 1004620

SUSE-RU-2016:2590-1: An update that has 5 recommended fixes can now be installed.

Category: recommended (moderate)
Bug References: 970943,979635,980200,984321,997637
CVE References: 
Sources used:
SUSE Linux Enterprise Software Development Kit 12-SP1 (src):    lvm2-2.02.120-74.1
SUSE Linux Enterprise Server 12-SP1 (src):    lvm2-2.02.120-74.1
SUSE Linux Enterprise High Availability 12-SP1 (src):    lvm2-2.02.120-74.1
SUSE Linux Enterprise Desktop 12-SP1 (src):    lvm2-2.02.120-74.1

close this bug since it proves to work in comment#29 and the request already accepted by sle12-sp2.

openSUSE-RU-2016:2685-1: An update that has 5 recommended fixes can now be installed.

Category: recommended (moderate)
Bug References: 970943,979635,980200,984321,997637
CVE References: 
Sources used:
openSUSE Leap 42.1 (src):    lvm2-2.02.120-70.1