Bug 807288

However, the file does in fact exists when phase 1 is completed:

# ls -l ../../var/lib/YaST2/runme_at_boot
-rw-r--r-- 1 root root 0 Mar  4 16:31 ../../var/lib/YaST2/runme_at_boot

Is the 2nd phase being run by someone/something else?

The file is being removed by

/usr/lib/systemd/system/YaST2-Second-Stage.service

(In reply to comment #0)
> When installing 12.3rc2 over ssh, I receive the following when I try to
> initiate phase2:
> 
> # /usr/lib/YaST2/startup/YaST2.ssh
> 
> File /var/lib/YaST2/runme_at_boot does not exist ...
> 
> Not running YaST ...
> 
> Bug#783172 is possibly related.
>
> Work-around:  touch /var/lib/YaST2/runme_at_boot
> 
> Reproducible: Always

I cannot confirm.

I did a minimal installation on a laptop using linux+initrd from the 12.3-RC2 DVD (DVD-iso mounted and accessible via http on another machine).

I did several tests with a minimal server installation. Each time yast.ssh ran well in the 2nd phase, although I had to fill resolv.conf, and do ifup eth0 after first login (hmm, bad for an ssh install).

AFAICS no relation with Bug#783172. There, in the 1st phase, runme_at_boot is created in the _host_ system (not in the system which is being installed).

Your issue is that runme_at_boot disappears in the 2nd phase (which I could not confirm).

DVD used: openSUSE-12.3-DVD-Build0095-i586.iso
sha1sum: 315b74c53743ff4b8ac72be450479f752673f434

Did you perhaps use the x86_64 DVD?

(In reply to comment #3)
> (In reply to comment #0)
>
> Did you perhaps use the x86_64 DVD?

I'm installing x86_64 via pxe+ssh with the linux+initrd from the NET iso.

(In reply to comment #4)
> I'm installing x86_64 via pxe+ssh with the linux+initrd from the NET iso.

I set up a dhcp/tftp-server and installed several times as you describe, but still could not confirm.

Created attachment 528506 [details]
y2logs

I have also observed this issue when installing via ssh/ftp on x86_64 UEFI machine.

touch /var/lib/YaST2/runme_at_boot seems to be usable as a workaround

Maybe the common factor for this problem occuring is installing an UEFI system.
Per is your system also using UEFI?
Could you please also attach y2log files.

Created attachment 529372 [details]
y2logs as saved by save_y2logs

Hi Thomas

no, my system isn't UEFI. For me, the new thing seems to be YaST Second Stage being called as part of the init-sequence.

Thanks for log files.

Sure, /usr/lib/YaST2/startup/YaST2.Second-Stage is called via systemd
service /usr/lib/systemd/system/YaST2-Second-Stage.service

But I do not see why this is a problem under some circumstances and
no problem under others. In all y2logs I could not see any problem in
first stage and would assume /var/lib/YaST2/runme_at_boot was present
at end of first stage. 

So I cannot really see why under some circumstances 
/var/lib/YaST2/runme_at_boot is gone when one logs in via ssh and 
starts yast.ssh

(In reply to comment #9)
> Thanks for log files.
> 
> Sure, /usr/lib/YaST2/startup/YaST2.Second-Stage is called via systemd
> service /usr/lib/systemd/system/YaST2-Second-Stage.service
> 
> But I do not see why this is a problem under some circumstances and
> no problem under others. In all y2logs I could not see any problem in
> first stage and would assume /var/lib/YaST2/runme_at_boot was present
> at end of first stage. 

I've checked it a couple of times, the file is there.

> So I cannot really see why under some circumstances 
> /var/lib/YaST2/runme_at_boot is gone when one logs in via ssh and 
> starts yast.ssh

I don't know why it always happens - it seems to me that the service unit for yast second stage is quite clear:

ExecStartPost=/bin/rm -f /var/lib/YaST2/runme_at_boot

(In reply to comment #10)
> I don't know why it always happens - it seems to me that the service unit for

.. why it DOESN'T always happen ....

> yast second stage is quite clear:
> 
> ExecStartPost=/bin/rm -f /var/lib/YaST2/runme_at_boot

> ExecStartPost=/bin/rm -f /var/lib/YaST2/runme_at_boot

But this would remove /var/lib/YaST2/runme_at_boot AFTER
 /usr/lib/YaST2/startup/YaST2.Second-Stage has finished, which
is exactly what we want.

In case of ssh based installation  
/usr/lib/YaST2/startup/YaST2.Second-Stage
is basically just waiting for  /var/lib/YaST2/runme_at_boot
to vanish. This is normally supposed to happen when user logs in 
via ssh, starts YaST and finishes second stage install in YaST.
Second stage YaST removes the file runme_at_boot when
it finishes.

So it looks as if something is removing /var/lib/YaST2/runme_at_boot
under some circumstances between end of first stage install and
the time /usr/lib/YaST2/startup/YaST2.Second-Stage is called.

I have now three cases, two here and I assume bnc#804283 is just 
the same. I do not see much in common except of ssh based 
install and architecture x86_64. So far I was testing with arch
i586 and could not imagin this to be a 32/64bit issue but there are
reports about other problems with system services being 64bit related. 
Will set up x86_64 virtual box maybe I can reproduce there.

(In reply to comment #12)
> > ExecStartPost=/bin/rm -f /var/lib/YaST2/runme_at_boot
> 
> But this would remove /var/lib/YaST2/runme_at_boot AFTER
>  /usr/lib/YaST2/startup/YaST2.Second-Stage has finished, which
> is exactly what we want.

I think it will remove that file after it has started YaST2.Second-Stage, not after YaST2.Second-Stage has finished.  That might explain why it happens sometimes, and sometimes not.

No I do not think so. At least it does not behave this way on i586 arch.
Otherwise one could never be fast enough to ssh into a system and
/var/lib/YaST2/runme_at_boot still being present. 

Since I do more than 90% of my installations via ssh I would certainly 
have recognized.  

Description on http://www.freedesktop.org/software/systemd/man/systemd.service.html
is not very precise though:

----------------------------------------------
ExecStartPre=, ExecStartPost=

    Additional commands that are executed before or after the command 
    in ExecStart=, respectively. Syntax is the same as for ExecStart=, 
    except that multiple command lines are allowed and the commands 
    are executed one after the other, serially.
----------------------------------------------

So I would assume it to be executed after the command in ExecStart
is finished.

Frederic, what is the intended behavior of ExecStartPost? 
Will it be executed after the command in ExecStart is finished 
or just after command in ExecStart has been started?
Are there any known bugs/differences in behavior between i586 
and x86_64 architecture?

(In reply to comment #14)

> Frederic, what is the intended behavior of ExecStartPost? 
> Will it be executed after the command in ExecStart is finished 
> or just after command in ExecStart has been started?

It is executed after the ExecStart is finished.

> Are there any known bugs/differences in behavior between i586 
> and x86_64 architecture?

No.

My tests on x86_64 did also not reproduce the problem.

Martin, is the problem reproducable wuuth your machine?
If yes, could you boot the machine into ssh install mode and 
provide me with ssh credentials so that I can retry install.

Hi Thomas

I tried the following:

a new install, stop before phase 1 reboots. 
edit /usr/lib/systemd/system/YaST2-Second-Stage.service and change :

ExecStartPost=/usr/bin/logger runme_at_boot   

Mar 13 15:36:39 linux kernel:   22.180219] systemd[1]: Started Name Service Cache Daemon.
Mar 13 15:36:39 linux kernel:   22.216672] systemd[1]: Starting Permit User Sessions...
Mar 13 15:36:39 linux kernel:   22.376069] systemd[1]: Started Login Service.
Mar 13 15:36:39 linux kernel:   22.440106] systemd[1]: Started LSB: CPUFreq modules loader.
Mar 13 15:36:39 linux kernel:   22.524075] systemd[1]: Started Permit User Sessions.
Mar 13 15:36:39 linux kernel:   22.572267] systemd[1]: Starting Serial Getty on ttyS0...
Mar 13 15:36:39 linux kernel:   22.608182] systemd[1]: Started Serial Getty on ttyS0.
Mar 13 15:36:39 linux kernel: [   23.108038] EXT4-fs error (device sda1): ext4_mb_generate_buddy:742: group 65, 24090 clusters in bitmap, 24091 in gd
Mar 13 15:36:41 linux purge-kernels[392]: /sbin/purge-kernels: Nothing to do.
Mar 13 15:36:41 linux systemd[1]: Started Purge old kernels.
Mar 13 15:36:44 linux logger: runme_at_boot
Mar 13 15:36:44 linux systemd[1]: Started YaST2 Second Stage.
Mar 13 15:36:44 linux systemd[1]: Started YaST2 Firstboot.
Mar 13 15:36:44 linux systemd[1]: Starting LSB: Configure network interfaces and set up routing...

Notice the line with "runme_at_boot".

Either YaST2-Second-Stage finishes immediately or systemd does not wait for it to finish.  Maybe I need to try this with systemd debug enabled.

That is indeed interesting.
You can check if yast2 second stage is still running by loggin in via ssh
and check with pstree -ap if /usr/lib/YaST2/startup/YaST2.Second-Stage
is still running. I will make the same change in my x86_64 install and
see how it differs.

I'd suggest to stop losing time on whetever systemd is starting "ExecStartPost" before ExecStart is finished, because it is not (check the code if you don't trust me) when service type is oneshot or daemon (it won't wait if it is type=simple or type=idle, but that's logic).

Thanks copied logs but did not find anything obvious.
I saw that you switched "Use Automatic Configuration" off.
So far I did not try this, will try an install with 
"Use Automatic Configuration" off and see if I can reproduce.

Yesterday I installed a xen guest with 12.3+updates (added at install-time), and I did not experience this problem. I also did an install without updates, also no problem.  What is REALLY weird - when comparing the log output to comment#17, I don't see most of those lines. 

Ignoring that these two tests were xen guests, the only difference I can really see is that my initial problem appeared when I installed on root on iSCSI.

Have just now done another 12.2 install with root on iSCSI.  As initially reported, I get "File /var/lib/YaST2/runme_at_boot does not exist ..." when I try to run /usr/lib/YaST2/startup/YaST2.ssh. 
Re comment#18, /usr/lib/YaST2/startup/YaST2.Second-Stage is not running at this point.

Same setup, but with openSUSE 13.1 M1 - problem seems to have disappeared.

> Same setup, but with openSUSE 13.1 M1 - problem seems to have disappeared.


So, let's assume the bug was fixed by some other fix and wait for next milestone/beta if it reapper and is simply reproducible.(In reply to comment #24)

Created attachment 541511 [details]
y2logs

Still reproducible issue with openSUSE 13.1 M1

Adjusting importance

It seems that despite installing over ssh, second stage yast was started after reboot - started with X server on machine as if installation was performed locally (I have not noticed this issue before since machine is located in different room ..).

Once yast second stage is started, it deletes runme_at_boot (deleted by YaST2-Second-Stage.service - see second comment).

Issue is obviously reproducible only when installation is performed via ssh, not locally.

Works as expected for me with SSH installation.

Please provide output of "ps axf" so that we can see how YaST was
started locally on your system.

Hmm it seems that yast did not started this time, runme_at_boot is however still missing when installing on same machine - issue is reproducible all the time. If you need access to the machine, it is same as from comment#20.

Installation was performed from network (ftp), machine is uefi, install cmd_line is:
install=ftp://fallback.suse.cz/install/SLP/openSUSE-13.1-Milestone1/x86_64/DVD1/ usessh=1 sshpassword=****** Y2DEBUG=1 debug linuxrclog=/dev/console console=tty0 console=ttyS0,115200

If you need any info please let me know.

Please follow these steps during new installation:

Select audit to be installed.

Before system reboots (e.g. during reboot countdown, mount target to /mnt,
chroot /mnt):

ln -sf /usr/lib/systemd/system/auditd.service /etc/systemd/system/default.target.wants/

Append to /etc/audit/audit.rules:

-w /var/lib/YaST2/runme_at_boot -k runme-watcher -p rwax

Continue with reboot.

After ssh-login check "auditctl -l", should list:

LIST_RULES: exit,always watch=/var/lib/YaST2/runme_at_boot perm=rwxa key=runme-watcher

After runme_at_boot has vanished provide output of "ausearch -f /var/lib/YaST2/runme_at_boot -i".

Also provide all YaST logs and ps axf from same installation to check
pids and times.

Created attachment 541564 [details]
y2logs

ausearch -f /var/lib/YaST2/runme_at_boot -i
----
type=PATH msg=audit(05/28/13 18:02:26.231:27) : item=1 name=/var/lib/YaST2/runme_at_boot inode=132678 dev=00:11 mode=file,644 ouid=root ogid=root rdev=00:00 
type=PATH msg=audit(05/28/13 18:02:26.231:27) : item=0 name=/var/lib/YaST2/ inode=49123 dev=00:11 mode=dir,755 ouid=root ogid=root rdev=00:00 
type=CWD msg=audit(05/28/13 18:02:26.231:27) :  cwd=/ 
type=SYSCALL msg=audit(05/28/13 18:02:26.231:27) : arch=x86_64 syscall=unlinkat success=yes exit=0 a0=ffffffffffffff9c a1=0x1c90750 a2=0x0 a3=0x7fff680bde60 items=2 ppid=1 pid=4071 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root ses=unset tty=(none) comm=rm exe=/usr/bin/rm key=runme-watcher

Created attachment 541565 [details]
ps axu

Thanks. So the process removing runme_at_boot is /usr/bin/rm and the parent
is /sbin/init.

Looks like a systemd problem after all.

(In reply to comment #34)
> Thanks. So the process removing runme_at_boot is /usr/bin/rm and the parent
> is /sbin/init.
> 
> Looks like a systemd problem after all.

Well, the only location doing the rm is /usr/lib/systemd/system/YaST2-Second-Stage.service, in the ExecStartPost section, ie after ExecStart section has been run has been started.

So, in short, second stage was started...

you should be able to double-check it with journalctl -b -u YaST2-Second-Stage.service

Yes, second stage was started. The y2start.log shows that. But normally
YaST (y2base) deletes runme_at_boot but not in this case.

I tried to do the same as it was done in /etc/rc.d/boot regarding YaST startup but if you think this rm shouldn't be done in the systemd service file, feel free to remove it (so it will be only be done in YaST code)..

Is this problem still existing in 13.1? 
Otherwise I'll close this bug after almost a year w/o comment

I haven't seen it in 13.1, feel free to close.

Fixed in 13.1 -> closing