Bug 780259

Summary: service atd cannot be started
Product: [openSUSE] openSUSE 12.2 Reporter: Giacomo Comes <comes>
Component: OtherAssignee: Cristian Rodríguez <crrodriguez>
Status: RESOLVED FIXED QA Contact: E-mail List <qa-bugs>
Severity: Major    
Priority: P5 - None CC: a.klausmann, arvidjaar, aspiers, bruno, fcrozat, forgotten_7XRNh5WElB, jengelh, k.bug, malcolmlewis, novell-ugeuder, suse-beta, sven
Version: Final   
Target Milestone: ---   
Hardware: PC   
OS: openSUSE 12.2   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---

Description Giacomo Comes 2012-09-13 15:35:16 UTC
User-Agent:       Mozilla/5.0 (X11; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0

running 'systemctl start atd.service' fails with this output from 
'systemctl status atd.service':

atd.service - Execution Queue Daemon
          Loaded: loaded (/lib/systemd/system/atd.service; disabled)
          Active: failed (Result: exit-code) since Thu, 13 Sep 2012 07:28:23 -0400; 5s ago
         Process: 12884 ExecStart=/usr/sbin/atd -f (code=exited, status=1/FAILURE)
          CGroup: name=systemd:/system/atd.service

The reason of the failure is that in the file /lib/systemd/system/atd.service
the line:
ExecStart=/usr/sbin/atd -f
contains an argument (-f) unsupported by atd. Removing -f makes the service start.

Reproducible: Always

Steps to Reproduce:
1.run systemctl start atd.service
2.
3.
Actual Results:  
service atd does not start

Expected Results:  
service atd should start
Comment 1 Giacomo Comes 2012-09-13 19:55:48 UTC
Actually removing -f does not make the service start.
After removing -f and executing systemctl start atd.service the output of
systemctl status atd.service is:

atd.service - Execution Queue Daemon
          Loaded: loaded (/lib/systemd/system/atd.service; disabled)
          Active: inactive (dead) since Thu, 13 Sep 2012 15:47:34 -0400; 2s ago
         Process: 2816 ExecStart=/usr/sbin/atd (code=exited, status=0/SUCCESS)
          CGroup: name=systemd:/system/atd.service

or:

atd.service - Execution Queue Daemon
          Loaded: loaded (/lib/systemd/system/atd.service; disabled)
          Active: inactive (dead)
          CGroup: name=systemd:/system/atd.service

Sep 13 15:40:22 linux.site atd[689]: pam_apparmor(atd:session): Unknown error occurred changi...tted

loocking in /var/log/messages there is:

Sep 13 15:40:22 linux atd[689]: pam_apparmor(atd:session): Unknown error occurred changing to root hat: Operation not permitted

So it appears there is a permission problem somewhere that doesn't allow atd to start properly.

So far the only way I have found to start the service is:
cd /etc/init.d ; ./atd start
Comment 2 Christian Boltz 2012-09-13 21:41:17 UTC
The apparmor messages are somehow related, but most probably aren't the cause of the problem.

The problem is that atd forks into background. I got it working with the following in the service file:

[Service]
ExecStart=/usr/sbin/atd
Type=forking

Long story short - I created request id 134202 ;-)
Comment 3 Bernhard Wiedemann 2012-09-17 12:00:07 UTC
This is an autogenerated message for OBS integration:
This bug (780259) was mentioned in
https://build.opensuse.org/request/show/134660 Factory / at
Comment 4 Malcolm Lewis 2012-10-08 20:18:22 UTC
Hi
A user also came across this in the forums for 12.2.
http://forums.opensuse.org/showthread.php?t=478387

The fix you have doesn't take into account the atd sysconfig settings? Or should this be dropped?

[Unit]
Description=Execution Queue Daemon
After=syslog.target
  
[Service]
Type=forking
EnvironmentFile=-/etc/sysconfig/atd
ExecStart=/usr/sbin/atd -b ${ATD_BATCH_INTERVAL} -l ${ATD_LOADAVG}
 
[Install]
WantedBy=multi-user.target

This also means modifying the atd sysconfig file to include;

ATD_BATCH_INTERVAL = 60
ATD_LOADAVG = 0.8

These are the compiled defaults, but at least offers the option to configure.
Comment 5 Christian Boltz 2012-10-08 21:41:13 UTC
comment 4 looks like a valid issue, but is beyond my knownledge of the atd internals.

Cristian, please take over ;-)  (and also prepare an update for 12.2 please)
Comment 6 Adam Spiers 2012-10-24 22:48:28 UTC
Looks like a dupe of bug 772111?
Comment 7 Christian Boltz 2012-10-25 16:44:24 UTC
*** Bug 772111 has been marked as a duplicate of this bug. ***
Comment 8 Sven Uebelacker 2012-11-07 13:15:50 UTC
I tried first the same idea mentioned in "Comment 2" by removing the "-f" in atd.service and reloading systemd. I'm running after 90s in a timeout error like

/var/log/messages
"""
systemd[1]: atd.service stopping timed out. Killing.
systemd[1]: Unit atd.service entered failed state.
"""

systemd:
"""
# systemctl start atd.service && systemctl status atd.service
atd.service - Execution Queue Daemon
          Loaded: loaded (/lib/systemd/system/atd.service; enabled)
          Active: deactivating (stop-sigterm) since Wed, 07 Nov 2012 13:49:55 +0100; 1s ago
         Process: 27868 ExecStart=/usr/sbin/atd (code=exited, status=0/SUCCESS)
          CGroup: name=systemd:/system/atd.service
                  └ 27869 /usr/sbin/atd
### after 90s:
# systemctl status atd.service
atd.service - Execution Queue Daemon
          Loaded: loaded (/lib/systemd/system/atd.service; enabled)
          Active: failed (Result: timeout) since Wed, 07 Nov 2012 13:51:25 +0100; 233ms ago
         Process: 27868 ExecStart=/usr/sbin/atd (code=exited, status=0/SUCCESS)
          CGroup: name=systemd:/system/atd.service

"""

Working without timeout but not surviving a reboot ist "Comment 1":
# cd /etc/init.d ; ./atd start

or as well on command line:
# /usr/sbin/atd -d


Using "Comment 4"'s approach in defining explicitly /etc/sysconfig/atd variables and altering /lib/systemd/system/atd.service is working without timeouts

"""
# systemctl status atd.service 
atd.service - Execution Queue Daemon
          Loaded: loaded (/lib/systemd/system/atd.service; enabled)
          Active: active (running) since Wed, 07 Nov 2012 13:57:36 +0100; 1min 51s ago
         Process: 28822 ExecStart=/usr/sbin/atd -b ${ATD_BATCH_INTERVAL} -l ${ATD_LOADAVG} (code=exited, status=0/SUCCESS)
        Main PID: 28823 (atd)
          CGroup: name=systemd:/system/atd.service
                  └ 28823 /usr/sbin/atd -b 60 -l 0.8
"""


Another working test by ignoring the /etc/sysconfig/atd configuration is possible by removing the "-f" and adding "Type=forking" in the original /lib/systemd/system/atd.service for avoiding the 90s timeout.
"""
# systemctl status atd.service 
atd.service - Execution Queue Daemon
          Loaded: loaded (/lib/systemd/system/atd.service; enabled)
          Active: active (running) since Wed, 07 Nov 2012 14:05:32 +0100; 3min 5s ago
         Process: 29788 ExecStart=/usr/sbin/atd (code=exited, status=0/SUCCESS)
        Main PID: 29789 (atd)
          CGroup: name=systemd:/system/atd.service
                  └ 29789 /usr/sbin/atd
"""


Either way with or without reading /etc/sysconfig/atd the last tests are working. I cannot decide which is best if default values are not easily set by systemd units.
Comment 9 Klaus Vink Slott 2012-11-24 08:44:32 UTC
(In reply to comment #4)
> A user also came across this in the forums for 12.2.
> http://forums.opensuse.org/showthread.php?t=478387
 
Just did my first migration to 12.2 and was also hit by this bug. The workaround described in the forum solved the issue for me. 

This issue is still around ... please fix.
Comment 10 Christian Boltz 2012-11-24 20:42:59 UTC
Frederic, is there a way to use a default value if a variable is empty in the ExecStart line of a service file?

Details: atd has some config variables in /etc/sysconfig, which are empty by default. Unfortunately startup fails if the variables are empty, so we have "failure by default" :-(

I'll describe two possible solutions in pseudocode - please tell me if/how this can be done in a service file.

if ($ATD_BATCH_INTERVAL != "") {
    ExecStart=/usr/sbin/atd -b ${ATD_BATCH_INTERVAL}
} else {
    ExecStart=/usr/sbin/atd    # without -b
}

- or -

if ($ATD_BATCH_INTERVAL != "") {
    $ATD_BATCH_INTERVAL = 60
}
ExecStart=/usr/sbin/atd -b ${ATD_BATCH_INTERVAL}

Can one of those be done in a service file?

(in practise, there are two variables, but that's a technical detail ;-)
Comment 11 Andrei Borzenkov 2012-11-25 05:57:46 UTC
(In reply to comment #10)
> if ($ATD_BATCH_INTERVAL != "") {
>     $ATD_BATCH_INTERVAL = 60
> }
> ExecStart=/usr/sbin/atd -b ${ATD_BATCH_INTERVAL}
> 
> Can one of those be done in a service file?
> 

Not really. You can do

Environment=ATD_BATCH_INTERVAL=60
EnvornmentFile=/etc/sysconfig/atd

but this will break if config file contains ATD_BATCH_INTERVAL="" as it does currently. So the only way to do it is

ExecStart=/bin/sh -c '[ -e /etc/sysconfig/atd ] && . /etc/sysconfig/atd; exec /usr/sbin/atd -b $${ATD_BATCH_INTERVAL:-60} -l $${ATD_LOADAVG:-0.8}'

($$ to please systemd environment replacement code).
Comment 12 Christian Boltz 2012-11-26 18:26:33 UTC
(In reply to comment #11)
> ExecStart=/bin/sh -c '[ -e /etc/sysconfig/atd ] && . /etc/sysconfig/atd; exec
> /usr/sbin/atd -b $${ATD_BATCH_INTERVAL:-60} -l $${ATD_LOADAVG:-0.8}'

Thanks, that works (and looks ugly ;-) 

However, I don't like to hardcode the defaults in the service file because this will cause funny effects[tm] if atd ever changes its defaults.
Fortunately bash also offers ${foo:+whatever} so that it's possible to set the parameter only if a value was set in the sysconfig file.

I also changed /bin/sh to /bin/bash because the variable magic looks bash-specific to me.

That all said - here's the final line:

ExecStart=/bin/bash -c '[ -e /etc/sysconfig/atd ] && . /etc/sysconfig/atd; exec /usr/sbin/atd $${ATD_BATCH_INTERVAL:+-b $$ATD_BATCH_INTERVAL} $${ATD_LOADAVG:+-l $$ATD_LOADAVG}'

I just commited the fixed atd.service:
- request #142950 to Base:System
- request #142949 for the 12.2 update
Comment 13 Bernhard Wiedemann 2012-11-26 19:00:10 UTC
This is an autogenerated message for OBS integration:
This bug (780259) was mentioned in
https://build.opensuse.org/request/show/142949 Maintenance /
Comment 14 Bernhard Wiedemann 2012-11-30 18:00:38 UTC
This is an autogenerated message for OBS integration:
This bug (780259) was mentioned in
https://build.opensuse.org/request/show/142986 Factory / at
Comment 15 Swamp Workflow Management 2012-12-05 13:08:43 UTC
openSUSE-RU-2012:1607-1: An update that has one recommended fix can now be installed.

Category: recommended (low)
Bug References: 780259
CVE References: 
Sources used:
openSUSE 12.2 (src):    at-3.1.8-1110.4.1
Comment 16 Sven Uebelacker 2012-12-06 12:44:07 UTC
Brilliant! It works. Thanks a lot!

Before:
atd.service - Execution Queue Daemon
	  Loaded: loaded (/lib/systemd/system/atd.service; enabled)
	  Active: active (running) since Thu, 06 Dec 2012 11:58:29 +0100; 1min 44s ago
	Main PID: 11415 (atd)
	  CGroup: name=systemd:/system/atd.service
		  └ 11415 /usr/sbin/atd -b 60 -l 0.8


After Update to at-3.1.8-1110.4.1:
atd.service - Execution Queue Daemon
	  Loaded: loaded (/lib/systemd/system/atd.service; enabled)
	  Active: active (running) since Thu, 06 Dec 2012 13:33:45 +0100; 6s ago
	 Process: 21823 ExecStart=/bin/bash -c [ -e /etc/sysconfig/atd ] && . /etc/sysconfig/atd; exec /usr/sbin/atd $${ATD_BATCH_INTERVAL:+-b $$ATD_BATCH_INTERVAL} $${ATD_LOADAVG:+-l $$ATD_LOADAVG} (code=exited, status=0/SUCCESS)
	Main PID: 21824 (atd)
	  CGroup: name=systemd:/system/atd.service
		  └ 21824 /usr/sbin/atd -b 60 -l 0.8
Comment 17 Christian Boltz 2012-12-06 17:39:37 UTC
Thanks for testing!

The update for 12.2 was released, therefore I'm closing this bug.

For the records: In the meantime, Factory got a different fix (a patch for atd) for reading the sysconfig file, so the service file looks different than in the 12.2 update.
Comment 18 Jan Engelhardt 2013-01-05 23:15:43 UTC
Instead of iterating over the map with HXmap_traverse, you can just inspect the keys directly, just like everybody would in a scripting language ($sysconfigmap{ATD_BATCH_INTERVAL}) - it's a lot cheaper.

char *v;
v = HXmap_get(sysconfigmap, "ATD_BATCH_INTERVAL");
if (v != NULL && strlen(v) > 0) {
    batch_interval = strtol(v, NULL, 0);
    syslog(LOG_INFO, "sysconfig requested...");
}
v = HXmap_get(sysconfigmap, "ATD_LOADAVG");
//same procedure.
Comment 19 Christian Boltz 2013-01-06 12:11:44 UTC
(In reply to comment #18)
> Instead of iterating over the map with HXmap_traverse, you can just inspect the
> keys directly, just like everybody would in a scripting language
> ($sysconfigmap{ATD_BATCH_INTERVAL}) - it's a lot cheaper.

Jan, you don't want me to touch C any code ;-)

Feel free to send a SR - I'm sure the atd maintainer (which I'm not) will happily accept it.
Comment 20 Jan Engelhardt 2013-01-10 11:33:55 UTC
The SR is 147894 - can you test?
Comment 21 Bernhard Wiedemann 2013-01-10 13:00:29 UTC
This is an autogenerated message for OBS integration:
This bug (780259) was mentioned in
https://build.opensuse.org/request/show/147899 Factory / at
Comment 22 Christian Boltz 2013-01-10 16:35:52 UTC
Jan, you added a bug to at-parse-suse-sysconfig.patch ;-)
Fortunately it's only a comment, but I still don't want to take credits for something I didn't do. The initial patch was created by Frederic Crozat, not by me.
Comment 23 Jan Engelhardt 2013-01-10 18:36:19 UTC
Dammit, that's why patch creators should always follow http://en.opensuse.org/openSUSE:Packaging_Patches_guidelines#Type_2:_Complete_Information_provided_in_patch .
Comment 24 Bernhard Wiedemann 2013-01-10 22:00:09 UTC
This is an autogenerated message for OBS integration:
This bug (780259) was mentioned in
https://build.opensuse.org/request/show/147979 Factory / at
Comment 25 Christian Boltz 2013-01-11 00:01:02 UTC
*argh* I found a bug in /dev/brain :-( and gave you a wrong name - sorry for that!

To avoid another error, I'll just paste the at changelog:

* Mi Nov 28 2012 crrodriguez@opensuse.org
- When sysconfig variables are empty, do not set stuff to zero
  just use the defaults.

* Mi Nov 28 2012 crrodriguez@opensuse.org
- Fix systemd unit and sysconfig parsing the right way.

* Mo Nov 26 2012 opensuse@cboltz.de
- use /etc/sysconfig/atd variables in atd.service (bnc#780259#c4)


(fortunately ;-) your SR was auto-declined because it has no .changes entry)
Comment 26 Jan Engelhardt 2013-01-11 00:07:52 UTC
Once you figure out who procured the patch, you can use that utility they call "osc" and edit the line in the patch.
Comment 27 Christian Boltz 2013-01-11 22:09:05 UTC
OK, I'll do that ;-)
Comment 28 Christian Boltz 2013-02-06 14:21:04 UTC
fixed with SR 151475
Comment 29 Bernhard Wiedemann 2013-02-06 15:00:09 UTC
This is an autogenerated message for OBS integration:
This bug (780259) was mentioned in
https://build.opensuse.org/request/show/151477 Factory / at
Comment 30 Bruno Friedmann 2013-11-05 15:06:23 UTC
Where are the maintenance request for 12.2 & 12.3

Still in 13.1 : atd.service doesn't pick sysconfig vars.
Comment 31 Christian Boltz 2013-11-16 18:38:44 UTC
(In reply to comment #30)
> Where are the maintenance request for 12.2 & 12.3

12.2 was fixed with
https://build.opensuse.org/project/show/openSUSE:Maintenance:1130

12.3 should have working code from the beginning - at least https://build.opensuse.org/request/show/142986 was accepted a year ago.

> Still in 13.1 : atd.service doesn't pick sysconfig vars.

Then the patch must be broken or was lost :-(

Anyway - I'm not the at maintainer, I just helped out a bit. I'll reassign to the current maintainer...
Comment 32 Cristian Rodríguez 2013-11-16 19:41:10 UTC
This bug (In reply to comment #30)
> Where are the maintenance request for 12.2 & 12.3
> 
> Still in 13.1 : atd.service doesn't pick sysconfig vars.

are you sure ? this bug was fixed a long while ago... patch is at-parse-suse-sysconfig.patch, I just tested it, by setting ATD_BATCH_INTERVAL and ATD_LOADAVG 

log says..


Nov 16 16:38:44 xps15z atd[4067]: sysconfig requested batch_interval to be set to 2
Nov 16 16:38:44 xps15z atd[4067]: sysconfig requested load_avg to be set to 2.000000

What is your exact problem ? of course atd.service does not pick the sysconfig values.. /etc/sysconfig/atd is parsed in the application itself.
Comment 33 Bruno Friedmann 2013-11-17 09:00:27 UTC
Okay then we have information misleading comprehension.

Previously in /etc/init.d/atd during the start 
        startproc $ATD_BIN $ATD_ARGS
You finished with a process which clearly indicate how and what option run.
ps auxw ...

Which is perfectly aligned with information contained in man atd

Now running the systemd service you finish only with 
/usr/sbin/atd -f
So yes the optional values are there, visible in the status at.service. But not on the proc line !

My question was wouldn't be better to use something like
EnvironmentFile=/etc/sysconfig/atd
ExecStart=/usr/sbin/atd -f $OPTS

Like the solution fedora has adopted, okay requiring to rewrite the sysconfig atd file with a one line but would have the advantage to be simplier
Then anybody looking at the daemon ps would seen that the options are there.

Patching code daemon to make them behaving differently than what is expected is really really strange.
Comment 34 Cristian Rodríguez 2013-11-17 15:51:55 UTC
(In reply to comment #33)
> Okay then we have information misleading comprehension.
> 
> Previously in /etc/init.d/atd during the start 

> My question was wouldn't be better to use something like
> EnvironmentFile=/etc/sysconfig/atd
> ExecStart=/usr/sbin/atd -f $OPTS


This is because we do not have $OPTS , but have separate variables for each option, doing what you said ends in the ugly mess described in comment 12, which other than ugly, is slow as it requires spawning a shell.

 The current approach is correct and works, there is nothing to be fixed.
Comment 35 Bruno Friedmann 2013-11-17 17:32:12 UTC
Without using the #12 mess, You miss two points which I believe are essential.

First you introduce a bug, that none other distribution has with atd
You can't anymore see parameters in ps auxw 

We have a patch that is not upstream. Not that good about our policy.

We reinvent our own soap, in place of trying to collaborate with our neighbors & friends distribution.

The current approach just work, with defect. That's all.