Bug 699400

adding aaa_base maintainer to CC

Note that I see similar problems with SLES11 SP1, will open a Service Request for that.

INVALID as ntpd force eleven minute mode (read manual page adjtimex(8)) that
is if the kernel is in eleven minute mode the hardware clock is synchronized
every eleven minutes with the system clock.

Please not this will not be changed as it is correct.

I have read the manpage and everything.

The CMOS clock is *not* synchronized. If it was, there would be nothing to correct for sntp in the initial comment.

We are hitting this also on an 1000+ Server SLES11SP1 installation, so just INVALIDing this bug here will not get you out of this ;-)

I would suggest to check if the kernel still does synchronize the CMOS clock when in "eleven minute mode". I guess it does no more.

Changelog of aaa:base of SLES11-SP1-UPDATE:


 Tue Dec 21 17:21:56 CET 2010
 - Do not remove /etc/adjtime (bnc#650719) but correct the third line
 - Test only for bit 64 (clock unsynchronized), if zero the kernel
  is within eleven minutes mode (Thanks goes to Rastislav David)

Update your system

Werner, this is not working. I guess something changed in the kernel.

On an 11.4 with:

nw8000:~ # date; hwclock 
Di 14. Jun 10:47:18 CEST 2011
Di 14 Jun 2011 10:47:19 CEST  -0.142187 seconds

# now de-sync hwclock by ~ 1 hour

nw8000:~ # hwclock --set --date '2011-06-14 09:48:55'
nw8000:~ # hwclock 
Di 14 Jun 2011 09:48:58 CEST  -0.386410 seconds
nw8000:~ # date
Di 14. Jun 10:49:26 CEST 2011
nw8000:~ # adjtimex --print
         mode: 0
       offset: 238363
    frequency: 1109687
     maxerror: 524011
     esterror: 918
       status: 24577
time_constant: 9
    precision: 1
    tolerance: 32768000
         tick: 10000
     raw time:  1308041385s 161255527us = 1308041385.161255527


Now wait for more than 11 minutes, see the hwclock is still desynchronized:

nw8000:~ # hwclock
Di 14 Jun 2011 10:04:30 CEST  -0.751888 seconds
nw8000:~ # date
Di 14. Jun 11:04:32 CEST 2011
nw8000:~ # ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 LOCAL(0)        .LOCL.          10 l 243m   64    0    0.000    0.000   0.000
*server.home.s3e .DCFa.           1 u  332  512  377    0.300    0.040   0.241

nw8000:~ # adjtimex --print|grep status
       status: 8193

shutting down now will not sync the clock.

I read the kernel code and I did not instantly find why it does not work, but it obviously does not work.

It works ... you have to wait upto the point where the kernel does not
do the eleven minute mode anymore. It is not supported to destroy the
sync between the hardware clock any the kernel system time as the
hardware clock is under the control of the kernel its self.

Now the status bits 24577 shows:

   Enabled PLL updates (is read/write)
   Resolution in nano seconds (readonly)
   Frequency-lock mode active (readonly)

now the status bits 8193

   Enabled PLL updates (is read/write)
   Resolution in nano seconds (readonly)

that's it (-> /usr/include/sys/timex.h part ``Status codes'').
Now explain what is wrong with this.

Beside this normally the hardware clock is running in UTC and there
is no need to enforce a de-synchronization with an offset of one
hour nor is there any need to touch the hardware clock if ntpd is
used as the nptd will enforce the kernel to switch into eleven
minute mode.  And even if you de-synchronize the hardware clock
by hand ... how should the kernel know about your change and how
should the kernel make a guess about the choosen time zone of
your hardware clock.  That is that synchronization of the ticks
of hardware clock and system clock its self had not changed.

At startup the kernel assumes UTC as its system clock is always
in UTC. If the hardware clock is not in UTC you have to specify
the time zone offset which is done in /etc/sysconfig/clock
(HWCLOCK="--localtime") and executing mkinitrd.  This because
the warpclock utility in initrd will inform the kernel about
the time zone of the hardware clock *before* the any file system
is touched e.g. for a file system check and mounting the root
files system.

There is nothing wrong with the bits - just the kernel does not sync the CMOS clock.

I did the following:

* set the cmos clock wrong (to 09:00)
* reboot

on boot:

Jun 14 09:01:45 nw8000 sntp[3655]: Started sntp
Jun 14 12:48:22 nw8000 ntpd[3664]: ntpd 4.2.6p3@1.2290-o Wed Feb 23 00:32:28 UTC 2011 (1)
Jun 14 12:48:22 nw8000 ntpd[3665]: proto: precision = 1.396 usec

So far, so good.

Now:
nw8000:~ # adjtimex --print|grep status
       status: 8193
nw8000:~ # uptime
 13:11  up   0:23,  3 users,  load average: 0,16, 0,05, 0,09
nw8000:~ # date;hwclock 
Di 14. Jun 13:11:24 CEST 2011
Di 14 Jun 2011 09:11:25 CEST  -1.015618 seconds


So it looks like only the minutes are actually synchronized.

There is actually a comment in the kernel source arch/x86/kernel/rtc.c:

 41 int mach_set_rtc_mmss(unsigned long nowtime)
 42 {
...
 59         /*
 60          * since we're only adjusting minutes and seconds,
 61          * don't interfere with hour overflow. This avoids
 62          * messing with unknown time zones but requires your
 63          * RTC not to be off by more than 15 minutes
 64          */
 65         real_seconds = nowtime % 60;
 66         real_minutes = nowtime / 60;
 67         /* correct for half hour time zone */
 68         if (((abs(real_minutes - cmos_minutes) + 15)/30) & 1)
 69                 real_minutes += 30;
 70         real_minutes %= 60;
 71 
 72         if (abs(real_minutes - cmos_minutes) < 30) {
 73                 if (!(save_control & RTC_DM_BINARY) || RTC_ALWAYS_BCD) {
 74                         real_seconds = bin2bcd(real_seconds);
 75                         real_minutes = bin2bcd(real_minutes);
 76                 }
 77                 CMOS_WRITE(real_seconds, RTC_SECONDS);
 78                 CMOS_WRITE(real_minutes, RTC_MINUTES);
 79         } else {
 80                 printk_once(KERN_NOTICE
 81                        "set_rtc_mmss: can't update from %d to %d\n",
 82                        cmos_minutes, real_minutes);
 83                 retval = -1;
 84         }

so assuming "kernel is in eleven-minute-mode => CMOS clock is set correctly" is wrong.
We should set the cmos clock on shutdown, even if the kernel is in eleven-minute-mode IMHO

This is exactly what I've said, see comment #7 (I'm aware of 
mach_set_rtc_mmss())... and I'll not override CMOS clock at shutdown
if the kernel is in eleven-minute-mode as this would cause that other
bugs will be reopen (those bugs had caused the implementation of the
eleven-minute-mode check in boot.clock).

Do you really have 1000 hosts around on which the hardware clock is
wrong by more than 15 minutes?  And please note that if the CMOS clock
is off by one hour this could be an other time zone (even with an
offset of an half hour this could be an other time zone).

IMHO this bug is invalid but should an other make this decision.

Yes, my customer is buying systems in lots of 100 servers, and they all come with pretty random dates set. Certainly several hours off (because they are either set to american or taiwanese localtime).

The same is true after exchanging broken hardware: a machine that had correct time before the replacement has random time after. And it never gets corrected.

At the customer, we simply solved it with a cron job that runs "hwclock --systohc --noadjfile --utc" once per day.

But I still think that during shutdown of the system, we have all needed information to set the hwclock correctly:
* we know if it is set to UTC or LOCAL
* we know if the system time is correct (NTP configured)
So we should be able to make an informed decision to set the clock correctly, which the kernel cannot make due to the timezone problem.

For my machines at home I'm now working around this with a line

readonly ELEVENMIN_MODE=no

in /etc/sysconfig/clock, but I don't think that's something to recommend ;)

The very least we need to do IMHO is document in /etc/sysconfig/clock that
SYSTOHC="yes" does not mean "write time to hwclock on shutdown" but actually in real systems with NTP means "never write time to hwclock, no matter what you are doing".

Normally the CMOS clock is set only once to UTC and all works flawless.
Setting the CMOS clock to localtime is a user error (and IMHO we should
not support this anymore as Windows7[tm] can handle UTC im CMOS).

For broken hardware you loose as there is no chance to correct anything.
Changing the CMSO clock if in eleven minute mode would be a bug as the
CMOS clock (the ticks, not the time zone offset!!) is in sync with
the system clock.

In old versions of NTP, you were expected to STEP the clock first on boot for large discrepancies, and only then slew the time after by cron job, or running NTP daemon.  It was no good having many unstable peers & servers.

It seems like bar bugs that this need was already accounted for.

In /etc/init.d/ntp :

  start)
    if [ "$NTPD_FORCE_SYNC_ON_STARTUP" = "yes" ]; then
        # get the initial date from the timeservers configured in ntp.conf
        ntpd_is_running || $0 ntptimeset
    fi

And :

  ntptimeset)
    NTPD_PROTO="$( get_ntpd_ip_proto )"
    for i in $(gawk '/^server/ { if( $2 !~ "^127.127." ) print $2","$3 }' $NTP_CONF)
    do
<snip>
       sntp -t 2 -l /dev/null -s $SNTP_OPT 2> /dev/null && { SYNCHRONISED=$SNTP_OPT; break; };
    done
    if [ "$SYNCHRONISED" ]
    then
            echo "Time synchronized with $SYNCHRONISED"
    else
            echo "Time could not be synchronized"
    fi
  ;;

So perhaps simply making the initial & any following daemon starts, where synchronisation has not succeeded with 24 hours, would solve this issue?

Adding code, roughly like :

    if [ "$SYNCHRONISED" ]
    then
            echo "Time synchronized with $SYNCHRONISED"
touch /var/lib/ntp/ntp-synchronised-start     # record synchronisation time
    else
  
On "ntp start":

# test for recent synchronisation
test `find /tmp/ntp -name ntp-synchronised-start -mtime -1 -print` || 
    NTPD_FORCE_SYNC_ON_STARTUP=yes
    if [ "$NTPD_FORCE_SYNC_ON_STARTUP" = "yes" ]; then
        # get the initial date from the timeservers configured in ntp.conf
        ntpd_is_running || $0 ntptimeset

Oops, timestamp ntp-synchronised-start, has to be somewhere like /var/run/ntp, or /var/lib/ntp/sync, same place of course!

(In reply to comment #11)
> Normally the CMOS clock is set only once to UTC and all works flawless.
> Setting the CMOS clock to localtime is a user error (and IMHO we should
> not support this anymore as Windows7[tm] can handle UTC im CMOS).
> 
> For broken hardware you loose as there is no chance to correct anything.

I tested Windows 7 running with UTC, as it is my preference.  The problem is it is not acceptable to users, because the time shown "localtime" is then UTC, with additional clock in true localtime.  Setting Windows local time clock to Europe/London GMT/BST wiith DST and "additional clock" to be UTC, causes the out by an hour problem!!

MS HAVE not properly supported, system clock in UTC BUT insist in doing DST, and non DST adjustment means it's not localtime.

sntp -s on boot, did indeed fix up the Linux clock, as well as CMOS clock and allow NTP to operate & synchronise from there on.  No 11 minute mode, if it's done before the NTP daemon is even started.

Frankly I cannot see why, an "sntp -s" with 2 or 3 defined servers, is not simply done on boot anyway, the CMOS clock is not nearly as accurate as NTP protocol.  Doing that EVERY time simplifies the rcntp script to.

(In reply to comment #14)

Hmmm ... if you have a flat rate or a static network with connection
forward to NTP server you may use sntp -s ... nevertheless with an
expense dailup/dailin connection this is a nogo (IMHO).

Very true, so in that case you don't define an NTP server externally, but use local time sources. :)

1000 server installation, 100 new machines on dial up, fine.  Set 1 Master, 2 stratum 2's, then stratum 3's which broadcast the NTP time in local nets.
Then that end user defines an "sntp -j" when the net link comes up to slew the clock, set initially by "sntp -s".

In actual fact what tends to matter, is that the clocks agree, not that they're accurate to the second.  If a dial up user complains having set external time servers, it's a configuration error; they are asking for those servers to be queried, by the server.

Rob, I'm not wanting to offend you, but your issues are totally orthogonal to this bug's subject.

This bug is about hwclock not being correcty synchronized on shutdown if NTP is running, leaving it often N hours off the correct UTC time, because the kernel only corrects the minutes and seconds.

I don't understand exactly what your problem is, but please file a separate bugreport for it in order to keep this one workable.

The CMOS clock should be set correctly before the NTP daemon is started.

for #18: I don't understand this comment. If NTP is used and before NTP is started is the last point in time where we have a _wrong_ clock. Why should we write that to CMOS ? Or did you mean "during NTP start write the time received from ntp to CMOS" ?

Because in the bug description, Stefan correctly points out that ntpd, is not setting the clock to the actual time, which is what the end user expects.

Werner saying (also IMO correctly), that NTP daemon behaviour should not be changed.

sntps -s sets system time and for hour off, the RTC/CMOS clock appears to get set right.  The issues I had, were IMO Windows bugs, where NTP synchronisation is set, yet they adjust the clock, causing issue on reboot to Linux.

Setting clock wildly out, it shows it corrected for by -12925 seconds, now I have :

fir:~ # date
Wed Jul 13 16:43:43 BST 2011
fir:~ # hwclock
Wed Jul 13 20:13:48 2011  -0.234731 seconds
fir:~ # echo peers |ntpdc
peers
     remote           local      st poll reach  delay   offset    disp
=======================================================================
=glfd-dmzutil-1. 82.26.243.130    2   64  177 0.01375 -0.000040 0.07019
=LOCAL(0)        127.0.0.1       10   64  160 0.00000  0.000000 1.39331
=winn-dmzutil-1. 82.26.243.130    2   64  177 0.01601  0.000648 0.06609
=dns1.rmplc.co.u 82.26.243.130    2   64  177 0.01511 -0.001409 0.06371
=utserv.mcc.ac.u 82.26.243.130    2   64  177 0.01910  0.003854 0.07028
*d.st1.ntp.br    82.26.243.130    1   64  177 0.24388  0.016828 0.05862

In /etc/init.d/ntp BEFORE ntpd is started, after system time is set by sntp -s, one can hwclock -w --noadjfile to make the CMOS clock, roughly correct.

That then lets NTP & kernel 11 minute mode correct for systematic drift as designed.

As we know & Windows can also these days synchronise to NTP time correctly, then the solution to this bug, is simple.  Set system time, write to hwclock, when booting up NTP.

    if [ "$SYNCHRONISED" ]
    then
            echo "Time synchronized with $SYNCHRONISED"
hwclock -w --noadjfile --localtime       ## Fix me to use logic of boot.clock for system UTC v localtime setting
    else

Reboot & now the Harward aka RTC/CMOS clock is in rough agreement with true time.

fir:~ # date
Wed Jul 13 17:02:22 BST 2011
fir:~ # hwclock
Wed Jul 13 17:02:26 2011  -0.578694 seconds

System time is stepped, so logically knowing the correct time, it's safe to step the CMOS clock and avoid early boot having wrong time before NTP started.

Currently, YaST starts reading system clock (with 'date') and on saving, it writes HW clock (hwclock --set) and synchronizes it back with system one (hwclock --hctosys).

So it actually should get corrected anyway, right?

... additionally, when NTP client is used to set the time during installation, new system time is synced back to HW clock: /sbin/hwclock --systohc

(In reply to comment #30)
ok. So we have verified that the problem is only existent in image based installation workflows. If one add these steps to the post install script the issue will not arise until he has a hardware problem. That should be sufficient.

I was assuming comment 29 referred to a change needed in yast ...

so what change is actually expected here ?

No idea. Jochen?

i have understood comment #30 that everything is fine when a server was setup
with yast and ntp-client. So there are no expectations to yast from my side.
For other (image-based) setup methods the admin has to set only once
the clock in a post-install routine. 
If this is not sufficient i will open a new bug ;)

So I assume the original report is actually not a bug.

We are seeing this problem also. To summarize:

1. Server arrives with crazy HW clock set in the future
2. We image it (using kiwi)
3. System start running, sets system clock from HW clock
4. System clock is crazy wrong, until...
5. ntpd synchronizes and corrects the system clock
6. Certain daemons see the clock jump backwards, start behaving incorrectly
7. System continues running for days, weeks, months
8. System reboots
9. Goto step #3

This problem repeats indefinitely and never corrects itself.

I don't see how this is not a bug.

The bug, as pointed out earlier, is in /etc/init.d/boot.clock's assumption that 11-minute mode implies accurate hardware clock... this is wrong!

So, my dumb question: why not just change the logic of "/etc/init.d/boot.clock stop" so that the hardware clock is updated even if the kernel is in 11-minute mode?

It seems like that would solve this problem and also has zero downside:

--- boot.clock.orig	2011-12-22 15:56:01.178611899 -0600
+++ boot.clock	2011-12-22 15:56:48.018584892 -0600
@@ -154,11 +154,6 @@
 	rc_status -v
 	;;
     stop)
-	if test "$ELEVENMIN_MODE" = yes ; then
-	    echo "The System Time is in sync with Hardware Clock ${stat}${done}good${norm}"
-	    rc_status
-	    rc_exit
-	fi
 	if test "$USE_HWCLOCK" = yes -a "$SYSTOHC" = yes ; then
 	    echo -n "Set Hardware Clock to the current System Time"
 	    #

What am I missing?

Archie, you are missing nothing IMVHO ;-)

I have explained how I work around the issue in http://seife.kernalert.de/blog/2011/12/23/if-you-dont-want-to-fight-package-maintainers/

executive summary: put

readonly ELEVENMIN_MODE=no

into /etc/sysconfig/clock.

(In reply to comment #38)

> The bug, as pointed out earlier, is in /etc/init.d/boot.clock's assumption that
> 11-minute mode implies accurate hardware clock... this is wrong!

If the kernel is in the 11-minute mode than the CMSO clock is accurate that
is touching the CMOS clock with hwclock does not increase the accuracy but
making it worse (sigh!).

The only problem with that is: if the CMOS clock is off by more than 15 minutes
the kernel does not touch the CMOS clock anymore due to the fact that this
could be an other timezone than UTC.

A more correct solution would be

  a) Correct the CMOS clock before starting ntp (see bug #730374) this
     already part of stable/openSUSE_Factory

  b) Modify the 11-minute mode check by not only using the status bit
     but also compare the current CMOS clock with the system clock.

  c) both a) and b)

(In reply to comment #39)

@Seife: It would be more powerful if you could think deeper into the problem
        and help to avoid such ``solutions''.  It is a matter of fact that
        is the kernel is in 11-minute mode the user space program like hwclock
        using a system call can not increase accuracy.  That is that such
        a ``solutions'' will break any correct configured system.

E.g. for more information see `man 8 hwclock' at section:

 Automatic Hardware Clock Synchronization By the Kernel
 [...]
       If  your  system  runs  with 11 minute mode on, don't use hwclock
       --adjust or hwclock --hctosys.  You'll just make a mess.  It is 
       acceptable to use a hwclock --hctosys at startup time to get a
       reasonable System Time until your system is able to set the
       System Time from the external source and start 11 minute mode.

In bash code the point b) could look like

ELEVENMIN_MODE=no
typeset -i STA_CLOCK=0
typeset -i STA_DIFF=$(date +'%s')
while IFS=: read tag value ; do
    value=${value##* }
    case "${tag}" in
    *status)
        let STA_CLOCK=$value ;;
    *raw\ time)
        let STA_DIFF-=${value%.*} ;;
    *return\ value*)
        let 'STA_CLOCK|=64'
    esac
done < <(/sbin/adjtimex --print)

((STA_DIFF < -900 || STA_DIFF > 900)) && let 'STA_CLOCK|=64'
((STA_CLOCK & 64)) || ELEVENMIN_MODE=yes

unset STA_CLOCK STA_DIFF tag value


... that is test if the clocks (minutes and/or seconds) are in sync
due 11 minute mode *and* testing if the offset is less then ±900
seconds. Also testing for a possible return value of the adjtimex(2)
system call (see `man 2 adjtimex'). In both cases we logical add
bit 7 (aka 64 aka 0x40, see /usr/include/bits/time.h)

(In reply to comment #42)
> In bash code the point b) could look like
> ...

Yes, option (b) makes the most sense to me as well. Can we get that added? Does somebody need to create a new feature request?

Thanks.

(In reply to comment #41)
> E.g. for more information see `man 8 hwclock' at section:
> 
>  Automatic Hardware Clock Synchronization By the Kernel
>  [...]
>        If  your  system  runs  with 11 minute mode on, don't use hwclock
>        --adjust or hwclock --hctosys.  You'll just make a mess.  It is 
>        acceptable to use a hwclock --hctosys at startup time to get a
>        reasonable System Time until your system is able to set the
>        System Time from the external source and start 11 minute mode.

During system shutdown, keeping the kernel internal view of the clock accurate is probably less important than making sure that the clock is at least in the correct century on next reboot.

I have updated my blog post with your annotation that this method is totally wrong (and added my evidence from running > 2000 servers with this hack which work fine only since we use this).

If we get a proper solution now, this is also fine.

This is a problem for us as well. To work around it, we've added commands to "halt.local" to explicitly set the hardware clock to system time (--systohc) each time the system is shut down.

In current factory I've added in /etc/init.d/boot.clock

 #
 # Get current status of kernel time variables, if kernel
 # is within "11 minute mode" do not adjust HW clock nor
 # write system time back to HW clock, bug bnc#492921.
 #
 ELEVENMIN_MODE=no
 if test -e /sys/class/rtc/rtc0/since_epoch ; then
     typeset -i STA_DIFF=0
     read -t 1 STA_DIFF < /sys/class/rtc/rtc0/since_epoch
     let STA_DIFF-=$(TZ=UTC /bin/date +'%s')
     if ((STA_DIFF > -900 && STA_DIFF < 900)) ; then
         typeset -i STA_CLOCK=64
         while IFS=: read tag value ; do
             value=${value##* }
             case "${tag}" in
             *status)
                 let STA_CLOCK=$value ;;
             *return\ value*)
                 let 'STA_CLOCK|=64'
             esac
         done < <(TZ=UTC /sbin/adjtimex $HWCLOCK --print)
     fi
     unset tag value
     ((STA_CLOCK & 64)) || ELEVENMIN_MODE=yes
     unset STA_CLOCK STA_DIFF
 fi

as this avoid overwriting a valid NTP sync with an imprecise hwclock
call ... nevertheless /etc/init.d/boot.clock is disabled in current
factory AFAIK due fate#312407 done by Ludwig, right?

FYI,

Since this bug is marked resolved/invalid, and nobody answered my question in comment #43, I went ahead and filed a feature request as Bug #754329.