Bug 774126

Summary: login via serial console fails
Product: [openSUSE] openSUSE Tumbleweed Reporter: Jeff Mahoney <jeffm>
Component: BasesystemAssignee: Mark van Tinteren <kigurame>
Status: RESOLVED FIXED QA Contact: E-mail List <qa-bugs>
Severity: Normal    
Priority: P5 - None CC: antoine.mechelynck, forgotten_sM9JzehKpy, kigurame, ncutler, security-team, snwint, systemd-maintainers, werner
Version: Current   
Target Milestone: Current   
Hardware: Other   
OS: Other   
Whiteboard:
Found By: Development Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Attachments: dmesg.txt

Description Jeff Mahoney 2012-08-02 03:50:18 UTC
I have a system where the primary console is via serial.

It is running systemd. There are two agetty processes running, once on ttyS0 and one on tty1. I haven't tried the one on tty1.

On ttyS0, I can enter the username but the login fails immediately after I type the first character of the password.

The log shows:
Aug  1 15:00:56 sled1 login: FAILED LOGIN 1 FROM ttyS0 FOR root, Authentication failure

... after that the results are unpredictable. It won't even accept the username.

From past experience, it seems as though there are multiple listeners on the serial port but there is only one agetty process running and lsof shows only that agetty process with that serial port open.
Comment 1 Frederic Crozat 2012-08-02 12:49:49 UTC
I don't see the issue with latest 12.2.

what is your cmdline exactly ?
Comment 2 Jeff Mahoney 2012-08-02 15:39:02 UTC
BOOT_IMAGE=/boot/vmlinuz-3.4.4-1.1-desktop root=UUID=019072c5-31df-414a-be30-b2f947436879 eth0=dhcp console=tty0 console=ttyS0,115200 resume=/dev/disk/by-id/ata-MB0500EAMZD_9WJ1R3LX-part5 splash=silent quiet showopts
Comment 3 Frederic Crozat 2012-08-02 15:56:09 UTC
no issue on my test VM.

could you boot with systemd.log_level=debug systemd.log_target=kmsg and when it crashes, login through ssh (or another tty, if you can) and get dmesg output  and attach it to this bug report ?
Comment 4 Jeff Mahoney 2012-08-02 16:22:59 UTC
Huh. It seems to only have happened for the boot immediately following installation. It works fine now.
Comment 5 Jeff Mahoney 2012-08-02 16:25:33 UTC
Oops. Or not. Enabling debugging seems to have affected the behavior. I'll try once more.
Comment 6 Jeff Mahoney 2012-08-02 16:39:23 UTC
Created attachment 500916 [details]
dmesg.txt

Ok. I misinterpreted.

The key thing is that a getty is presented *before* the rest of the jobs have been started. If I try to log in prior to the boot process finishing, getty crashes or otherwise exits early. If I wait until the remaining jobs complete, the login proceeds normally.

It may be a factor that I have NIS enabled on the system, even though I don't actually use it for logins. +:::::: isn't in /etc/passwd.
Comment 7 Frederic Crozat 2012-08-03 08:07:20 UTC
I'll be able to fix "present getty after all services have started" only after 12.2 is released because new type of service (called Idle) has been introduced in upstream systemd to fix that but backporting it would be too invasive.

from the log, it looks like waiting for ypbind triggered a timer:
[   26.631738] systemd[1]: serial-getty@ttyS0.service holdoff time over, scheduling restart.

could you try to edit /lib/systemd/system/getty@.service and on the line After= add ypbind.service ?
Comment 8 Jeff Mahoney 2012-08-03 14:47:35 UTC
Adding that line didn't affect the results.

I also tried removing the NIS components from /etc/shadow but it had no impact.

Disabling NIS completely had no impact.

Unfortunately replacing agetty with a script that runs it through strace didn't work either.
Comment 9 Frederic Crozat 2012-08-03 15:09:22 UTC
looking at the trace, ypbind is started before getty so it is not the issue.

just to test, you could add After=xdm.service (since xdm is one of the last service) but it probably won't help..
Comment 10 Dr. Werner Fink 2013-12-08 21:53:03 UTC
Is this bug still valid?  This because I'm able to login as root via serial console after adding /dev/ttyS0 to /etc/securetty. I'm running also grub on serial console and using console=ttyS0,1152000 console=tty0 on the kernel command line.
Comment 11 Jeff Mahoney 2013-12-08 22:18:22 UTC
I'll need to install 13.1 on this machine to test. FWIW, I don't think it should be necessary to add ttyS0 to /etc/securetty when it was used as a system console during installation.
Comment 12 systemd maintainers 2014-01-08 13:47:24 UTC
Without ttyS0 in /etc/securetty root is not allowed to use /dev/ttyS0 over a serial line.

Let's ask the maintainer of yast2-bootloader if the serial terminal line is added to the /etc/securetty in case of using the line for root access.

I guess this is not the case ... also I've added the security team to CC list as IMHO this could be a securitry issue as a serial line may or may not be a secure line for root access ... in last case this bug would become invalid.
Comment 13 Dr. Werner Fink 2014-01-10 13:23:36 UTC
This is not systemd but plymouth.  The plymouth process locks the termios structure with the ioctl TIOCSLCKTRMIOS and this leads to the effect that the first agetty on the system console e.g. the serial line does not change the termios structure with tcsetattr().

I've added a simple routine

  for (i = 0; i < 10; i++) {
          /*
           * Even with TTYReset=no it seems with systemd or plymouth
           * the termios flags become changed from under the first
           * agetty on a serial system console as the flags are locked.
           */
          struct termios lock;
          memset(&lock, 0, sizeof(struct termios));
          if (ioctl(STDIN_FILENO, TIOCGLCKTRMIOS, &lock) < 0)
                  break;
          if (!lock.c_iflag && !lock.c_oflag && !lock.c_cflag && !lock.c_lflag)
                  break;
          debug("termios locked\n");
          tcsendbreak(STDIN_FILENO, 0);           /* 0.25 upto 0.5 seconds */
  }

in termio_init() of agetty
Comment 14 Bernhard Wiedemann 2014-01-10 14:00:15 UTC
This is an autogenerated message for OBS integration:
This bug (774126) was mentioned in
https://build.opensuse.org/request/show/213466 Factory / util-linux
Comment 15 Dr. Werner Fink 2014-01-10 14:24:52 UTC
Even if I've added a workaround to agetty there remains a bug in plymouth as systemd uses

   /usr/bin/plymouth --wait

in plymouth-quit-wait.service with 20 seconds timeout but plymouth does not unlock the termios structure within this timeout.  Clearly serial-getty@.service does depend that plymouth-quit-wait.service is finished.
Comment 16 Bernhard Wiedemann 2014-01-10 15:00:10 UTC
This is an autogenerated message for OBS integration:
This bug (774126) was mentioned in
https://build.opensuse.org/request/show/213474 Factory / util-linux
Comment 17 Dr. Werner Fink 2014-01-13 13:16:51 UTC
Beside this problem that /usr/bin/plymouth --wait does not restore the consoles termios structure the other problem that the serial console is not part of the file /etc/securetty still remains.

IMHO and after a talk with Steffen the physical device of the system console should become part of /etc/securetty in the YaST2 module used for asking/setting the password of the super user root.
Comment 18 Dr. Werner Fink 2014-01-13 13:52:33 UTC
@ Raymond :  Please could you report the problem upstream, that is that plymouth
with option --wait does not unlock the termios structure of the the physical devices used for /dev/console aka the system console.  This seems to be similar to a problem with the X Window System, compare with bug #809806.

Currently systemd uses the plymouth-quit-wait.service with

 [Service]
 ExecStart=-/usr/bin/plymouth --wait
 Type=oneshot
 TimeoutSec=20
 [Install]
 WantedBy=multi-user.target

which is required by serial-getty@.service but it seems not to work.  That is without my patch as workaround the agetty on the first serial console will be fooled by plymouth.
Comment 19 Forgotten User sM9JzehKpy 2014-01-14 04:05:13 UTC
https://bugs.freedesktop.org/show_bug.cgi?id=73585
Comment 20 Forgotten User sM9JzehKpy 2014-01-14 04:06:20 UTC
reported upstream.
Comment 21 Bernhard Wiedemann 2014-01-23 13:00:20 UTC
This is an autogenerated message for OBS integration:
This bug (774126) was mentioned in
https://build.opensuse.org/request/show/214769 Factory / util-linux
Comment 25 Dr. Werner Fink 2014-02-17 10:21:26 UTC
Maybe it should be noted that the current workaround is that now sulogin and agetty uses a loop to wait on TIOCGLCKTRMIOS and after the timeout of 20 seconds for agetty and 5 seconds for sulogin both programs now break the lock with TIOCSLCKTRMIOS. The  tcsendbreak() on stdin I've replaced with sleep() for a second. Also I've added lines to quit and wait on plymouthd in console-shell.service, rescue.service, and emergency.service.
Comment 26 Nathan Cutler 2017-08-15 21:48:51 UTC
openSUSE 12.2 is out of maintenance. It seems like this bug may have been fixed already, but I'll be cautious and move the bug to Tumbleweed instead. Setting needinfo on the assignee for an update.
Comment 27 Dr. Werner Fink 2017-08-16 07:28:49 UTC
IMHO this one is fixed ... to be exact I'm using serial console withut any problems. If this is not fixed for the reporter (Hi Jeff) then please reopen.