Bug 1174534

Apparently emacs/emacsclient don't honour $XAUTHORITY, reassigning.

I branched the openSUSE:Factory/sddm package at Revision 49 <https://build.opensuse.org/package/show/openSUSE:Factory/sddm?rev=49> (which is just before the suspicious commits) and built and installed the package.  Now emacsclient is working again.  So the problem definitely has something to do with the recent changes to sddm (or to emacsclient's incompatibility with them).

I'm using an older Tumblweed with new GNU Emacs and use emacsclient (restarted!) as well and it does wotk

(In reply to Dr. Werner Fink from comment #3)
> I'm using an older Tumblweed with new GNU Emacs and use emacsclient
> (restarted!) as well and it does wotk

With recent enough sddm?

The only difference I have done is to modify my ~/bin/emacs script in such a way the now both wit hand without frame the emacsclient is started now with -c

(In reply to Fabian Vogt from comment #4)
> (In reply to Dr. Werner Fink from comment #3)
> > I'm using an older Tumblweed with new GNU Emacs and use emacsclient
> > (restarted!) as well and it does wotk
> 
> With recent enough sddm?

werner/emacs> rpm -q sddm
sddm-0.18.1-3.1.x86_64

Ahhh ... here I have

 echo $XAUTHORITY 
 /run/user/223/xauth_tAtOOg

 werner/emacs> xauth list
 noether/unix:0  MIT-MAGIC-COOKIE-1  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
 #ffff##:0  MIT-MAGIC-COOKIE-1  XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

(key replaced)

This is IMHO a bug in sddm ... or one of the login scripts used by the user/reporter which overrides XAUTHORITY of the X session (even only partly)

I don't know what emacsclient -c is or does. As xauth extract works, both $XAUTHORITY and its content are correct. So I don't think this is an issue in sddm. If it connects to a "server", maybe that "server" needs to be started with the right $XAUTHORITY already?

(In reply to Fabian Vogt from comment #9)
> I don't know what emacsclient -c is or does. As xauth extract works, both
> $XAUTHORITY and its content are correct. So I don't think this is an issue
> in sddm. If it connects to a "server", maybe that "server" needs to be
> started with the right $XAUTHORITY already?

The emacs server needs the correct MIT-MAGIC-COOKIE-1 for the current DISPLAY ... which Xauthority file is used therefore should not matter  ... the X server gets its Xauthority information from its command line with

    -auth /run/sddm/{......}

The only problem could be if there is NO XAUTHORITY variable set at start time of emacs server/daemon of if XAUTHORITY is set to ~/.Xauthority then the content should be the same as in later XAUTHORITY file

Maybe a

  ps aex | grep /usr/bin/emacs | grep -v grep | tr ' ' '\n' | grep AUTH

does show more

It should also noted that emacsclient its self does not use X nor Xauthority ... only the emacs server/daemon does

I just did a distribution upgrade of Tumbleweed (including sddm) from 20200724 to 20200817 and now I can no longer reproduce the "Display :0 can't be opened" problem.

I am, however, experiencing a different problem with Emacs in that it can no longer access any of my system fonts, leaving me with an ugly bitmap as default.  Attempting to change the font back to one of the system fonts (via Options->Set Default Font…) results in a "Font not available" error in the status bar.

I'm not sure if this font problem is related to the xauth issue.  If not, then the current report can probably be closed (though I would appreciate advice on the font issue, including whether to file a separate bug for it).

(In reply to Tristan Miller from comment #13)
> I am, however, experiencing a different problem with Emacs in that it can no
> longer access any of my system fonts, leaving me with an ugly bitmap as
> default.  Attempting to change the font back to one of the system fonts (via
> Options->Set Default Font…) results in a "Font not available" error in the
> status bar.
> 
> I'm not sure if this font problem is related to the xauth issue.  If not,
> then the current report can probably be closed (though I would appreciate
> advice on the font issue, including whether to file a separate bug for it).

Never mind; it seems this issue was already reported as Bug 1175372.

(In reply to Tristan Miller from comment #13)
> I just did a distribution upgrade of Tumbleweed (including sddm) from
> 20200724 to 20200817 and now I can no longer reproduce the "Display :0 can't
> be opened" problem.

Great, then it might've been fixed by the Apparmor profile change as well.

Btw: if /usr/lib/systemd/user/emacs.service is used ... e.g. by linking into ~/.config/systemd/user/ and enabling it, then a restart might help the current emacs server/daemon to find a new DISPLAY entry in the Xauthority file.  Only if DISPALY :0 on local console is found in a different Xauthority file than e.g. DISPLAY :3 (e.g. as a VNC screen), the export into the Xauthority file used by system service unit helps out.

(In reply to Fabian Vogt from comment #15)
> (In reply to Tristan Miller from comment #13)
> > I just did a distribution upgrade of Tumbleweed (including sddm) from
> > 20200724 to 20200817 and now I can no longer reproduce the "Display :0 can't
> > be opened" problem.
> 
> Great, then it might've been fixed by the Apparmor profile change as well.

The problem has been recurring for me.  In fact, I think it never really went away; it's just unpredictable as to whether or not the service successfully starts on any given boot.  If it does fail to start, then issuing a "systemctl restart --user emacs.service" will fix the problem.

The unit configuration file I'm using isn't much different from the one in /usr/lib/systemd/user/emacs.service.  The only differences are that my file sets a couple environment variables (SSH_AUTH_SOCK and PATH) and uses Type=simple instead of Type=notify.  (But even if I change this to Type=notify, the problem still occurs.)

(In reply to Tristan Miller from comment #17)
> (In reply to Fabian Vogt from comment #15)
> > (In reply to Tristan Miller from comment #13)
> > > I just did a distribution upgrade of Tumbleweed (including sddm) from
> > > 20200724 to 20200817 and now I can no longer reproduce the "Display :0 can't
> > > be opened" problem.
> > 
> > Great, then it might've been fixed by the Apparmor profile change as well.
> 
> The problem has been recurring for me.  In fact, I think it never really
> went away; it's just unpredictable as to whether or not the service
> successfully starts on any given boot.  If it does fail to start, then
> issuing a "systemctl restart --user emacs.service" will fix the problem.
> 
> The unit configuration file I'm using isn't much different from the one in
> /usr/lib/systemd/user/emacs.service.  The only differences are that my file
> sets a couple environment variables (SSH_AUTH_SOCK and PATH) and uses
> Type=simple instead of Type=notify.  (But even if I change this to
> Type=notify, the problem still occurs.)

I'm also using user based emacs.service and I *never* ever had this issue. Never on Tumblweed nor on any Leap. It simply works.

I'm doubt that this is an emacs problem, this looks more like a timing and/or access problem with the (default) xauthority file.  Beside this I do not use apparmor

(In reply to Dr. Werner Fink from comment #19)
> I'm doubt that this is an emacs problem, this looks more like a timing
> and/or access problem with the (default) xauthority file.  Beside this I do
> not use apparmor

Is there anything else I can try then?  Maybe some way of introducing a delay before the emacs service starts, to give the system time to establish the Xauthority file?

Maybe it is more a problem with the used desktop.  I'm using my own ~/.xsession file with fvwm2 and lxsession. And for X Vnc sessions (home office) I'm sourcing the file as well /etc/X11/xinit/xinitrc.common which loads some extensions.

Ok. Which displaymanager, which desktop? How to start emacs Deamon? Once I know this I can try to reproduce on current TW.

Created attachment 843386 [details]
Systemd unit configuration file for emacs daemon

(In reply to Stefan Dirsch from comment #22)
> Ok. Which displaymanager, which desktop? How to start emacs Deamon? Once I
> know this I can try to reproduce on current TW.

I'm using SDDM 0.18.1 to log into KDE Plasma 5.20.2.  The Emacs daemon is started automatically with the attached unit configuration file in $HOME/.config/systemd/user/emacs.service, which I had enabled by invoking "systemctl enable --user emacs".  Note that the problem occurs only when the daemon is started automatically; starting or restarting it manually, once I've already logged in, with "systemctl restart --user emacs.service", always works fine.

Indeed I can reproduce this issue with your emacs.service file in ~/.config/systemd/user/emacs.service - with ssdm, but also with gdm

tux@tw:~> emacsclient -c
Waiting for Emacs...
*ERROR*: Display :1 can’t be opened
tux@tw:~> systemctl stop --user emacs.service
tux@tw:~> systemctl start --user emacs.service
tux@tw:~> emacsclient -c
Waiting for Emacs...
< emacs is starting >

> xauth extract - $DISPLAY | xauth -f ~/.Xauthority merge -

Indeed this also helps. No idea why. Is emacs-server/client hardcoded to use values of ~/.Xauthority somehow? 

And yes, neither sddm nor gdm use ~/.Xauthority  Check $XAUTHORITY

I think, removing ~/.Xauthority also fixes the issue permanently. Not 100% sure. I had mixed results when doing this.

(In reply to Stefan Dirsch from comment #25)
> > xauth extract - $DISPLAY | xauth -f ~/.Xauthority merge -
> 
> Indeed this also helps. No idea why. Is emacs-server/client hardcoded to use
> values of ~/.Xauthority somehow? 
> 
> And yes, neither sddm nor gdm use ~/.Xauthority  Check $XAUTHORITY
> 
> I think, removing ~/.Xauthority also fixes the issue permanently. Not 100%
> sure. I had mixed results when doing this.

AFAIK GNU Emacs behave as all X clients, with XAUTHORITY set in environment this is used, without set the default ~/.Xauthority is used.  Looks like a race as I guess (or better hope) that the environment variable XAUTHORITY is set.

If it is a race then a line like

  ExecStartPre=/bin/bash -c "echo $XAUTHORITY; /usr/bin/sleep 5"

before starting Emacs might help.

Indeed. This would make sense, but it's interesting that this broke for gdm and sddm at the same time.

So, Tristan,  could you please try the workaround in previous comment #26? 

In the end I think it's a corner case. It's the first time I hear anything about systemd User Units ...

(In reply to Stefan Dirsch from comment #27)
> So, Tristan,  could you please try the workaround in previous comment #26? 

No, I'm afraid the workaround from Comment 26 doesn't work for me.

Oh, well. For some reason the emcas systemd User Unit is no longer started for me automatically during login, so I can't investigate this any longer (tried also with freshly created user). Maybe you want to give this line a try

ExecStartPre=/bin/bash -c "echo \"before 5s sleep: XAUTHORITY = $XAUTHORITY\" > $HOME/xauth.txt; /usr/bin/sleep 5; echo \"after 5s sleep: XAUTHORITY = $XAUTHORITY\" >> $HOME/xauth.txt"

And check afterwards the content of your ~/xauth.txt.

Anyway, I think such things, which in the end need a DISPLAY shouldn't be done by a systemd User Unit. Instead better try to use autostart mechanism of your desktop.

Try this little script below in your personal ~/bin/ directory.
Give it the name emacs and do nto forget to run

  chmod u+x ~/bin/emacs

this should do the job. If the daemon is already running
as well if not.

------------------------* snip *------------------------------------------------------
#!/bin/bash
while ! frame=$(emacsclient -n -e "(if (> (length (frame-list)) 1) 't)" 2>/dev/null)
do
    /usr/bin/emacs --daemon 2>/dev/null
    sleep 1
done
for socket in /tmp/emacs${UID}/server /run/user/${UID}/emacs/server
do
    test -n "$DISPLAY" || break
    test -S $socket || continue
    pid=$(fuser $socket 2>/dev/null)
    if test -n "$pid"
    then
        xauth=$(ps e $pid | tr ' ' '\n' | sed -rn 's/XAUTHORITY=//p')
        if test -e "$xauth" -a \( -z "$XAUTHORITY" -o "$XAUTHORITY" != "$xauth" \)
        then
            entry=$(xauth list $DISPLAY)
            xauth -f "$xauth" add $entry
        fi
    fi
done
exec -a emacs /usr/bin/emacsclient -n ${DISPLAY+-d $DISPLAY} -c ${1+"$@"}

Reassigning back to desktop. Let's use GNOME this time, since it's affected as well.

(In reply to Stefan Dirsch from comment #30)
> Oh, well. For some reason the emcas systemd User Unit is no longer started
> for me automatically during login, so I can't investigate this any longer
> (tried also with freshly created user). Maybe you want to give this line a
> try
> 
> ExecStartPre=/bin/bash -c "echo \"before 5s sleep: XAUTHORITY =
> $XAUTHORITY\" > $HOME/xauth.txt; /usr/bin/sleep 5; echo \"after 5s sleep:
> XAUTHORITY = $XAUTHORITY\" >> $HOME/xauth.txt"
> 
> And check afterwards the content of your ~/xauth.txt.

Unfortunately, it seems the variable isn't set even after five seconds:

before 5s sleep: XAUTHORITY = 
after 5s sleep: XAUTHORITY = 

> Anyway, I think such things, which in the end need a DISPLAY shouldn't be
> done by a systemd User Unit. Instead better try to use autostart mechanism
> of your desktop.

If this is true, and sound arguments can be made in support of it, then maybe this should be reported upstream, since running the Emacs daemon as a systemd service has been officially supported since Emacs 26.1 -- this version onwards includes a unit configuration file and documentation on how to use it.  (I'm loath to make any such report myself, since I had been running Emacs this way on openSUSE without a problem for several years, and other GNU/Linux distributions also seem to support or advise running Emacs this way.)

(In reply to Tristan Miller from comment #33)
> Unfortunately, it seems the variable isn't set even after five seconds:

Even using a delay of 20 seconds (by which time my desktop environment has been fully loaded) doesn't work around the problem.  So maybe this is not a race condition after all, but rather something else?

Also reproducible for me with Leap 15.2.

(In reply to Tristan Miller from comment #35)
> Also reproducible for me with Leap 15.2.

Please give the script in comment #31 a try as ~/bin/emacs (don't forget the chmod 755), then call emacs not emacsclient

(In reply to Dr. Werner Fink from comment #36)
> Please give the script in comment #31 a try as ~/bin/emacs (don't forget the
> chmod 755), then call emacs not emacsclient

The script doesn't work for me:

$ ~/bin/emacs
*ERROR*: Display :0 can’t be opened

*** Bug 1179854 has been marked as a duplicate of this bug. ***

I have some more input on this.

GDM creates a static xauthority file in
`XAUTHORITY=/run/user/1000/gdm/Xauthority`.

If I run the user service in a Session which was launched in GNOME it will work
in all consecutive calls emacs, even after restart.

This is thanks to the emacs distribution I am using, Spacemacs. It creates
a ~/.spacemacs.env File which will be loaded upon start by emacs. It contains the XAUTHORITY variable with the static path.

If I am using sddm however, the path to the XAUTHORITY file is changing on each new login. Hence the environment file is not working because the Path to the xauthority file is not the same.
e.g.

First Login with sddm:
XAUTHORITY=/run/user/1000/xauth_xpMWCc

Second Login sddm:
XAUTHORITY=/run/user/1000/xauth_bhgNcT


Now I assume that sddm is responsible for creating the right environment for
the programs and passing it to the GUI applications.
It does so for traditional apps.

But let's compare the environment for the emacsdaemon and a Konsole window.

```
regrau@tumblevm:~> tr '\0' '\n' < /proc/$PID_OF_KONSOLE/environ | wc -l
87
regrau@tumblevm:~> tr '\0' '\n' < /proc/$PID_OF_KONSOLE/environ | grep XAUTH
XAUTHORITY=/run/user/1000/gdm/Xauthority
XAUTHLOCALHOSTNAME=tumblevm

```

```
regrau@tumblevm:~> tr '\0' '\n' < /proc/$PID_OF_EMACSDAEMON/environ | wc -l
18
regrau@tumblevm:~> tr '\0' '\n' < /proc/$PID_OF_EMACSDAEMON/environ | grep XAUTH
```

As one can see the EMACSDAEMON environment does not get the right environment vars
from sddm.

It's not a race condition as adding waiting time to emacsdaemon service unit does
help. I've gone up as high as 5 Minutes before starting up the service. It did
not help.


So there has to be a way to start the unit after the initialization of the
environment is completed or to pass the environment to the emacs daemon afterwards.

My TW is 20201119. I works on GNOME, the displaymanager is gdm.
After I installed emacs, and run 'systemctl --user enable emace.service', restart system. Then I can run 'emacsclient -c' without issue.

You can try this: before run 'emacsclient -c', run command
dbus-update-activation-environment --systemd --all

(In reply to xiaoguang wang from comment #40)
> My TW is 20201119. I works on GNOME, the displaymanager is gdm.
> After I installed emacs, and run 'systemctl --user enable emace.service',
> restart system. Then I can run 'emacsclient -c' without issue.
> 
> You can try this: before run 'emacsclient -c', run command
> dbus-update-activation-environment --systemd --all

For my VNC sessions I use

```
if test -n "$VNCDESKTOP" -a -z "$RESTART"; then
    exec > "$HOME/.vnc/${VNCDESKTOP}.log" 2>&1
    unset SESSION_MANAGER
    unset DBUS_SESSION_BUS_ADDRESS
    test -e /etc/X11/xinit/xinitrc.common && . /etc/X11/xinit/xinitrc.common
fi
```

in my personal ~/.xsession used as VNC xstartup and this xinitrc.common will also load

```
/etc/X11/xinit/xinitrc.d/50-systemd-user.sh
```

which does use dbus-update-activation-environment ...

I've added a patch to GNU Emacs which does start the GNU Emacs server with the environment variable XAUTHORITY set to /run/user/<UID>/emacs/xauth in emacs.service as well as an extended server.el to extract the Xauth cookie of the current Display and merge this to the new XAUTHORITY file. This should work not only for default ~/.Xauthority but also for gdm and sddm with their session value of XAUTHORITY 

Testers are welcome, see OBS editors/emacs

(In reply to Dr. Werner Fink from comment #42)
> I've added a patch to GNU Emacs which does start the GNU Emacs server with
> the environment variable XAUTHORITY set to /run/user/<UID>/emacs/xauth in
> emacs.service as well as an extended server.el to extract the Xauth cookie
> of the current Display and merge this to the new XAUTHORITY file. This
> should work not only for default ~/.Xauthority but also for gdm and sddm
> with their session value of XAUTHORITY 
> 
> Testers are welcome, see OBS editors/emacs

So I tried it right now, unfortunately no changes of the behaviour.

Is there any input I can provide to aid?

(In reply to Georg Grauberger from comment #43)
> (In reply to Dr. Werner Fink from comment #42)
> > I've added a patch to GNU Emacs which does start the GNU Emacs server with
> > the environment variable XAUTHORITY set to /run/user/<UID>/emacs/xauth in
> > emacs.service as well as an extended server.el to extract the Xauth cookie
> > of the current Display and merge this to the new XAUTHORITY file. This
> > should work not only for default ~/.Xauthority but also for gdm and sddm
> > with their session value of XAUTHORITY 
> > 
> > Testers are welcome, see OBS editors/emacs
> 
> So I tried it right now, unfortunately no changes of the behaviour.
> 
> Is there any input I can provide to aid?

Does the ~/.config/systemd/user/emacs.service file include the line

  Environment=XAUTHORITY=%t/emacs/xauth

if not it can not work.  Also check changelog ...

 * Mon Mar 08 2021 Dr. Werner Fink <werner@suse.de>
 - Add patch emacs-27.1-Xauthority4server.patch
   * Allow GNU Emacs server to open X Display even if the Xauthority
     file is not the default expected by XCloseDisplay()
   * Hopefully fix boo#1174534 and boo#1179854

(In reply to Dr. Werner Fink from comment #44)
> (In reply to Georg Grauberger from comment #43)
> > (In reply to Dr. Werner Fink from comment #42)
> > > I've added a patch to GNU Emacs which does start the GNU Emacs server with
> > > the environment variable XAUTHORITY set to /run/user/<UID>/emacs/xauth in
> > > emacs.service as well as an extended server.el to extract the Xauth cookie
> > > of the current Display and merge this to the new XAUTHORITY file. This
> > > should work not only for default ~/.Xauthority but also for gdm and sddm
> > > with their session value of XAUTHORITY 
> > > 
> > > Testers are welcome, see OBS editors/emacs
> > 
> > So I tried it right now, unfortunately no changes of the behaviour.
> > 
> > Is there any input I can provide to aid?
> 
> Does the ~/.config/systemd/user/emacs.service file include the line
> 
>   Environment=XAUTHORITY=%t/emacs/xauth
> 
> if not it can not work.  Also check changelog ...
> 
>  * Mon Mar 08 2021 Dr. Werner Fink <werner@suse.de>
>  - Add patch emacs-27.1-Xauthority4server.patch
>    * Allow GNU Emacs server to open X Display even if the Xauthority
>      file is not the default expected by XCloseDisplay()
>    * Hopefully fix boo#1174534 and boo#1179854

You're right the line was missing. I added it but the error is still the same.
Waiting for Emacs...
*ERROR*: Display :0 can’t be opened


For an installation on latest tumbleweed with KDE and default sddm.

(In reply to Dr. Werner Fink from comment #42)
> I've added a patch to GNU Emacs which does start the GNU Emacs server with
> the environment variable XAUTHORITY set to /run/user/<UID>/emacs/xauth in
> emacs.service as well as an extended server.el to extract the Xauth cookie
> of the current Display and merge this to the new XAUTHORITY file. This
> should work not only for default ~/.Xauthority but also for gdm and sddm
> with their session value of XAUTHORITY 
> 
> Testers are welcome, see OBS editors/emacs

This version seems works for me on Tumbleweed; thanks!

(In reply to Georg Grauberger from comment #45)
> (In reply to Dr. Werner Fink from comment #44)
> > Does the ~/.config/systemd/user/emacs.service file include the line
> > 
> >   Environment=XAUTHORITY=%t/emacs/xauth
> 
> You're right the line was missing. I added it but the error is still the
> same.
> Waiting for Emacs...
> *ERROR*: Display :0 can’t be opened

Perhaps a stupid question, but did you reboot after making the change to your emacs.service file?

(In reply to Tristan Miller from comment #46)

> Perhaps a stupid question, but did you reboot after making the change to
> your emacs.service file?

Not stupid at all :) that's the troubleshooting 101.

I did, but the restart did not help.

I have multiple ways to test with prepared VM's. So I will investigate further and see if I get it running.

(In reply to Georg Grauberger from comment #47)
> (In reply to Tristan Miller from comment #46)
> 
> > Perhaps a stupid question, but did you reboot after making the change to
> > your emacs.service file?
> 
> Not stupid at all :) that's the troubleshooting 101.
> 
> I did, but the restart did not help.
> 
> I have multiple ways to test with prepared VM's. So I will investigate
> further and see if I get it running.

Guess only: You might have caught the emacs before the changed version had finished its builds ...

 /suse/werner> ll -G ~/.config/systemd/user/emacs.service
 lrwxrwxrwx 1 werner 35 May  8  2019 /suse/werner/.config/systemd/user/emacs.service -> /usr/lib/systemd/user/emacs.service
 /suse/werner> rpm -q --changelog -f /usr/lib/systemd/user/emacs.service | head -n 4
 * Mon Mar 08 2021 Dr. Werner Fink <werner@suse.de>
 - Add patch emacs-27.1-Xauthority4server.patch
   * Allow GNU Emacs server to open X Display even if the Xauthority
     file is not the default expected by XCloseDisplay()

For debugging I suggest to see which service is active

  systemctl --user status emacs.service

make sure to use latest /usr/lib/systemd/user/emacs.service
then you might restart it

  systemctl --user restart emacs.service

then have a look at the process table (or redo status) to get major pid
of the emacs daemon

  pidof emacs

with this pid you can attach a strace with

  strace -o emacs.trace -s 256 -f -p $(pidof emacs)

and then start in an other kconsole/xterm/gnome-terminal

  emacsclient -c

then stop the strace command with Ctrl-C ... you might attach this trace

No answer yet