Bug 1111373

Created attachment 785615 [details]
winbind log

Created attachment 785616 [details]
smb config

Can you set:
log level = 2 winbind:10
in your smb.conf and get the winbind debug from startup again?

Created attachment 785785 [details]
winbind log

This is the winbind log after setting the loglevel = 2 winbind:10 in smb.conf

(In reply to Philip Raets from comment #4)
> Created attachment 785785 [details]
> winbind log
> 
> This is the winbind log after setting the loglevel = 2 winbind:10 in smb.conf

Well, that didn't provide any additional debug. I'll have to see if I can reproduce it.

I just found out this happens when I get an IP from the DHCP-server and samba is started before an ip-address is given.

If I have a fixed IP, samba doesn't crash (but don't want to have a fixed ip).

My network:

bridge0: connected to Boma
        "bridge0"
        bridge, D4:BE:D9:A1:09:C8, sw, mtu 1500
        ip4 default
        inet4 192.168.10.85/24
        route4 0.0.0.0/0
        route4 192.168.10.0/24

eth0: connected to bridge0 slave 1
        "Intel 82579LM"
        ethernet (e1000e), D4:BE:D9:A1:09:C8, hw, mtu 1500
        master bridge0
        route6 ff00::/8

vnet0: connected to vnet0
        "vnet0"
        tun, FE:54:00:99:08:3C, sw, mtu 1500
        master bridge0

lo: unmanaged
        "lo"
        loopback (unknown), 00:00:00:00:00:00, sw, mtu 65536

Winbind is configured to not start until the network-online target is reached, though what this means depends on the network service.
Which network service are you using (wicked/networkmanager)?

I am using NetworkManager (bridge created with nm-connection-editor, which is used for kvm)

Also, try adding this line to the Service section of /usr/lib/systemd/system/winbind.service:
ExecStartPre=/bin/sh -c 'until ping -c1 google.com; do sleep 1; done;'

This is just to test if winbind starts if we delay until you get a dhcp address. I suspect you have a slow dhcp server (although winbind should not crash when you haven't gotten an address yet).

After adding the line, winbind now waits and starts without crashing (was pinging an internal address):

PRAETS:/home/BOMA/praets # journalctl -b -u winbind.service 
-- Logs begin at Wed 2016-12-21 15:46:32 CET, end at Tue 2018-10-23 16:33:11 CEST. --
okt 23 16:26:52 PRAETS.boma.local systemd[1]: Starting Samba Winbind Daemon...
okt 23 16:26:52 PRAETS.boma.local sh[4118]: connect: Network is unreachable
okt 23 16:26:53 PRAETS.boma.local sh[4118]: connect: Network is unreachable
okt 23 16:26:54 PRAETS.boma.local sh[4118]: connect: Network is unreachable
okt 23 16:26:55 PRAETS.boma.local sh[4118]: connect: Network is unreachable
okt 23 16:26:56 PRAETS.boma.local sh[4118]: connect: Network is unreachable
okt 23 16:26:57 PRAETS.boma.local sh[4118]: connect: Network is unreachable
okt 23 16:26:58 PRAETS.boma.local sh[4118]: connect: Network is unreachable
okt 23 16:26:59 PRAETS.boma.local sh[4118]: connect: Network is unreachable
okt 23 16:27:00 PRAETS.boma.local sh[4118]: connect: Network is unreachable
okt 23 16:27:01 PRAETS.boma.local sh[4118]: PING 10.10.255.2 (10.10.255.2) 56(84) bytes of data.
okt 23 16:27:01 PRAETS.boma.local sh[4118]: 64 bytes from 10.10.255.2: icmp_seq=1 ttl=64 time=0.608 ms
okt 23 16:27:01 PRAETS.boma.local sh[4118]: --- 10.10.255.2 ping statistics ---
okt 23 16:27:01 PRAETS.boma.local sh[4118]: 1 packets transmitted, 1 received, 0% packet loss, time 0ms
okt 23 16:27:01 PRAETS.boma.local sh[4118]: rtt min/avg/max/mdev = 0.608/0.608/0.608/0.000 ms
okt 23 16:27:02 PRAETS.boma.local winbindd[4334]: [2018/10/23 16:27:02.014722,  0, pid=4334, effective(0, 0), real(0, 0)] ../lib/util/become_daemon.c:138(daemon_ready)
okt 23 16:27:02 PRAETS.boma.local winbindd[4334]:   daemon_ready: STATUS=daemon 'winbindd' finished starting up and ready to serve connections
okt 23 16:27:02 PRAETS.boma.local systemd[1]: Started Samba Winbind Daemon.

Can you enable the repo-debug repo, then install samba-winbind-debuginfo? Also install gdb for getting a backtrace:
sudo zypper mr -e repo-debug && sudo zypper ref
sudo zypper in samba-winbind-debuginfo gdb

Reproduce the failure, then get a backtrace from systemd:

# coredumpctl gdb winbindd
(gdb) bt
#0  0x00007fbc00b8508b in raise () from /lib64/libc.so.6
etc...

this is the output:

#0  0x00007f7a5909a08b in raise () from /lib64/libc.so.6
#1  0x00007f7a590834e9 in abort () from /lib64/libc.so.6
#2  0x00007f7a59b63bf3 in dump_core () from /usr/lib64/libsmbconf.so.0
#3  0x00007f7a59b54beb in smb_panic_s3 () from /usr/lib64/libsmbconf.so.0
#4  0x00007f7a5a86231f in smb_panic () from /usr/lib64/libsamba-util.so.0
#5  0x00007f7a5a862556 in ?? () from /usr/lib64/libsamba-util.so.0
#6  <signal handler called>
#7  0x00007f7a59105374 in __memmove_sse2_unaligned_erms () from /lib64/libc.so.6
#8  0x00007f7a5a860ed1 in smb_memdup () from /usr/lib64/libsamba-util.so.0
#9  0x00007f7a59b652f0 in load_interfaces () from /usr/lib64/libsmbconf.so.0
#10 0x00005634d7cc48f0 in reload_services_file (lfile=0x0) at ../source3/winbindd/winbindd.c:124
#11 0x00005634d7cc1221 in main (argc=<optimized out>, argv=<optimized out>) at ../source3/winbindd/winbindd.c:1752

(In reply to Philip Raets from comment #12)
> this is the output:
> 
> #0  0x00007f7a5909a08b in raise () from /lib64/libc.so.6
> #1  0x00007f7a590834e9 in abort () from /lib64/libc.so.6
> #2  0x00007f7a59b63bf3 in dump_core () from /usr/lib64/libsmbconf.so.0
> #3  0x00007f7a59b54beb in smb_panic_s3 () from /usr/lib64/libsmbconf.so.0
> #4  0x00007f7a5a86231f in smb_panic () from /usr/lib64/libsamba-util.so.0
> #5  0x00007f7a5a862556 in ?? () from /usr/lib64/libsamba-util.so.0
> #6  <signal handler called>
> #7  0x00007f7a59105374 in __memmove_sse2_unaligned_erms () from
> /lib64/libc.so.6
> #8  0x00007f7a5a860ed1 in smb_memdup () from /usr/lib64/libsamba-util.so.0
> #9  0x00007f7a59b652f0 in load_interfaces () from /usr/lib64/libsmbconf.so.0
> #10 0x00005634d7cc48f0 in reload_services_file (lfile=0x0) at
> ../source3/winbindd/winbindd.c:124
> #11 0x00005634d7cc1221 in main (argc=<optimized out>, argv=<optimized out>)
> at ../source3/winbindd/winbindd.c:1752

Perfect. That's what I was looking for.

(In reply to Philip Raets from comment #0)
> Created attachment 785614 [details]
> smb service log
> 
> I log on with a domain user in Tumbleweed, but after the upgrade to samba
> 4.9.1  the smb and winbind services don't start at boot, they crash. I have
> to log on with a local user, start the services manually, and then they
> start fine. After this I can log on with a domain user.
> 
> Nothing changed in smb.conf

What version of samba did you upgrade from? 4.8.x? 4.7.x?

I found a recent change that causes load_interfaces() to memdup NULL when only 1 interface is present (such as when the only interface that's up is lo), this would cause a segfault. I suspect that may be what we're seeing here.

I'm building a test package and would like you to test it when it's ready (details will follow).

The test build is running here:
https://build.opensuse.org/package/show/home:dmulder:4.9.1-get_interfaces-segfault/samba

When the build completes, you should be able to install and test it by adding my repository and duping it:
sudo zypper ar https://download.opensuse.org/repositories/home:/dmulder:/4.9.1-get_interfaces-segfault/openSUSE_Tumbleweed dmulder-samba
sudo zypper dup -r dmulder-samba

I just zypper dup once or twice a week, so it upgraded from the previous version of samba in Tumbleweed.

I updated to your packages, and they work, winbind and samba are started at boot.

Maybe unrelated:
but I have a virtual machine (with KVM) which is set to autostart at boot, and this fails now to start at boot. (I can start it after I log in)

If I revert the packages back to the original ones, and apply the workaround in /usr/lib/systemd/system/winbind.service, my virtual machine starts fine at boot.

(In reply to Philip Raets from comment #17)
> Maybe unrelated:
> but I have a virtual machine (with KVM) which is set to autostart at boot,
> and this fails now to start at boot. (I can start it after I log in)
> 
> If I revert the packages back to the original ones, and apply the workaround
> in /usr/lib/systemd/system/winbind.service, my virtual machine starts fine
> at boot.

Is there crash at startup, or is it simply not starting the process? You say you revert the packages, what do you mean? Are you reverting 4.9.1 to 4.8.x? Are you talking about the test packages I built for you? You didn't specify.

Could you collect debug, etc? If there is a crash, follow the steps I gave you before and collect a crash dump.

(In reply to David Mulder from comment #18)
> (In reply to Philip Raets from comment #17)
> > Maybe unrelated:
> > but I have a virtual machine (with KVM) which is set to autostart at boot,
> > and this fails now to start at boot. (I can start it after I log in)
> > 
> > If I revert the packages back to the original ones, and apply the workaround
> > in /usr/lib/systemd/system/winbind.service, my virtual machine starts fine
> > at boot.
> 
> Is there crash at startup, or is it simply not starting the process? You say
> you revert the packages, what do you mean? Are you reverting 4.9.1 to 4.8.x?
> Are you talking about the test packages I built for you? You didn't specify.
> 
> Could you collect debug, etc? If there is a crash, follow the steps I gave
> you before and collect a crash dump.

Oh, I just realized you mean the *VM* doesn't start at boot, not *samba*. It's probably a related issue. I'd guess there's a bug in network manager that is causing processes to start before network up. It just happened to bring a different bug in samba to the surface. You should file a separate bug against network manager.

I've posted a bug upstream here:
https://bugzilla.samba.org/show_bug.cgi?id=13665

And the patch has been reviewed and should land in master soon. It should end up in 4.9.1 (tumbleweed) shortly after.

Fixed as of 4.9.2