Bug 463465

Summary:	CIFS: cifsd hangs in uninterruptible sleep
Product:	[openSUSE] openSUSE 11.1	Reporter:	florian florian <novell.com>
Component:	Kernel	Assignee:	E-mail List <kernel-maintainers>
Status:	RESOLVED FIXED	QA Contact:	E-mail List <qa-bugs>
Severity:	Normal
Priority:	P5 - None	CC:	christian.fertig, forgotten_lMRCKDTytB, tt
Version:	Final
Target Milestone:	---
Hardware:	All
OS:	openSUSE 11.1
URL:	https://bugzilla.samba.org/show_bug.cgi?id=5989
Whiteboard:
Found By:	Customer	Services Priority:
Business Priority:		Blocker:	---
Marketing QA Status:	---	IT Deployment:	---

Description florian florian 2009-01-04 20:28:57 UTC

After mounting and unmounting a remote cifs share, three cifs kernel processes stay in the process table:

$ ps aux
[...]
root     11351  0.0  0.0      0     0 ?        S<   22:30   0:00 [cifsoplockd]
root     11352  0.0  0.0      0     0 ?        S<   22:30   0:00 [cifsdnotifyd]
root     11357  0.7  0.0      0     0 ?        D<   22:30   0:00 [cifsd]
[...]

cifsd is in an uninterruptible sleep waiting for i/o (but what i/o?), without
any open files:

$ lsof -p 11357
COMMAND   PID USER   FD      TYPE DEVICE SIZE/OFF NODE NAME
cifsd   11357 root  cwd       DIR   22,3      536    2 /
cifsd   11357 root  rtd       DIR   22,3      536    2 /
cifsd   11357 root  txt   unknown                      /proc/11357/exe

In this state, cifsd is not interruptible by any signals, of course. An attempt to kill it by

$ modprobe -r cifs

results in removing the kernel module (!) as well as the sleeping processes
cifsoplockd and cifsdnotifyd, but puts this error message into the syslog:

Dec 23 22:36:04 lex kernel: slab error in kmem_cache_destroy(): cache
`cifs_request': Can't free all
Dec 23 22:36:04 lex kernel: Pid: 11436, comm: modprobe Tainted: G         
2.6.27.7-9-default #1
Dec 23 22:36:04 lex kernel:  [<c0106570>] dump_trace+0x6b/0x249
Dec 23 22:36:04 lex kernel:  [<c01070a5>] show_trace+0x20/0x39
Dec 23 22:36:04 lex kernel:  [<c0343c02>] dump_stack+0x71/0x76
Dec 23 22:36:04 lex kernel:  [<c018c376>] kmem_cache_destroy+0x88/0xd6
Dec 23 22:36:04 lex kernel:  [<d10dc84e>] cifs_destroy_request_bufs+0x14/0x28
[cifs]
Dec 23 22:36:04 lex kernel:  [<d10fc79c>] exit_cifs+0x3c/0xc8 [cifs]
Dec 23 22:36:04 lex kernel:  [<c014cbf9>] sys_delete_module+0x1ce/0x228
Dec 23 22:36:04 lex kernel:  [<c0104d92>] syscall_call+0x7/0xb
Dec 23 22:36:04 lex kernel:  [<b7ec7d24>] 0xb7ec7d24
Dec 23 22:36:04 lex kernel:  =======================
Dec 23 22:36:04 lex kernel: slab error in kmem_cache_destroy(): cache
`cifs_small_rq': Can't free all
Dec 23 22:36:04 lex kernel: Pid: 11436, comm: modprobe Tainted: G         
2.6.27.7-9-default #1
Dec 23 22:36:04 lex kernel:  [<c0106570>] dump_trace+0x6b/0x249
Dec 23 22:36:04 lex kernel:  [<c01070a5>] show_trace+0x20/0x39
Dec 23 22:36:04 lex kernel:  [<c0343c02>] dump_stack+0x71/0x76
Dec 23 22:36:04 lex kernel:  [<c018c376>] kmem_cache_destroy+0x88/0xd6
Dec 23 22:36:04 lex kernel:  [<d10fc79c>] exit_cifs+0x3c/0xc8 [cifs]
Dec 23 22:36:04 lex kernel:  [<c014cbf9>] sys_delete_module+0x1ce/0x228
Dec 23 22:36:04 lex kernel:  [<c0104d92>] syscall_call+0x7/0xb
Dec 23 22:36:04 lex kernel:  [<b7ec7d24>] 0xb7ec7d24
Dec 23 22:36:04 lex kernel:  =======================

Re-inserting the module fails:

Dec 23 22:43:42 lex kernel: kmem_cache_create: duplicate cache cifs_request
Dec 23 22:43:42 lex kernel: Pid: 11472, comm: modprobe Tainted: G         
2.6.27.7-9-default #1
Dec 23 22:43:42 lex kernel:  [<c0106570>] dump_trace+0x6b/0x249
Dec 23 22:43:42 lex kernel:  [<c01070a5>] show_trace+0x20/0x39
Dec 23 22:43:42 lex kernel:  [<c0343c02>] dump_stack+0x71/0x76
Dec 23 22:43:42 lex kernel:  [<c018c4a0>] kmem_cache_create+0xdc/0x3c4
Dec 23 22:43:42 lex kernel:  [<d10dc8d0>] cifs_init_request_bufs+0x5c/0x191
[cifs]
Dec 23 22:43:42 lex kernel:  [<d107c35d>] init_cifs+0x35d/0x368 [cifs]
Dec 23 22:43:42 lex kernel:  [<c010112b>] _stext+0x3b/0x127
Dec 23 22:43:42 lex kernel:  [<c014c8bf>] sys_init_module+0x8a/0x19e
Dec 23 22:43:42 lex kernel:  [<c0104d92>] syscall_call+0x7/0xb
Dec 23 22:43:42 lex kernel:  [<b7e6ff0e>] 0xb7e6ff0e
Dec 23 22:43:42 lex kernel:  =======================
Dec 23 22:43:42 lex modprobe: FATAL: Error inserting cifs
(/lib/modules/2.6.27.7-9-default/kernel/fs/
Dec 23 22:43:50 lex kernel: kmem_cache_create: duplicate cache cifs_request
Dec 23 22:43:50 lex kernel: Pid: 11478, comm: modprobe Tainted: G         
2.6.27.7-9-default #1
Dec 23 22:43:50 lex kernel:  [<c0106570>] dump_trace+0x6b/0x249
Dec 23 22:43:50 lex kernel:  [<c01070a5>] show_trace+0x20/0x39
Dec 23 22:43:50 lex kernel:  [<c0343c02>] dump_stack+0x71/0x76
Dec 23 22:43:50 lex kernel:  [<c018c4a0>] kmem_cache_create+0xdc/0x3c4
Dec 23 22:43:50 lex kernel:  [<d10dc8d0>] cifs_init_request_bufs+0x5c/0x191
[cifs]
Dec 23 22:43:50 lex kernel:  [<d107c35d>] init_cifs+0x35d/0x368 [cifs]
Dec 23 22:43:50 lex kernel:  [<c010112b>] _stext+0x3b/0x127
Dec 23 22:43:50 lex kernel:  [<c014c8bf>] sys_init_module+0x8a/0x19e
Dec 23 22:43:50 lex kernel:  [<c0104d92>] syscall_call+0x7/0xb
Dec 23 22:43:50 lex kernel:  [<b7f54f0e>] 0xb7f54f0e
Dec 23 22:43:50 lex kernel:  =======================

After rebooting, I could mount cifs shares again. 'Mount, unmount, mount, unmount, mount' results in two "uninterruptible sleep" cifsd processes (remainder of first and second unmount) and one sleeping cifsd process (responsible for the third mounted, not yet unmounted share).

I already reported this bug upstream <https://bugzilla.samba.org/show_bug.cgi?id=5989>, but today I noticed that a self-compiled plain vanilla 2.6.28 kernel works fine: The cifsd kernel process disappears after unmounting all cifs shares.

I believe that the uninterruptible cifsd process is responsible for hibernation failing, but I am not sure yet (more testing required).

Comment 1 Christian Fertig 2009-01-06 15:47:52 UTC

Additionally, after several mounts, the load value increases dramatically, allthough the feeled load is normal (idle in this case)

top - 16:46:12 up 4 days, 18:41,  1 user,  load average: 30.00, 30.00, 29.95
                                                         ^^^^^^

Tasks: 109 total,   1 running, 108 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.3%us,  0.0%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:    513248k total,   487764k used,    25484k free,   102980k buffers
Swap:  1052216k total,      140k used,  1052076k free,   339828k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
    1 root      20   0  1008  360  308 S  0.0  0.1   0:03.40 init
[..]
  458 root      15  -5     0    0    0 D  0.0  0.0   0:00.00 cifsd
  471 root      15  -5     0    0    0 D  0.0  0.0   0:00.02 cifsd
  484 root      15  -5     0    0    0 D  0.0  0.0   0:00.00 cifsd
  497 root      15  -5     0    0    0 D  0.0  0.0   0:00.00 cifsd
  510 root      15  -5     0    0    0 D  0.0  0.0   0:00.00 cifsd

Comment 2 Christian Fertig 2009-01-06 15:51:08 UTC

x86_64 kernel-default is affected too

Comment 3 Greg Kroah-Hartman 2009-01-26 00:41:00 UTC

Should now be resolved in the latest kernel-of-the-day, the updated kernel will be pushed through the update channels soon.

Comment 4 Forgotten User lMRCKDTytB 2009-02-23 18:29:57 UTC

I am affected by this bug as well.