Bugzilla – Bug 463465
CIFS: cifsd hangs in uninterruptible sleep
Last modified: 2009-02-23 18:29:57 UTC
After mounting and unmounting a remote cifs share, three cifs kernel processes stay in the process table: $ ps aux [...] root 11351 0.0 0.0 0 0 ? S< 22:30 0:00 [cifsoplockd] root 11352 0.0 0.0 0 0 ? S< 22:30 0:00 [cifsdnotifyd] root 11357 0.7 0.0 0 0 ? D< 22:30 0:00 [cifsd] [...] cifsd is in an uninterruptible sleep waiting for i/o (but what i/o?), without any open files: $ lsof -p 11357 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME cifsd 11357 root cwd DIR 22,3 536 2 / cifsd 11357 root rtd DIR 22,3 536 2 / cifsd 11357 root txt unknown /proc/11357/exe In this state, cifsd is not interruptible by any signals, of course. An attempt to kill it by $ modprobe -r cifs results in removing the kernel module (!) as well as the sleeping processes cifsoplockd and cifsdnotifyd, but puts this error message into the syslog: Dec 23 22:36:04 lex kernel: slab error in kmem_cache_destroy(): cache `cifs_request': Can't free all Dec 23 22:36:04 lex kernel: Pid: 11436, comm: modprobe Tainted: G 2.6.27.7-9-default #1 Dec 23 22:36:04 lex kernel: [<c0106570>] dump_trace+0x6b/0x249 Dec 23 22:36:04 lex kernel: [<c01070a5>] show_trace+0x20/0x39 Dec 23 22:36:04 lex kernel: [<c0343c02>] dump_stack+0x71/0x76 Dec 23 22:36:04 lex kernel: [<c018c376>] kmem_cache_destroy+0x88/0xd6 Dec 23 22:36:04 lex kernel: [<d10dc84e>] cifs_destroy_request_bufs+0x14/0x28 [cifs] Dec 23 22:36:04 lex kernel: [<d10fc79c>] exit_cifs+0x3c/0xc8 [cifs] Dec 23 22:36:04 lex kernel: [<c014cbf9>] sys_delete_module+0x1ce/0x228 Dec 23 22:36:04 lex kernel: [<c0104d92>] syscall_call+0x7/0xb Dec 23 22:36:04 lex kernel: [<b7ec7d24>] 0xb7ec7d24 Dec 23 22:36:04 lex kernel: ======================= Dec 23 22:36:04 lex kernel: slab error in kmem_cache_destroy(): cache `cifs_small_rq': Can't free all Dec 23 22:36:04 lex kernel: Pid: 11436, comm: modprobe Tainted: G 2.6.27.7-9-default #1 Dec 23 22:36:04 lex kernel: [<c0106570>] dump_trace+0x6b/0x249 Dec 23 22:36:04 lex kernel: [<c01070a5>] show_trace+0x20/0x39 Dec 23 22:36:04 lex kernel: [<c0343c02>] dump_stack+0x71/0x76 Dec 23 22:36:04 lex kernel: [<c018c376>] kmem_cache_destroy+0x88/0xd6 Dec 23 22:36:04 lex kernel: [<d10fc79c>] exit_cifs+0x3c/0xc8 [cifs] Dec 23 22:36:04 lex kernel: [<c014cbf9>] sys_delete_module+0x1ce/0x228 Dec 23 22:36:04 lex kernel: [<c0104d92>] syscall_call+0x7/0xb Dec 23 22:36:04 lex kernel: [<b7ec7d24>] 0xb7ec7d24 Dec 23 22:36:04 lex kernel: ======================= Re-inserting the module fails: Dec 23 22:43:42 lex kernel: kmem_cache_create: duplicate cache cifs_request Dec 23 22:43:42 lex kernel: Pid: 11472, comm: modprobe Tainted: G 2.6.27.7-9-default #1 Dec 23 22:43:42 lex kernel: [<c0106570>] dump_trace+0x6b/0x249 Dec 23 22:43:42 lex kernel: [<c01070a5>] show_trace+0x20/0x39 Dec 23 22:43:42 lex kernel: [<c0343c02>] dump_stack+0x71/0x76 Dec 23 22:43:42 lex kernel: [<c018c4a0>] kmem_cache_create+0xdc/0x3c4 Dec 23 22:43:42 lex kernel: [<d10dc8d0>] cifs_init_request_bufs+0x5c/0x191 [cifs] Dec 23 22:43:42 lex kernel: [<d107c35d>] init_cifs+0x35d/0x368 [cifs] Dec 23 22:43:42 lex kernel: [<c010112b>] _stext+0x3b/0x127 Dec 23 22:43:42 lex kernel: [<c014c8bf>] sys_init_module+0x8a/0x19e Dec 23 22:43:42 lex kernel: [<c0104d92>] syscall_call+0x7/0xb Dec 23 22:43:42 lex kernel: [<b7e6ff0e>] 0xb7e6ff0e Dec 23 22:43:42 lex kernel: ======================= Dec 23 22:43:42 lex modprobe: FATAL: Error inserting cifs (/lib/modules/2.6.27.7-9-default/kernel/fs/ Dec 23 22:43:50 lex kernel: kmem_cache_create: duplicate cache cifs_request Dec 23 22:43:50 lex kernel: Pid: 11478, comm: modprobe Tainted: G 2.6.27.7-9-default #1 Dec 23 22:43:50 lex kernel: [<c0106570>] dump_trace+0x6b/0x249 Dec 23 22:43:50 lex kernel: [<c01070a5>] show_trace+0x20/0x39 Dec 23 22:43:50 lex kernel: [<c0343c02>] dump_stack+0x71/0x76 Dec 23 22:43:50 lex kernel: [<c018c4a0>] kmem_cache_create+0xdc/0x3c4 Dec 23 22:43:50 lex kernel: [<d10dc8d0>] cifs_init_request_bufs+0x5c/0x191 [cifs] Dec 23 22:43:50 lex kernel: [<d107c35d>] init_cifs+0x35d/0x368 [cifs] Dec 23 22:43:50 lex kernel: [<c010112b>] _stext+0x3b/0x127 Dec 23 22:43:50 lex kernel: [<c014c8bf>] sys_init_module+0x8a/0x19e Dec 23 22:43:50 lex kernel: [<c0104d92>] syscall_call+0x7/0xb Dec 23 22:43:50 lex kernel: [<b7f54f0e>] 0xb7f54f0e Dec 23 22:43:50 lex kernel: ======================= After rebooting, I could mount cifs shares again. 'Mount, unmount, mount, unmount, mount' results in two "uninterruptible sleep" cifsd processes (remainder of first and second unmount) and one sleeping cifsd process (responsible for the third mounted, not yet unmounted share). I already reported this bug upstream <https://bugzilla.samba.org/show_bug.cgi?id=5989>, but today I noticed that a self-compiled plain vanilla 2.6.28 kernel works fine: The cifsd kernel process disappears after unmounting all cifs shares. I believe that the uninterruptible cifsd process is responsible for hibernation failing, but I am not sure yet (more testing required).
Additionally, after several mounts, the load value increases dramatically, allthough the feeled load is normal (idle in this case) top - 16:46:12 up 4 days, 18:41, 1 user, load average: 30.00, 30.00, 29.95 ^^^^^^ Tasks: 109 total, 1 running, 108 sleeping, 0 stopped, 0 zombie Cpu(s): 0.3%us, 0.0%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 513248k total, 487764k used, 25484k free, 102980k buffers Swap: 1052216k total, 140k used, 1052076k free, 339828k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1 root 20 0 1008 360 308 S 0.0 0.1 0:03.40 init [..] 458 root 15 -5 0 0 0 D 0.0 0.0 0:00.00 cifsd 471 root 15 -5 0 0 0 D 0.0 0.0 0:00.02 cifsd 484 root 15 -5 0 0 0 D 0.0 0.0 0:00.00 cifsd 497 root 15 -5 0 0 0 D 0.0 0.0 0:00.00 cifsd 510 root 15 -5 0 0 0 D 0.0 0.0 0:00.00 cifsd
x86_64 kernel-default is affected too
Should now be resolved in the latest kernel-of-the-day, the updated kernel will be pushed through the update channels soon.
I am affected by this bug as well.