Bug 689414

Summary:	Repeatable NFS mount hang
Product:	[openSUSE] openSUSE 11.3	Reporter:	Jeffrey Katcher <jmkatcher>
Component:	Kernel	Assignee:	Neil Brown <nfbrown>
Status:	RESOLVED FIXED	QA Contact:	E-mail List <qa-bugs>
Severity:	Critical
Priority:	P5 - None	CC:	jeffm, jmkatcher
Version:	Final
Target Milestone:	---
Hardware:	x86-64
OS:	openSUSE 11.3
Whiteboard:	maint:released:11.3:42359
Found By:	---	Services Priority:
Business Priority:		Blocker:	---
Marketing QA Status:	---	IT Deployment:	---

Description Jeffrey Katcher 2011-04-22 22:32:15 UTC

User-Agent:       Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.16) Gecko/20110319 SUSE/3.6.16-0.2.1 Firefox/3.6.16

Same as Fedora: https://bugzilla.redhat.com/show_bug.cgi?id=669204#c12

Extensive use of NFS causes a mount to get stuck.  Processors gradually lock up while the console starts spitting the likes of:

[611775.142230] Code: 0f eb f5 c3 66 90 fa b8 00 00 01 00 f0 0f c1 07 0f b7 d0 c1 e8
10 39 c2 74 07 f3 90 0f b7 17 eb f5 c3 0f 1f 40 00 66 ff 07 56 9d <c3> 66 2e 0f 1f 84
 00 00 00 00 00 ba ff ff ff ff f0 0f c1 17 83
[611775.142244] Call Trace:
[611775.142246]  [<ffffffff8145ba18>] __wait_on_bit_lock+0x38/0xb0
[611775.142248]  [<ffffffff8145bb07>] out_of_line_wait_on_bit_lock+0x77/0x90
[611775.142254]  [<ffffffffa03108b9>] nfs_commit_inode+0xb9/0x2a0 [nfs]
[611775.142268]  [<ffffffffa0310cb6>] nfs_wb_page+0x76/0xb0 [nfs]
[611775.142280]  [<ffffffff810ebdc0>] invalidate_inode_pages2_range+0x1e0/0x2d0
[611775.142284]  [<ffffffffa03030e8>] nfs_revalidate_mapping+0x118/0x170 [nfs]
[611775.142292]  [<ffffffffa03001c3>] nfs_file_read+0x73/0x130 [nfs]
[611775.142297]  [<ffffffff8112eaff>] do_sync_read+0xbf/0x100
[611775.142299]  [<ffffffff8112f2e3>] vfs_read+0xb3/0x190
[611775.142301]  [<ffffffff8112f40e>] sys_read+0x4e/0x90
[611775.142303]  [<ffffffff81002efb>] system_call_fastpath+0x16/0x1b
[611775.142305]  [<00007f2aafc393e0>] 0x7f2aafc393e0
[611840.640059] BUG: soft lockup - CPU#1 stuck for 61s! [zsh:945]

I noticed that the Fedora bugzilla reports a patch that isn't present in OS 11.2, 11.3 or 11.4, so I'm just switching a test system over to an 11.3 kernel with this included.


Reproducible: Sometimes

Steps to Reproduce:
Large amounts of NFS traffic but not always reproducible.  Have seen this with both Broadcom and Intel Gbit Enet cards. 
Actual Results:  
See Details

Expected Results:  
I/O should continue to run, processor should not lock

Comment 1 Jeff Mahoney 2011-04-27 17:02:13 UTC

This is the same fingerprint as:

https://bugzilla.novell.com/show_bug.cgi?id=645610

.. though it's waiting on NFS I/O rather than disk I/O.

Comment 2 Neil Brown 2011-04-28 02:04:22 UTC

Thanks for the report.
The patch mention in the redhat bugzilla was committed to the 11.4 kernel
about 1 month ago but I don't think there has been an update released since then.

I have just committed the same fix to 11.3.
They should both be in the 'kotd' kernel in a day or so.

It looks very likely that the patch you found (thanks!) will address your
problem.  I propose that we leave the bug open for 2 weeks.  If the problem recurs we will look deeper than.  If it doesn't we can close the bug an assume it is fixed.

Thanks.

Comment 3 Neil Brown 2011-05-17 05:15:10 UTC

Closing as fixed - see previous comment.

Comment 4 Jeffrey Katcher 2011-06-10 21:25:47 UTC

Not to be a pest but is there a scheduled date when a kernel containing this fix will be formally released?

Thanks!

Comment 5 Jeff Mahoney 2011-06-10 21:36:49 UTC

I'm not sure when the next update for 11.3 is scheduled, but you can install the fixed kernel from: http://download.opensuse.org/repositories/Kernel:/openSUSE-11.3/

Comment 6 Swamp Workflow Management 2011-08-02 07:59:43 UTC

Update released for: kernel-debug, kernel-debug-base, kernel-debug-base-debuginfo, kernel-debug-debuginfo, kernel-debug-debugsource, kernel-debug-devel, kernel-debug-devel-debuginfo, kernel-default, kernel-default-base, kernel-default-base-debuginfo, kernel-default-debuginfo, kernel-default-debugsource, kernel-default-devel, kernel-default-devel-debuginfo, kernel-desktop, kernel-desktop-base, kernel-desktop-base-debuginfo, kernel-desktop-debuginfo, kernel-desktop-debugsource, kernel-desktop-devel, kernel-desktop-devel-debuginfo, kernel-devel, kernel-ec2, kernel-ec2-base, kernel-ec2-base-debuginfo, kernel-ec2-debuginfo, kernel-ec2-debugsource, kernel-ec2-devel, kernel-ec2-devel-debuginfo, kernel-ec2-extra, kernel-ec2-extra-debuginfo, kernel-pae, kernel-pae-base, kernel-pae-base-debuginfo, kernel-pae-debuginfo, kernel-pae-debugsource, kernel-pae-devel, kernel-pae-devel-debuginfo, kernel-source, kernel-source-vanilla, kernel-syms, kernel-trace, kernel-trace-base, kernel-trace-base-debuginfo, kernel-trace-debuginfo, kernel-trace-debugsource, kernel-trace-devel, kernel-trace-devel-debuginfo, kernel-vanilla, kernel-vanilla-base, kernel-vanilla-base-debuginfo, kernel-vanilla-debuginfo, kernel-vanilla-debugsource, kernel-vanilla-devel, kernel-vanilla-devel-debuginfo, kernel-vmi, kernel-vmi-base, kernel-vmi-base-debuginfo, kernel-vmi-debuginfo, kernel-vmi-debugsource, kernel-vmi-devel, kernel-vmi-devel-debuginfo, kernel-xen, kernel-xen-base, kernel-xen-base-debuginfo, kernel-xen-debuginfo, kernel-xen-debugsource, kernel-xen-devel, kernel-xen-devel-debuginfo, preload-kmp-default, preload-kmp-desktop
Products:
openSUSE 11.3 (debug, i586, x86_64)