Yes, 4.1.28 seems containing a few regressions.  A test kernel with a partial revert is being built on OBS home:tiwai:bnc989084-2 repo.  You can try this kernel once when the build finishes.

Other than that, keep using the previous 4.1.27 kernel from Leap 42.1 for now.

I could reproduce the problem on openSUSE Leap 42.1.

The only additional thing I tried was to set

FW_USE_IPTABLES_BATCH="no"

in order to see if (by some stroke of luck) only the batch processing of iptables might be affected.

Unfortunately, that only changed the process hanging from "iptables-batch" to "iptables" (no surprises there).

The respective iptables/iptables-batch process is completely stuck and can not be killed even with "kill -5" or "kill -9".

AK

P.S. Changed "Product" and "Version" to 42.1.

(In reply to Takashi Iwai from comment #1)
> Yes, 4.1.28 seems containing a few regressions.  A test kernel with a
> partial revert is being built on OBS home:tiwai:bnc989084-2 repo.  You can
> try this kernel once when the build finishes.

Will do and report back ASAP.

AK

(In reply to Takashi Iwai from comment #1)

> partial revert is being built on OBS home:tiwai:bnc989084-2 repo.  You can
> try this kernel once when the build finishes.

I just downloaded that kernel

kernel-default-4.1.28-1.1.gae3ccbc.x86_64.rpm

read the bug report at

https://bugzilla.opensuse.org/show_bug.cgi?id=989084

and had a look at the changelog.

Although the latter did not indicate any changes related to netfilter stuff, I gave it a try.

Unfortunately (and most likely not surprising to you) the problem is still there.

However, I am volunteering to test any packages you may provide in order to fix the netfilter related regressions.

Just give me a quick heads up and I will download/install/test them.

Greetings,

AK

When you reproduce the issue, could you try getting the contents of
/proc/PID/stack (with "PID" replaced by PID of the stuck iptables or
iptables-batch process) few times (5-10) and attach it here?

Also, what does "rpm -q iptables" say?

First of all, I used 

kernel-default-4.1.28-1.1.gcdbda6b.x86_64

from Kernel:openSUSE-42.1/standard on openSUSE Leap 42.1 x86_64 to reproduce the issue.

(In reply to Michal Kubeček from comment #5)
> When you reproduce the issue, could you try getting the contents of
> /proc/PID/stack (with "PID" replaced by PID of the stuck iptables or
> iptables-batch process) few times (5-10) and attach it here

I hope three times is enough, actually I do not only have to reboot after every test, the system also hangs on shut down and I have to do a hard reset.

Here are the respective outputs of cat /proc/$(pidof iptables-batch)/stack after trying to start SuSEfirewall2:

-----------------------------------------------------------------
[<ffffffff816660d9>] retint_kernel+0x1b/0x1d
[<ffffffff811a7e49>] vmap_page_range_noflush+0x279/0x390
[<ffffffffa0776160>] translate_table+0x720/0x820 [ip_tables]
[<ffffffffa07760fd>] translate_table+0x6bd/0x820 [ip_tables]
[<ffffffff811aa4ad>] vmalloc_node+0x4d/0x60
[<ffffffffa11e55ee>] xt_alloc_table_info+0xde/0x124 [x_tables]
[<ffffffffa11e55bd>] xt_alloc_table_info+0xad/0x124 [x_tables]
[<ffffffffa0776e91>] do_ipt_set_ctl+0x121/0x1df [ip_tables]
[<ffffffff8159b62e>] nf_setsockopt+0x3e/0x60
[<ffffffff815a9fcf>] ip_setsockopt+0x7f/0xa0
[<ffffffff8154c5af>] SyS_setsockopt+0x6f/0xd0
[<ffffffff816654f2>] system_call_fastpath+0x16/0x75
[<ffffffffffffffff>] 0xffffffffffffffff

-----------------------------------------------------------------
[<ffffffff816660d9>] retint_kernel+0x1b/0x1d
[<ffffffff811a7e49>] vmap_page_range_noflush+0x279/0x390
[<ffffffffa067b16c>] translate_table+0x72c/0x820 [ip_tables]
[<ffffffffa067b0fd>] translate_table+0x6bd/0x820 [ip_tables]
[<ffffffff811aa4ad>] vmalloc_node+0x4d/0x60
[<ffffffffa11f85ee>] xt_alloc_table_info+0xde/0x124 [x_tables]
[<ffffffffa11f85bd>] xt_alloc_table_info+0xad/0x124 [x_tables]
[<ffffffffa067be91>] do_ipt_set_ctl+0x121/0x1df [ip_tables]
[<ffffffff8159b62e>] nf_setsockopt+0x3e/0x60
[<ffffffff815a9fcf>] ip_setsockopt+0x7f/0xa0
[<ffffffff8154c5af>] SyS_setsockopt+0x6f/0xd0
[<ffffffff816654f2>] system_call_fastpath+0x16/0x75
[<ffffffffffffffff>] 0xffffffffffffffff

-----------------------------------------------------------------
[<ffffffff816660d9>] retint_kernel+0x1b/0x1d
[<ffffffff811a7e49>] vmap_page_range_noflush+0x279/0x390
[<ffffffffa067b160>] translate_table+0x720/0x820 [ip_tables]
[<ffffffffa067b0fd>] translate_table+0x6bd/0x820 [ip_tables]
[<ffffffff811aa4ad>] vmalloc_node+0x4d/0x60
[<ffffffffa11f85ee>] xt_alloc_table_info+0xde/0x124 [x_tables]
[<ffffffffa11f85bd>] xt_alloc_table_info+0xad/0x124 [x_tables]
[<ffffffffa067be91>] do_ipt_set_ctl+0x121/0x1df [ip_tables]
[<ffffffff8159b62e>] nf_setsockopt+0x3e/0x60
[<ffffffff815a9fcf>] ip_setsockopt+0x7f/0xa0
[<ffffffff8154c5af>] SyS_setsockopt+0x6f/0xd0
[<ffffffff816654f2>] system_call_fastpath+0x16/0x75
[<ffffffffffffffff>] 0xffffffffffffffff

 
> Also, what does "rpm -q iptables" say?

iptables-1.4.21-4.1.x86_64

AK

(In reply to Axel Köllhofer from comment #6)
> (In reply to Michal Kubeček from comment #5)
> > When you reproduce the issue, could you try getting the contents of
> > /proc/PID/stack (with "PID" replaced by PID of the stuck iptables or
> > iptables-batch process) few times (5-10) and attach it here
> 
> I hope three times is enough, actually I do not only have to reboot after
> every test, the system also hangs on shut down and I have to do a hard reset.

Sorry for the confusion, I didn't want you to reboot between the attempts,
I meant just getting the stack few times with few seconds betweeen them
to check whether it's stuck somewhere or is actually doing something (even
if too slowly).

Anyway, as the stack seems to look the same all the time, three snapshots
are sufficient.

I can't see the cause from the stack but it's likely to be related to the
bsc#986362 fix. I guess I'll try to reproduce the issue on my test system,
that should make the investigation much easier.

(In reply to Michal Kubeček from comment #7)

> Sorry for the confusion, 

Well, I actually confused myself in this case.

> I didn't want you to reboot between the attempts,
> I meant just getting the stack few times with few seconds between them
> to check whether it's stuck somewhere or is actually doing something (even
> if too slowly).

Yup, this is what I did and as the output was always exactly the same, I thought "Maybe it was meant the other way".

So just to confirm your suspicion, yes the stack looks the same and the iptables/iptables-batch process really hangs without proceeding at that point.

AK

P.S. I changed the severity of this bug to "Major" as it renders a core part of the OS unusable.

So far I haven't been able to reproduce a hang (soft lockup?). I'm testing
4.1.28-2.gae3ccbc-default but that should only differ in the ecryptfs fix.

However, I still can see some problems: SuSEfirewall does create only small
part of netfilter rules compared to a 3.12.61 kernel and when I dump the
rules with iptables-save on 3.12.61 and try to pass them to iptables-restore
on 4.1.28-2, I get an error (on "COMMIT" line).

I checked the source and it seems that current openSUSE-42.1 differs from my
original backport of bsc#986362 fixes in a few details. Apparently 4.1.28
also adds a backport of commit d7591f0c41 which I didn't include. But a more
important difference is IMHO that the stable-4.1 backport ignores the fact
that 4.1 doesn't have the simplification done in

  482cfc318559  netfilter: xtables: avoid percpu ruleset duplication

so that 4.1.28 is missing this part of check_compat_entry_size_and_hooks():

       /* And one copy for every other CPU */
       for_each_possible_cpu(i)
               if (newinfo->entries[i] && newinfo->entries[i] != entry1)
                       memcpy(newinfo->entries[i], entry1, newinfo->size);

This might explain why the rule set is incomplete. Worse, the 4.1.28 backport
doesn't handle entries being a per-cpu array in find_jump_target(), neither
in the way I did nor in the way it was done in other stable backports. I'll
try to fix these two problems on top of current openSUSE-42.1 so that we can
check if this is the cause.

I checked the comparison of each patch, and found some significant differences.

patches.fixes/ipv4-Don-t-do-expensive-useless-work-during-inetdev-.patch
4.1.x: 86de8271be91cce66aace5a3ae8afd3f28094957

patches.fixes/netfilter-x_tables-validate-targets-of-jumps.patch
4.1.x: 8163327a3a927c36f7c032bb67957e6c0f4ec27d

patches.fixes/netfilter-x_tables-do-compat-validation-via-translat.patch
4.1.x: af815d264b7ed1cdceb0e1fdf4fa7d3bd5fe9a99

Especially the second one looks suspicious to lead to a crash.

I'm building another test kernel to fix the inconsistency in commit 8163327a3a927c36f7c032bb67957e6c0f4ec27d.  It's found in OBS home:tiwai:bnc989176 repo.

Created attachment 684518 [details]
tentative fix

Tried this patch and it seems to resolve the issues I found.

A test kernel with it (on top of current head of openSUSE-42.1 branch) is
building in OBS project home:mkubecek:989176

(In reply to Michal Kubeček from comment #12)
> Created attachment 684518 [details]
> tentative fix
> 
> Tried this patch and it seems to resolve the issues I found.
> 
> A test kernel with it (on top of current head of openSUSE-42.1 branch) is
> building in OBS project home:mkubecek:989176

Ah good, you made the very same fix :)  Then I'll scratch my OBS repo in comment 11.

Could you guys test Michal's kernel please?  Thanks!

Ok, this is what ended up in 3.12:
https://git.kernel.org/cgit/linux/kernel/git/jirislaby/linux-stable.git/commit/?h=stable-3.12-queue&id=42b891251b3b9f241fb5f0a90e87b15d2fef0a74

It is from Florian's backport for 3.14:
http://article.gmane.org/gmane.linux.kernel.stable/182808

Is this OK?

BTW there is a report of performance regression at:
http://article.gmane.org/gmane.linux.kernel.stable/184096

(In reply to Jiri Slaby from comment #14)
> Ok, this is what ended up in 3.12:
> https://git.kernel.org/cgit/linux/kernel/git/jirislaby/linux-stable.git/
> commit/?h=stable-3.12-queue&id=42b891251b3b9f241fb5f0a90e87b15d2fef0a74
> 
> It is from Florian's backport for 3.14:
> http://article.gmane.org/gmane.linux.kernel.stable/182808
> 
> Is this OK?

Yes, that's another way to do find_jump_target() correctly on pre-4.2 kernels
(a bit better than mine, I believe).

However, I believe there is another problem: stable-4.1 backport of commit

  09d9686047db  netfilter: x_tables: do compat validation via translate_table

drops the code copying new entry to all other per-cpu copies (there are none
since v4.2-rc1). That's what the translate_compat_table() fragments of the
patch from comment 12 add back.

> BTW there is a report of performance regression at:
> http://article.gmane.org/gmane.linux.kernel.stable/184096

I'm aware of it, this is what bug 986362 comments 45 and 46 are about.

Test kernel build has finished, packages can be retrieved from OBS repository
home:mkubecek:bsc989176 (the name in comment 12 was wrong) or downloaded at

  http://download.opensuse.org/repositories/home:/mkubecek:/bsc989176/openSUSE_Leap_42.1/

Axel, please check if it resolves the issue on your system.

(In reply to Michal Kubeček from comment #16)
> Test kernel build has finished, packages can be retrieved from OBS repository
> home:mkubecek:bsc989176 (the name in comment 12 was wrong) or downloaded at
> 
>  
> http://download.opensuse.org/repositories/home:/mkubecek:/bsc989176/
> openSUSE_Leap_42.1/
> 
> Axel, please check if it resolves the issue on your system.

Just downloaded, installed and tested kernel-default-4.1.28-1.1.x86_64 from home:mkubecek:989176 and it seems to fix the issue.

Thanks.

AK

Just to give some feedback ASAP:

The packages are not published yet but osc getbinaries is a nice way of getting rpms, so I gave it a shot.

I just installed

kernel-default-4.1.28-3.1.gd509193.x86_64

from Kernel:openSUSE-42.1:standard including the fix mentioned above and it works as expected.

AK

For the record: I deleted the home:mkubecek:bsc989176 OBS project as the fix
is now available in KotD.

Changed status to RESOLVED FIXED.

I hope this is OK.

AK

openSUSE-SU-2016:2290-1: An update that solves 17 vulnerabilities and has 9 fixes is now available.

Category: security (important)
Bug References: 963931,970948,971126,971360,974266,978821,978822,979018,979213,979879,980371,981058,981267,986362,986365,986570,987886,989084,989152,989176,990058,991110,991608,991665,994296,994520
CVE References: CVE-2015-8787,CVE-2016-1237,CVE-2016-2847,CVE-2016-3134,CVE-2016-3156,CVE-2016-4485,CVE-2016-4486,CVE-2016-4557,CVE-2016-4569,CVE-2016-4578,CVE-2016-4580,CVE-2016-4805,CVE-2016-4951,CVE-2016-4998,CVE-2016-5696,CVE-2016-6480,CVE-2016-6828
Sources used:
openSUSE Leap 42.1 (src):    drbd-8.4.6-8.1, hdjmod-1.28-24.1, ipset-6.25.1-5.1, kernel-debug-4.1.31-30.2, kernel-default-4.1.31-30.2, kernel-docs-4.1.31-30.3, kernel-ec2-4.1.31-30.2, kernel-obs-build-4.1.31-30.3, kernel-obs-qa-4.1.31-30.1, kernel-obs-qa-xen-4.1.31-30.1, kernel-pae-4.1.31-30.2, kernel-pv-4.1.31-30.2, kernel-source-4.1.31-30.1, kernel-syms-4.1.31-30.1, kernel-vanilla-4.1.31-30.2, kernel-xen-4.1.31-30.2, lttng-modules-2.7.0-2.1, pcfclock-0.44-266.1, vhba-kmp-20140928-5.1