Bug 1180377

Summary: libpython3_6m1_0 undefined symbol: PyFPE_jbuf (was borgbackup)
Product: [openSUSE] openSUSE Distribution Reporter: Jonathan Wiebe <jonathan.b.wiebe>
Component: OtherAssignee: Matej Cepl <mcepl>
Status: RESOLVED FIXED QA Contact: E-mail List <qa-bugs>
Severity: Critical    
Priority: P1 - Urgent CC: andreas.taschner, asoraruf, christian, felix.niederwanger, ismail, julien.adamek, k.bug, ldevulder, lidong.zhong, markus, me, meissner, mlin, os-dev, pdostal, rich.paredes, skliu, suse-beta, suse, tetsuya.inoue
Version: Leap 15.2   
Target Milestone: ---   
Hardware: Other   
OS: Other   
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---
Bug Depends on:    
Bug Blocks: 1180331    

Description Jonathan Wiebe 2020-12-26 23:18:49 UTC
My apologies if I am misfiling this. It is my first bug report for OpenSUSE.

I updated my system this afternoon using the commands:
sudo zypper patch
sudo zypper update

After doing this I rebooted and my backup application (vorta) auto-started as usual but immediately exited.

Looking a bit deeper I found the the underlying error was with borgbackup (which vorta uses as a backend).

Here is the error I get when running the command 'borg' manually:

Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line 38, in <module>
    from . import helpers
  File "/usr/lib64/python3.6/site-packages/borg/helpers.py", line 75, in <module>
    import borg.crypto.low_level
ImportError: /usr/lib64/python3.6/site-packages/borg/crypto/low_level.cpython-36m-x86_64-linux-gnu.so: undefined symbol: PyFPE_jbuf

I get this error even when running 'borg --version".

Current versions taken from zypper info borgbackup:

Information for package borgbackup:
-----------------------------------
Repository     : Main Repository
Name           : borgbackup
Version        : 1.1.13-lp152.1.1
Arch           : x86_64
Vendor         : openSUSE
Installed Size : 3.6 MiB
Installed      : Yes
Status         : up-to-date
Source package : borgbackup-1.1.13-lp152.1.1.src
Summary        : Deduplicating backup program with compression and authenticated encryption

And zypper info python3:

Repository     : Main Update Repository
Name           : python3
Version        : 3.6.12-lp152.4.12.2
Arch           : x86_64
Vendor         : openSUSE
Installed Size : 141.3 KiB
Installed      : Yes (automatically)
Status         : up-to-date
Source package : python3-3.6.12-lp152.4.12.2.src
Summary        : Python 3 Interpreter

Please let me know if I can provide any additional information to help in tracking down this issue.
Comment 1 Alessandro Soraruf 2020-12-27 01:15:59 UTC
Python3 numpy and others broken due to openSUSE-2020-2332 update. The nerw libpython3_6m1_0 (libpython3_6m1_0-3.6.12-lp152.4.12.2.x86_64.rpm) introduces an unresolved PyFPE_jbuf symbol error.

As per SLES ( see https://www.suse.com/support/kb/doc/?id=000019818 ) even on Leap the workaround is the downgrade of libpython3_6m1_0 package:

zypper in --oldpackage libpython3_6m1_0-3.6.12-lp152.4.9.1

Thanks.
Alessandro
Comment 2 Marcus Meissner 2020-12-27 10:10:47 UTC
the 15.2 python3 updarte removes the fpectl with ... 

-    --with-fpectl \

was this intended matej?
Comment 3 Felix Niederwanger 2020-12-28 11:15:50 UTC
I'm also affected by this bug and can confirm that the suggestion of Alessandro Soraruf to downgrade helps to fix numpy:

zypper in --oldpackage libpython3_6m1_0-3.6.12-lp152.4.9.1

A easy reproducer is to just import numpy

    $ python3
    Python 3.6.12 (default, Dec 02 2020, 09:44:23) [GCC] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import numpy

I'm also opening https://progress.opensuse.org/issues/81376 for creating a regression test for numpy and scipy, as I could not find anything there. Input is appreciated, I'll start to work on this once I'm back from my vacation.
Comment 4 Kurt Garloff 2020-12-28 15:01:24 UTC
This affects me as well, trying to use kiwi.
An easy way to reproduce is
python3 -c "import lxml.etree"
which worked before and got broken by dropping
--with-fpectl
from the python3 build.
I suspect all packages using cython need recompilation after such an incompatible change.

So I definitely agree this is P1/Critical.
Comment 5 Marcus Meissner 2020-12-28 15:10:20 UTC
I removed the buggy python3 update from the leap 15.2 and 15.1 update codestream.
Comment 6 Julien ADAMEK 2020-12-28 15:45:18 UTC
Hello, we are also impacted when we configure a cluster on SLE15GA/SP1/SP2:
Fatal error:
    /usr/lib64/python3.6/site-packages/lxml/etree.cpython-36m-x86_64-linux-gnu.so: undefined symbol: PyFPE_jbuf
https://bugzilla.suse.com/show_bug.cgi?id=1180283
Comment 7 Marcus Meissner 2020-12-28 15:58:30 UTC
submitted
  233247 review moderate: ~msmeissn:.../python3.SUSE_SLE-15_Update -> SUSE:SLE-15:Update [Requesting review by bugowner]
Comment 9 Lidong Zhong 2020-12-29 06:45:07 UTC
(In reply to Marcus Meissner from comment #7)
> submitted
>   233247 review moderate: ~msmeissn:.../python3.SUSE_SLE-15_Update ->
> SUSE:SLE-15:Update [Requesting review by bugowner]

Hi Marcus, is it OK to release it as an EMU? I already have two L3 incidents about this problem.(bsc#1180357 and bsc#1180379). Thanks.
Comment 10 Marcus Meissner 2020-12-29 07:54:13 UTC
i requested an emu
Comment 11 Gustavo Figueira 2020-12-29 09:46:33 UTC
*** Bug 1180430 has been marked as a duplicate of this bug. ***
Comment 12 Robert Schweikert 2020-12-29 15:20:45 UTC
*** Bug 1180379 has been marked as a duplicate of this bug. ***
Comment 13 Julien ADAMEK 2020-12-29 15:59:32 UTC
*** Bug 1180283 has been marked as a duplicate of this bug. ***
Comment 14 Swamp Workflow Management 2020-12-29 20:16:26 UTC
SUSE-RU-2020:3946-1: An update that has one recommended fix can now be installed.

Category: recommended (important)
Bug References: 1180377
CVE References: 
JIRA References: 
Sources used:
SUSE Linux Enterprise Server for SAP 15 (src):    python3-3.6.12-3.70.1, python3-core-3.6.12-3.70.1
SUSE Linux Enterprise Server 15-LTSS (src):    python3-3.6.12-3.70.1, python3-core-3.6.12-3.70.1
SUSE Linux Enterprise Module for Development Tools 15-SP3 (src):    python3-core-3.6.12-3.70.1
SUSE Linux Enterprise Module for Development Tools 15-SP2 (src):    python3-core-3.6.12-3.70.1
SUSE Linux Enterprise Module for Development Tools 15-SP1 (src):    python3-core-3.6.12-3.70.1
SUSE Linux Enterprise Module for Basesystem 15-SP3 (src):    python3-3.6.12-3.70.1, python3-core-3.6.12-3.70.1
SUSE Linux Enterprise Module for Basesystem 15-SP2 (src):    python3-3.6.12-3.70.1, python3-core-3.6.12-3.70.1
SUSE Linux Enterprise Module for Basesystem 15-SP1 (src):    python3-3.6.12-3.70.1, python3-core-3.6.12-3.70.1
SUSE Linux Enterprise High Performance Computing 15-LTSS (src):    python3-3.6.12-3.70.1, python3-core-3.6.12-3.70.1
SUSE Linux Enterprise High Performance Computing 15-ESPOS (src):    python3-3.6.12-3.70.1, python3-core-3.6.12-3.70.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 15 Matej Cepl 2020-12-30 12:02:50 UTC
Update has been released.
Comment 16 Swamp Workflow Management 2020-12-30 14:15:58 UTC
openSUSE-RU-2020:2362-1: An update that has one recommended fix can now be installed.

Category: recommended (important)
Bug References: 1180377
CVE References: 
JIRA References: 
Sources used:
openSUSE Leap 15.1 (src):    python3-3.6.12-lp151.6.36.1, python3-core-3.6.12-lp151.6.36.1, python3-documentation-3.6.12-lp151.6.36.1
Comment 17 Matej Cepl 2020-12-30 18:23:08 UTC
Certainly removing this flag from the existing package was a mistake, for which I am sorry.

However, let me just remind anybody who puts his trust in `fpectl` library that it goes away forever in 3.7 (never packaged for SUSE, but we have 3.8 and newer) and it won’t come again. https://docs.python.org/3.9/whatsnew/changelog.html?highlight=fpectl#python-3-7-0-alpha-4 (search for “fpectl” string).
Comment 18 Matej Cepl 2020-12-30 18:35:46 UTC
This bug occurred because we have synchronized all python SPEC files among all versions of Python and all platforms, and this option (and whole ‘fpectl’ library) has been eliminated in Python 3.7. Therefore this option is also missing in packages:

* python36 in openSUSE:Factory
* python36 in SUSE:SLE-12-SP3:Update:Products:Teradata:Update
* python36 in SUSE:SLE-12-SP5:Update

(it is still OK in python3 in SUSE:SLE-12:Update, which is 3.4 so it hasn’t been synchronized, as well as in python 2.7 on all platforms).

I understand that there are not many packages build against these packages, but the question is, should I return it there as well, or not?
Comment 19 Matej Cepl 2020-12-30 18:51:14 UTC
*** Bug 1180331 has been marked as a duplicate of this bug. ***
Comment 20 Matej Cepl 2020-12-30 18:53:15 UTC
*** Bug 1180382 has been marked as a duplicate of this bug. ***
Comment 21 Andreas Taschner 2020-12-31 07:47:55 UTC
(In reply to Matej Cepl from comment #17)
> Certainly removing this flag from the existing package was a mistake, for
> which I am sorry.
> 
> However, let me just remind anybody who puts his trust in `fpectl` library
> that it goes away forever in 3.7 (never packaged for SUSE, but we have 3.8
> and newer) and it won’t come again.
> https://docs.python.org/3.9/whatsnew/changelog.html?highlight=fpectl#python-
> 3-7-0-alpha-4 (search for “fpectl” string).

Thank you, Matej. 
I relayed that message to my customer, who asked for a verification of whether they have understood things correctly :
<snip>
But just to clarify this situation seems to be arising from indirect reference in the available python libraries. 
For example eventhough the user might not be using FPE directly, importing the lxml.etree seems to bring in this error.

I would assume this dependency is addressed internally in the python library
during the removal.
</snip>
Comment 22 Matej Cepl 2020-12-31 12:59:07 UTC
(In reply to Andreas Taschner from comment #21)
> I relayed that message to my customer, who asked for a verification of
> whether they have understood things correctly :
> <snip>
> But just to clarify this situation seems to be arising from indirect
> reference in the available python libraries. 
> For example eventhough the user might not be using FPE directly, importing
> the lxml.etree seems to bring in this error.

Yes, the problem we had was only because all packages in SLE-15 were originally built against Python which had fpectl on, so they referred appropriate symbols in it. When the Python interpreter suddenly didn’t contain these symbols, they failed. When those same packages are build against Python interpreters without those symbols in the first place, everything works. Obviously, we have python-lxml working just fine in Factory, build against Python 3.8, which doesn’t have fpectl anymore.

Nice explanation about what fpectl is, and why it is not needed (or why it might be actually harmful) is in the old version of fpectl module documentation at https://docs.python.org/3.6/library/fpectl.html .
Comment 24 Marcus Meissner 2021-01-02 10:11:14 UTC
(In reply to Matej Cepl from comment #18)
> This bug occurred because we have synchronized all python SPEC files among
> all versions of Python and all platforms, and this option (and whole
> ‘fpectl’ library) has been eliminated in Python 3.7. Therefore this option
> is also missing in packages:
> 
> * python36 in openSUSE:Factory
> * python36 in SUSE:SLE-12-SP3:Update:Products:Teradata:Update
> * python36 in SUSE:SLE-12-SP5:Update
> 
> (it is still OK in python3 in SUSE:SLE-12:Update, which is 3.4 so it hasn’t
> been synchronized, as well as in python 2.7 on all platforms).
> 
> I understand that there are not many packages build against these packages,
> but the question is, should I return it there as well, or not?

In general we MUST NOT break binary compatibility, you saw what happens.

So what was there before, needs to stay.

For newly added python versions we can leave it out I think.
Comment 26 Matej Cepl 2021-01-04 12:40:01 UTC
All three requests were submitted.
Comment 27 OBSbugzilla Bot 2021-01-04 13:10:11 UTC
This is an autogenerated message for OBS integration:
This bug (1180377) was mentioned in
https://build.opensuse.org/request/show/860180 Factory / python36
Comment 28 Jason Craig 2021-01-05 19:15:21 UTC
*** Bug 1180417 has been marked as a duplicate of this bug. ***
Comment 29 Felix Niederwanger 2021-01-06 10:23:25 UTC
Not sure if this is relevant, but the bug appears to be still present on Leap 15-SP3: http://duck-norris.qam.suse.de/tests/4424#step/python_scientific/10

But I guess this is already known.
Comment 30 Liu Shukui 2021-01-12 10:37:20 UTC
hi, the bug seems no fixed after updating to python36-3.6.12-4.28.1 on sles12sp5

s12sp5-t5810:/usr/src/packages/SPECS # rpm -qa| grep python36
python36-3.6.12-4.28.1.x86_64
python36-base-3.6.12-4.28.1.x86_64

s12sp5-t5810:/usr/src/packages/SPECS # type python3.6
python3.6 is hashed (/usr/bin/python3.6)
s12sp5-t5810:/usr/src/packages/SPECS # rpm -qf  /usr/bin/python3.6 
python36-base-3.6.12-4.28.1.x86_64


s12sp5-t5810:/usr/src/packages/SPECS # python3.6
Python 3.6.12 (default, Nov 25 2020, 21:46:03) [GCC] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import fpectl
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'fpectl'
>>>
Comment 31 Marcus Meissner 2021-01-12 15:06:16 UTC
i think this might not be the same thing.

nm -D /usr/lib64/libpython3.6*|grep -i PyFPE_jbuf

to see if the symbol is present
Comment 33 Liu Shukui 2021-01-13 07:47:20 UTC
(In reply to Marcus Meissner from comment #31)
> i think this might not be the same thing.
> 
> nm -D /usr/lib64/libpython3.6*|grep -i PyFPE_jbuf
> 
> to see if the symbol is present

you are right, thank you.

BEFORE
~~~~~~
s12sp5-t5810:~ # nm -D /usr/lib64/libpython3.6*|grep -i PyFPE_jbuf
s12sp5-t5810:~ # 

the symbol "PyFPE_jbuf" is not present.

--> BUG REPRODUCED

AFTER
~~~~~
s12sp5-t5810:~ # nm -D /usr/lib64/libpython3.6*|grep -i PyFPE_jbuf
00000000004e1780 B PyFPE_jbuf

the symbol "PyFPE_jbuf" is present.

--> BUG FIXED
Comment 34 Swamp Workflow Management 2021-01-13 14:30:01 UTC
SUSE-RU-2021:0112-1: An update that has one recommended fix can now be installed.

Category: recommended (moderate)
Bug References: 1180377
CVE References: 
JIRA References: 
Sources used:
SUSE Linux Enterprise Server 12-SP5 (src):    python36-3.6.12-4.28.1, python36-core-3.6.12-4.28.1

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.
Comment 35 Matej Cepl 2021-01-20 22:10:55 UTC
Packages released.
Comment 38 OBSbugzilla Bot 2021-10-06 14:44:35 UTC
This is an autogenerated message for OBS integration:
This bug (1180377) was mentioned in
https://build.opensuse.org/request/show/923499 Factory / python36
Comment 39 OBSbugzilla Bot 2021-10-22 08:44:40 UTC
This is an autogenerated message for OBS integration:
This bug (1180377) was mentioned in
https://build.opensuse.org/request/show/926876 Factory / python36