Bug 1207989

Summary: liblapacke.so.3 has unresolved references
Product: [openSUSE] openSUSE Tumbleweed Reporter: Richard Biener <rguenther>
Component: DevelopmentAssignee: E-mail List <screening-team-bugs>
Status: RESOLVED FIXED QA Contact: E-mail List <qa-bugs>
Severity: Normal    
Priority: P5 - None CC: eich, rguenther, stefan.bruens
Version: Current   
Target Milestone: ---   
Hardware: Other   
OS: Other   
See Also: https://bugzilla.redhat.com/show_bug.cgi?id=1287405
https://bugzilla.suse.com/show_bug.cgi?id=1208141
Whiteboard:
Found By: --- Services Priority:
Business Priority: Blocker: ---
Marketing QA Status: --- IT Deployment: ---

Description Richard Biener 2023-02-07 11:51:12 UTC
This was created to carry the info from the closed https://bugzilla.suse.com/show_bug.cgi?id=1087426

--

There undefined references in shared object file for the lapack library.

Simple test program to reproduce

#include <stdio.h>
#include "lapacke.h"

int main()
{
        printf("Hello world!\n");

        int* pi = NULL;
        double* pd = NULL;

        LAPACKE_dgesv(0, 1, 1, pd, 1, pi, pd, 1);

        return 0;
}

When trying to compile dynamically linked it give following error:

s12-sp2-sss:~ # gcc -o test test.c -llapack -llapacke 
/usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../lib64/liblapacke.so: undefined reference to `clagge_'
/usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../lib64/liblapacke.so: undefined reference to `clatms_'
/usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../lib64/liblapacke.so: undefined reference to `dlagsy_'
/usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../lib64/liblapacke.so: undefined reference to `slatms_'
/usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../lib64/liblapacke.so: undefined reference to `zlatms_'
/usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../lib64/liblapacke.so: undefined reference to `slagsy_'
/usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../lib64/liblapacke.so: undefined reference to `dlagge_'
/usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../lib64/liblapacke.so: undefined reference to `clagsy_'
/usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../lib64/liblapacke.so: undefined reference to `claghe_'
/usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../lib64/liblapacke.so: undefined reference to `zlagsy_'
/usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../lib64/liblapacke.so: undefined reference to `slagge_'
/usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../lib64/liblapacke.so: undefined reference to `zlagge_'
/usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../lib64/liblapacke.so: undefined reference to `dlatms_'
/usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../lib64/liblapacke.so: undefined reference to `zlaghe_'
collect2: error: ld returned 1 exit status

When lapacke is linked static it does work
Comment 1 Richard Biener 2023-02-07 11:58:56 UTC
This was reported against SLE12 SP2.  The issue was identified as lapacke having testing functions compiled in but not the required initialization routines from tmglib.  The solution was to add those as well to not remove any symbols and break the ABI.

The Factory liblapacke misses these symbols though the original problem cannot be reproduced, the shared library doesn't contain any references to clagge_

But the static link fails when doing

> gcc -c test.c
> gfortran test.o -static -Wl,--begin-group -llapack -llapacke -Wl,--end-group


The SONAME of the lapack shared libraries is still liblapack.so.3 so we should
see to maintain ABI compatibility here.
Comment 2 Stefan Brüns 2023-02-07 12:43:57 UTC
So, which lapacke is this, the one from the lapack (source) package, or the one from openBLAS?

I think this is a problem in lapacke, and in lapacke only. CLAGGE etc. are auxiliary testing routines only. These are not, and have never been part of the lapack interface. lapacke should not contain any undefined references for CLAGGE etc.

What happens when you use e.g. Intel MKL or FlexiBLAS? Both don't provide these functions.
Comment 3 Egbert Eich 2023-02-07 13:04:47 UTC
This was from the netlib-lapack package. The TW OpenBLAS package contains all the functions listed.
Comment 4 Richard Biener 2023-02-07 13:55:20 UTC
Yes this was for the lapack (source) package.

The main issue is that there's no agreed upon ABI all of the implementations
have to supply and what to do with symbols provided that are out of this
set of common symbols.

Plus there's no QA that all of the implementations do provide these functions
and best not any additional ones (like the testing ones that got into lapacke).

--

It might be possible for example to have a separate lapack-abi package
that provides a linker script that can be used to hide unwanted symbols
plus maybe some test program testing availability of wanted symbols.
Comment 5 Richard Biener 2023-02-07 13:56:23 UTC
It might be best if I split the SR into one bringing back the DEPRECATED symbols and one "completing" the test ones which we might choose to not adopt.
Comment 6 Stefan Brüns 2023-02-09 11:21:29 UTC
Some context for the deprecated functions:

Most were already deprecated in LAPACK 3.1 (atleast), released in June 2006. Only outlier is xGGSVD, which was only deprecated in LAPACK 3.6 (released November 2015).

AFAICS, the deprecated functions all have replacements which are strict supersets. E.g. xGEGS vs xGGES, the later only adds a SORT parameter, and when SORT='N', both behave identical.

Unfortunately, the deprecated functions are open coded instead of being trivial wrappers around the current ones.
Comment 7 Stefan Brüns 2023-02-09 22:19:36 UTC
Now lapacke is broken:
/usr/lib64/gcc/x86_64-suse-linux/12/../../../../x86_64-suse-linux/bin/ld: /usr/lib64/liblapacke.so: undefined reference to `LAPACK_cggsvp'


Taking liblapack/liblapacke from science:

for l in   /usr/lib64/liblapack*.so.3 ; do echo $l ; nm -DC $l | grep cggsvp ; done 
/usr/lib64/liblapack.so.3
0000000000090910 T cggsvp3_
000000000008f1c0 T cggsvp_
/usr/lib64/liblapacke.so.3
0000000000069170 T LAPACKE_cggsvp
00000000000694a0 T LAPACKE_cggsvp3
0000000000067070 T LAPACKE_cggsvp3_work
0000000000067920 T LAPACKE_cggsvp_work
                 U LAPACK_cggsvp
                 U cggsvp3_


From Tumbleweed 20230208:
for l in   /usr/lib64/liblapack*.so.3 ; do echo $l ; nm -DC $l | grep cggsvp ; done 
/usr/lib64/liblapacke.so.3
                 U cggsvp3_
00000000000669d0 T LAPACKE_cggsvp3
0000000000066120 T LAPACKE_cggsvp3_work
/usr/lib64/liblapack.so.3
0000000000089050 T cggsvp3_

cggsvp is one of the deprecated functions (replaced by cggsvp3).
Comment 8 Stefan Brüns 2023-02-09 22:57:27 UTC
(In reply to Stefan Brüns from comment #7)
> Now lapacke is broken:
> /usr/lib64/gcc/x86_64-suse-linux/12/../../../../x86_64-suse-linux/bin/ld:
> /usr/lib64/liblapacke.so: undefined reference to `LAPACK_cggsvp'

Reverting the 'BUILD_DEPRECATED' change fixes this.
Comment 9 Egbert Eich 2023-02-10 07:58:07 UTC
@Richi, Stefan fixed it in:
https://build.opensuse.org/request/show/1064110
could you have a look, please?

@Stefan, thank you for the SR!
Comment 10 Richard Biener 2023-02-10 08:09:33 UTC
I wonder why the lapacke reference is mangled to LAPACK_cggsvp ...

OK, I think that LAPACKE/include/lapack.h simply misses declarations of the
deprecated functions and the call in LAPACKE/src/lapacke_cggsvp_work.c
is likely not prototyped:

        /* Call LAPACK function and adjust info */
        LAPACK_cggsvp( &jobu, &jobv, &jobq, &m, &p, &n, a, &lda, b, &ldb, &tola,
                       &tolb, k, l, u, &ldu, v, &ldv, q, &ldq, iwork, rwork,
                       tau, work, &info );

I suppose that's a bug upstream, eventually the deprecated functions got
removed from lapack.h and there's failure to conditionally provide it
under some #ifdef or in an internal header for the purpose of building
the deprecated functions into the library.

I wonder how other distros resolved the API deprecation, like whether they bumped the SONAME and removed the interfaces.
Comment 11 Richard Biener 2023-02-10 08:14:38 UTC
(In reply to Egbert Eich from comment #9)
> @Richi, Stefan fixed it in:
> https://build.opensuse.org/request/show/1064110
> could you have a look, please?
> 
> @Stefan, thank you for the SR!

It looks fine to me, but I have no accept rights for science[/lapack],
but you seem to?  @Egbert
Comment 12 Egbert Eich 2023-02-10 08:43:43 UTC
I've accepted it. It's on it's way to Factory.
@Stefan, thank for your help :)  - also for making sure this gets noticed in the future.
Comment 13 Stefan Brüns 2023-02-10 10:59:06 UTC
(In reply to Richard Biener from comment #10)
> I wonder why the lapacke reference is mangled to LAPACK_cggsvp ...
> 
> OK, I think that LAPACKE/include/lapack.h simply misses declarations of the
> deprecated functions and the call in LAPACKE/src/lapacke_cggsvp_work.c
> is likely not prototyped:
> [...] 
> I wonder how other distros resolved the API deprecation, like whether they
> bumped the SONAME and removed the interfaces.

The cherry-picked commit which readded the declarations is part of LAPACK 3.9.1, and current is 3.11.0.

I think we should upgrade to at least 3.9.1 for SLE/Leap, and 3.11.0 for TW.
Comment 14 Richard Biener 2023-02-10 11:06:34 UTC
(In reply to Stefan Brüns from comment #13)
> (In reply to Richard Biener from comment #10)
> > I wonder why the lapacke reference is mangled to LAPACK_cggsvp ...
> > 
> > OK, I think that LAPACKE/include/lapack.h simply misses declarations of the
> > deprecated functions and the call in LAPACKE/src/lapacke_cggsvp_work.c
> > is likely not prototyped:
> > [...] 
> > I wonder how other distros resolved the API deprecation, like whether they
> > bumped the SONAME and removed the interfaces.
> 
> The cherry-picked commit which readded the declarations is part of LAPACK
> 3.9.1, and current is 3.11.0.
> 
> I think we should upgrade to at least 3.9.1 for SLE/Leap, and 3.11.0 for TW.

IMHO there's no need to deviate and a version upgrade for SLE/Leap 15 needs justification anyway (and the maintainance bots will complain about the
missing symbols we've added for the original issue, but it might be possible
to get a waiver for them).
Comment 15 Stefan Brüns 2023-02-10 11:43:15 UTC
(In reply to Richard Biener from comment #14)

> > I think we should upgrade to at least 3.9.1 for SLE/Leap, and 3.11.0 for TW.
> 
> IMHO there's no need to deviate and a version upgrade for SLE/Leap 15 needs
> justification anyway (and the maintainance bots will complain about the
> missing symbols we've added for the original issue, but it might be possible
> to get a waiver for them).

Why should the bots complain? BUILD_DEPRECATED also works for 3.9.1 and any later version.

The fact 3.9.1 is a bug fix release for 3.9.0, and some of these fixes are already cherry-picked (but most are not), should be reason enough.
Comment 16 Richard Biener 2023-02-10 11:56:52 UTC
(In reply to Stefan Brüns from comment #15)
> (In reply to Richard Biener from comment #14)
> 
> > > I think we should upgrade to at least 3.9.1 for SLE/Leap, and 3.11.0 for TW.
> > 
> > IMHO there's no need to deviate and a version upgrade for SLE/Leap 15 needs
> > justification anyway (and the maintainance bots will complain about the
> > missing symbols we've added for the original issue, but it might be possible
> > to get a waiver for them).
> 
> Why should the bots complain? BUILD_DEPRECATED also works for 3.9.1 and any
> later version.
> 
> The fact 3.9.1 is a bug fix release for 3.9.0, and some of these fixes are
> already cherry-picked (but most are not), should be reason enough.

SLE/Leap(?) 15 still has lapack 3.5.0 though.
Comment 19 Stefan Brüns 2023-02-10 12:33:09 UTC
(In reply to Richard Biener from comment #16)
> > The fact 3.9.1 is a bug fix release for 3.9.0, and some of these fixes are
> > already cherry-picked (but most are not), should be reason enough.
> 
> SLE/Leap(?) 15 still has lapack 3.5.0 though.

Well, yes and no.

The lapack package is still 3.5.0. Which unfortunately implies no CBLAS (also on Leap, which had it up to Leap 15.2).

On the other hand, the openBLAS (0.3.21) package contains Lapack 3.10.1. (3.11.0 is in Git).
Comment 24 Richard Biener 2023-02-14 09:08:04 UTC
The difference in exported symbols from the SLE15 lapack packages compared to the version in Factory is:

libblas.so.3: none

liblapack.so.3:

removed symbols:

-clagge_
-claghe_
-clagsy_
-clahilb_
-clakf2_
-clarge_
-clarnd_
-claror_
-clarot_
-clatm1_
-clatm2_
-clatm3_
-clatm5_
-clatm6_
-clatme_
-clatmr_
-clatms_
-clatmt_
-dlagge_
-dlagsy_
-dlahilb_
-dlakf2_
-dlaran_
-dlarge_
-dlarnd_
-dlaror_
-dlarot_
-dlatm1_
-dlatm2_
-dlatm3_
-dlatm5_
-dlatm6_
-dlatm7_
-dlatme_
-dlatmr_
-dlatms_
-dlatmt_
-slagge_
-slagsy_
-slahilb_
-slakf2_
-slaran_
-slarge_
-slarnd_
-slaror_
-slarot_
-slatm1_
-slatm2_
-slatm3_
-slatm5_
-slatm6_
-slatm7_
-slatme_
-slatmr_
-slatms_
-slatmt_
-zlagge_
-zlaghe_
-zlagsy_
-zlahilb_
-zlakf2_
-zlarge_
-zlarnd_
-zlaror_
-zlarot_
-zlatm1_
-zlatm2_
-zlatm3_
-zlatm5_
-zlatm6_
-zlatme_
-zlatmr_
-zlatms_
-zlatmt_

added symbols:

+cgejsv_
+cgelq_
+cgelqt3_
+cgelqt_
+cgemlq_
+cgemlqt_
+cgemqr_
+cgeqr_
+cgesvdq_
+cgesvdx_
+cgesvj_
+cgetrf2_
+cgetsls_
+cgges3_
+cggev3_
+cgghd3_
+cggsvd3_
+cggsvp3_
+cgsvj0_
+cgsvj1_
+chb2st_kernels_
+chbev_2stage_
+chbevd_2stage_
+chbevx_2stage_
+checon_3_
+cheev_2stage_
+cheevd_2stage_
+cheevr_2stage_
+cheevx_2stage_
+chegv_2stage_
+chesv_aa_
+chesv_aa_2stage_
+chesv_rk_
+chetf2_rk_
+chetrd_2stage_
+chetrd_hb2st_
+chetrd_he2hb_
+chetrf_aa_
+chetrf_aa_2stage_
+chetrf_rk_
+chetri_3_
+chetri_3x_
+chetrs_3_
+chetrs_aa_
+chetrs_aa_2stage_
+clahef_aa_
+clahef_rk_
+clamswlq_
+clamtsqr_
+clarfy_
+claswlq_
+clasyf_aa_
+clasyf_rk_
+clatsqr_
+claunhr_col_getrfnp2_
+claunhr_col_getrfnp_
+cpotrf2_
+csycon_3_
+csyconvf_
+csyconvf_rook_
+csysv_aa_
+csysv_aa_2stage_
+csysv_rk_
+csytf2_rk_
+csytrf_aa_
+csytrf_aa_2stage_
+csytrf_rk_
+csytri_3_
+csytri_3x_
+csytrs_3_
+csytrs_aa_
+csytrs_aa_2stage_
+ctplqt2_
+ctplqt_
+ctpmlqt_
+ctrevc3_
+cungtsqr_
+cunhr_col_
+cunm22_
+dbdsvdx_
+dcombssq_
+dgelq_
+dgelqt3_
+dgelqt_
+dgemlq_
+dgemlqt_
+dgemqr_
+dgeqr_
+dgesvdq_
+dgesvdx_
+dgetrf2_
+dgetsls_
+dgges3_
+dggev3_
+dgghd3_
+dggsvd3_
+dggsvp3_
+dlamswlq_
+dlamtsqr_
+dlaorhr_col_getrfnp2_
+dlaorhr_col_getrfnp_
+dlarfy_
+dlaswlq_
+dlasyf_aa_
+dlasyf_rk_
+dlatsqr_
+dorgtsqr_
+dorhr_col_
+dorm22_
+dpotrf2_
+dsb2st_kernels_
+dsbev_2stage_
+dsbevd_2stage_
+dsbevx_2stage_
+dsycon_3_
+dsyconvf_
+dsyconvf_rook_
+dsyev_2stage_
+dsyevd_2stage_
+dsyevr_2stage_
+dsyevx_2stage_
+dsygv_2stage_
+dsysv_aa_
+dsysv_aa_2stage_
+dsysv_rk_
+dsytf2_rk_
+dsytrd_2stage_
+dsytrd_sb2st_
+dsytrd_sy2sb_
+dsytrf_aa_
+dsytrf_aa_2stage_
+dsytrf_rk_
+dsytri_3_
+dsytri_3x_
+dsytrs_3_
+dsytrs_aa_
+dsytrs_aa_2stage_
+dtplqt2_
+dtplqt_
+dtpmlqt_
+dtrevc3_
+ilaenv2stage_
+iparam2stage_
+sbdsvdx_
+scombssq_
+sgelq_
+sgelqt3_
+sgelqt_
+sgemlq_
+sgemlqt_
+sgemqr_
+sgeqr_
+sgesvdq_
+sgesvdx_
+sgetrf2_
+sgetsls_
+sgges3_
+sggev3_
+sgghd3_
+sggsvd3_
+sggsvp3_
+slamswlq_
+slamtsqr_
+slaorhr_col_getrfnp2_
+slaorhr_col_getrfnp_
+slarfy_
+slaswlq_
+slasyf_aa_
+slasyf_rk_
+slatsqr_
+sorgtsqr_
+sorhr_col_
+sorm22_
+spotrf2_
+ssb2st_kernels_
+ssbev_2stage_
+ssbevd_2stage_
+ssbevx_2stage_
+ssycon_3_
+ssyconvf_
+ssyconvf_rook_
+ssyev_2stage_
+ssyevd_2stage_
+ssyevr_2stage_
+ssyevx_2stage_
+ssygv_2stage_
+ssysv_aa_
+ssysv_aa_2stage_
+ssysv_rk_
+ssytf2_rk_
+ssytrd_2stage_
+ssytrd_sb2st_
+ssytrd_sy2sb_
+ssytrf_aa_
+ssytrf_aa_2stage_
+ssytrf_rk_
+ssytri_3_
+ssytri_3x_
+ssytrs_3_
+ssytrs_aa_
+ssytrs_aa_2stage_
+stplqt2_
+stplqt_
+stpmlqt_
+strevc3_
+zgejsv_
+zgelq_
+zgelqt3_
+zgelqt_
+zgemlq_
+zgemlqt_
+zgemqr_
+zgeqr_
+zgesvdq_
+zgesvdx_
+zgesvj_
+zgetrf2_
+zgetsls_
+zgges3_
+zggev3_
+zgghd3_
+zggsvd3_
+zggsvp3_
+zgsvj0_
+zgsvj1_
+zhb2st_kernels_
+zhbev_2stage_
+zhbevd_2stage_
+zhbevx_2stage_
+zhecon_3_
+zheev_2stage_
+zheevd_2stage_
+zheevr_2stage_
+zheevx_2stage_
+zhegv_2stage_
+zhesv_aa_
+zhesv_aa_2stage_
+zhesv_rk_
+zhetf2_rk_
+zhetrd_2stage_
+zhetrd_hb2st_
+zhetrd_he2hb_
+zhetrf_aa_
+zhetrf_aa_2stage_
+zhetrf_rk_
+zhetri_3_
+zhetri_3x_
+zhetrs_3_
+zhetrs_aa_
+zhetrs_aa_2stage_
+zlahef_aa_
+zlahef_rk_
+zlamswlq_
+zlamtsqr_
+zlarfy_
+zlaswlq_
+zlasyf_aa_
+zlasyf_rk_
+zlatsqr_
+zlaunhr_col_getrfnp2_
+zlaunhr_col_getrfnp_
+zpotrf2_
+zsycon_3_
+zsyconvf_
+zsyconvf_rook_
+zsysv_aa_
+zsysv_aa_2stage_
+zsysv_rk_
+zsytf2_rk_
+zsytrf_aa_
+zsytrf_aa_2stage_
+zsytrf_rk_
+zsytri_3_
+zsytri_3x_
+zsytrs_3_
+zsytrs_aa_
+zsytrs_aa_2stage_
+ztplqt2_
+ztplqt_
+ztpmlqt_
+ztrevc3_
+zungtsqr_
+zunhr_col_
+zunm22_

liblapacke.so.3:

Removed(!):
-LAPACKE_clagge
-LAPACKE_clagge_work
-LAPACKE_claghe
-LAPACKE_claghe_work
-LAPACKE_clagsy
-LAPACKE_clagsy_work
-LAPACKE_clatms
-LAPACKE_clatms_work
-LAPACKE_dlagge
-LAPACKE_dlagge_work
-LAPACKE_dlagsy
-LAPACKE_dlagsy_work
-LAPACKE_dlatms
-LAPACKE_dlatms_work
-LAPACKE_slagge
-LAPACKE_slagge_work
-LAPACKE_slagsy
-LAPACKE_slagsy_work
-LAPACKE_slatms
-LAPACKE_slatms_work
-LAPACKE_zlagge
-LAPACKE_zlagge_work
-LAPACKE_zlaghe
-LAPACKE_zlaghe_work
-LAPACKE_zlagsy
-LAPACKE_zlagsy_work
-LAPACKE_zlatms
-LAPACKE_zlatms_work

Added: (plus a _work variant)

+LAPACKE_cgejsv
+LAPACKE_cgelq
+LAPACKE_cgemlq
+LAPACKE_cgemqr
+LAPACKE_cgeqr
+LAPACKE_cgesvdq
+LAPACKE_cgesvdx
+LAPACKE_cgesvj
+LAPACKE_cgetrf2
+LAPACKE_cgetsls
+LAPACKE_cgges3
+LAPACKE_cggev3
+LAPACKE_cgghd3
+LAPACKE_cggsvd3
+LAPACKE_cggsvp3
+LAPACKE_chbev_2stage
+LAPACKE_chbevd_2stage
+LAPACKE_chbevx_2stage
+LAPACKE_checon_3
+LAPACKE_cheev_2stage
+LAPACKE_cheevd_2stage
+LAPACKE_cheevr_2stage
+LAPACKE_cheevx_2stage
+LAPACKE_chegv_2stage
+LAPACKE_chesv_aa
+LAPACKE_chesv_aa_2stage
+LAPACKE_chesv_rk
+LAPACKE_chetrf_aa
+LAPACKE_chetrf_aa_2stage
+LAPACKE_chetrf_rk
+LAPACKE_chetrf_rook
+LAPACKE_chetri_3
+LAPACKE_chetrs_3
+LAPACKE_chetrs_aa
+LAPACKE_chetrs_aa_2stage
+LAPACKE_chetrs_rook
+LAPACKE_clacrm
+LAPACKE_clapmt
+LAPACKE_clarcm
+LAPACKE_clascl
+LAPACKE_classq
+LAPACKE_cpotrf2
+LAPACKE_csycon_3
+LAPACKE_csysv_aa
+LAPACKE_csysv_aa_2stage
+LAPACKE_csysv_rk
+LAPACKE_csytrf_aa
+LAPACKE_csytrf_aa_2stage
+LAPACKE_csytrf_rk
+LAPACKE_csytrf_rook
+LAPACKE_csytri_3
+LAPACKE_csytrs_3
+LAPACKE_csytrs_aa
+LAPACKE_csytrs_aa_2stage
+LAPACKE_csytrs_rook
+LAPACKE_cuncsd2by1
+LAPACKE_dbdsvdx
+LAPACKE_dgelq
+LAPACKE_dgemlq
+LAPACKE_dgemqr
+LAPACKE_dgeqr
+LAPACKE_dgesvdq
+LAPACKE_dgesvdx
+LAPACKE_dgetrf2
+LAPACKE_dgetsls
+LAPACKE_dgges3
+LAPACKE_dggev3
+LAPACKE_dgghd3
+LAPACKE_dggsvd3
+LAPACKE_dggsvp3
+LAPACKE_dlapmt
+LAPACKE_dlascl
+LAPACKE_dlassq
+LAPACKE_dorcsd2by1
+LAPACKE_dpotrf2
+LAPACKE_dsbev_2stage
+LAPACKE_dsbevd_2stage
+LAPACKE_dsbevx_2stage
+LAPACKE_dsycon_3
+LAPACKE_dsyev_2stage
+LAPACKE_dsyevd_2stage
+LAPACKE_dsyevr_2stage
+LAPACKE_dsyevx_2stage
+LAPACKE_dsygv_2stage
+LAPACKE_dsysv_aa
+LAPACKE_dsysv_aa_2stage
+LAPACKE_dsysv_rk
+LAPACKE_dsytrf_aa
+LAPACKE_dsytrf_aa_2stage
+LAPACKE_dsytrf_rk
+LAPACKE_dsytrf_rook
+LAPACKE_dsytri_3
+LAPACKE_dsytrs_3
+LAPACKE_dsytrs_aa
+LAPACKE_dsytrs_aa_2stage
+LAPACKE_dsytrs_rook
+LAPACKE_get_nancheck
+LAPACKE_sbdsvdx
+LAPACKE_set_nancheck
+LAPACKE_sgelq
+LAPACKE_sgemlq
+LAPACKE_sgemqr
+LAPACKE_sgeqr
+LAPACKE_sgesvdq
+LAPACKE_sgesvdx
+LAPACKE_sgetrf2
+LAPACKE_sgetsls
+LAPACKE_sgges3
+LAPACKE_sggev3
+LAPACKE_sgghd3
+LAPACKE_sggsvd3
+LAPACKE_sggsvp3
+LAPACKE_slapmt
+LAPACKE_slascl
+LAPACKE_slassq
+LAPACKE_sorcsd2by1
+LAPACKE_spotrf2
+LAPACKE_ssbev_2stage
+LAPACKE_ssbevd_2stage
+LAPACKE_ssbevx_2stage
+LAPACKE_ssycon_3
+LAPACKE_ssyev_2stage
+LAPACKE_ssyevd_2stage
+LAPACKE_ssyevr_2stage
+LAPACKE_ssyevx_2stage
+LAPACKE_ssygv_2stage
+LAPACKE_ssysv_aa
+LAPACKE_ssysv_aa_2stage
+LAPACKE_ssysv_rk
+LAPACKE_ssytrf_aa
+LAPACKE_ssytrf_aa_2stage
+LAPACKE_ssytrf_rk
+LAPACKE_ssytrf_rook
+LAPACKE_ssytri_3
+LAPACKE_ssytrs_3
+LAPACKE_ssytrs_aa
+LAPACKE_ssytrs_aa_2stage
+LAPACKE_ssytrs_rook
+LAPACKE_stpqrt
+LAPACKE_zgejsv
+LAPACKE_zgelq
+LAPACKE_zgemlq
+LAPACKE_zgemqr
+LAPACKE_zgeqr
+LAPACKE_zgesvdq
+LAPACKE_zgesvdx
+LAPACKE_zgesvj
+LAPACKE_zgetrf2
+LAPACKE_zgetsls
+LAPACKE_zgges3
+LAPACKE_zggev3
+LAPACKE_zgghd3
+LAPACKE_zggsvd3
+LAPACKE_zggsvp3
+LAPACKE_zhbev_2stage
+LAPACKE_zhbevd_2stage
+LAPACKE_zhbevx_2stage
+LAPACKE_zhecon_3
+LAPACKE_zheev_2stage
+LAPACKE_zheevd_2stage
+LAPACKE_zheevr_2stage
+LAPACKE_zheevx_2stage
+LAPACKE_zhegv_2stage
+LAPACKE_zhesv_aa
+LAPACKE_zhesv_aa_2stage
+LAPACKE_zhesv_rk
+LAPACKE_zhetrf_aa
+LAPACKE_zhetrf_aa_2stage
+LAPACKE_zhetrf_rk
+LAPACKE_zhetrf_rook
+LAPACKE_zhetri_3
+LAPACKE_zhetrs_3
+LAPACKE_zhetrs_aa
+LAPACKE_zhetrs_aa_2stage
+LAPACKE_zhetrs_rook
+LAPACKE_zlacrm
+LAPACKE_zlapmt
+LAPACKE_zlarcm
+LAPACKE_zlascl
+LAPACKE_zlassq
+LAPACKE_zpotrf2
+LAPACKE_zsycon_3
+LAPACKE_zsysv_aa
+LAPACKE_zsysv_aa_2stage
+LAPACKE_zsysv_rk
+LAPACKE_zsytrf_aa
+LAPACKE_zsytrf_aa_2stage
+LAPACKE_zsytrf_rk
+LAPACKE_zsytrf_rook
+LAPACKE_zsytri_3
+LAPACKE_zsytrs_3
+LAPACKE_zsytrs_aa
+LAPACKE_zsytrs_aa_2stage
+LAPACKE_zsytrs_rook
+LAPACKE_zuncsd2by1


The removed symbols are all from TMGLIB, there's LAPACKE_WITH_TMG that would
enable including those symbols.  The lapack removed symbols are from
TESTING/TMGLIB which we've included as fix for bsc#1087426
Comment 25 Richard Biener 2023-02-14 09:16:18 UTC
Since the lapacke TMG interfaces are part of the API (they are declared in the
header) I would argue we should include them, thus define LAPACKE_WITH_TMG even in Factory.
Comment 26 Richard Biener 2023-02-14 09:20:54 UTC
(In reply to Richard Biener from comment #25)
> Since the lapacke TMG interfaces are part of the API (they are declared in
> the
> header) I would argue we should include them, thus define LAPACKE_WITH_TMG
> even in Factory.

Note the original issue was likely that liblapacke.so.3 contained the testing
functions but they refer to LAPACK_zlatms and friends which were not included
and the fix was to include all of TMGLIB into liblapack.so.3 (instead of
just the strictly required subset to make --no-undefined work).
Comment 27 Egbert Eich 2023-02-14 09:51:59 UTC
Ok, so this problem was 'introduced' when the c-bindings were added and headers were not sufficiently separated. Fortran did not have this problem :/

TMG is a 'Test Matrix Generator' that is used to test LAPACK itself. 
It's probably not used by an 'ordinary' consumer of lapack.
There are projects using it - for testing LAPACK:
https://christoph-conrads.name/compiling-stetester/
and a paper:
https://netlib.org/lapack/lawnspdf/lawn182.pdf

Wonder if there's an option to provide LIBTMG as a separate library...
Comment 28 Egbert Eich 2023-02-14 10:14:15 UTC
(In reply to Richard Biener from comment #26)
> (In reply to Richard Biener from comment #25)
> > Since the lapacke TMG interfaces are part of the API (they are declared in
> > the
> > header) I would argue we should include them, thus define LAPACKE_WITH_TMG
> > even in Factory.
> 
> Note the original issue was likely that liblapacke.so.3 contained the testing
> functions but they refer to LAPACK_zlatms and friends which were not included
> and the fix was to include all of TMGLIB into liblapack.so.3 (instead of
> just the strictly required subset to make --no-undefined work).

LAPACKE_WITH_TMG=yes will do exactly that: build the TMGLIB C-Bindings into LAPACKE. It will not build ltmglib - nor include it into the lapack library.

Looking at all this, it seems like the c-bindings for tmglib have become part of the LAPACKE API (by virtue of not cleanly separated headers), these test functions require tmglib (the Fortran library). Thus, by shipping LAPACKE, there is probably no choice but to ship tmglib as well (probably separately, not as part of liblapack).
Comment 29 Richard Biener 2023-02-14 10:17:12 UTC
Does openblas have the TMGLIB subset in LAPACKE?  Does it have the declarations in the header?

We can always provide TMGLIB as static library only for example.  But then
users using the lapacke subset will have to adapt and link that manually.
Comment 30 Egbert Eich 2023-02-14 10:20:49 UTC
Right, but without lazy bindings, wouldn't you have to link against TMGLIB always when you link against lapacke?
It's a bit confusing if you have to include a static lib when you link against a dynamic liblapacke.
Comment 31 Egbert Eich 2023-02-14 10:29:42 UTC
About your question on OpenBLAS, yes it does. For the 'zlatms' example:
0000000001c7bfa0 T LAPACKE_zlatms
0000000001c7c1a0 T LAPACKE_zlatms_work
0000000001acf870 T zlatms_
Comment 32 Richard Biener 2023-02-14 13:39:27 UTC
(In reply to Egbert Eich from comment #31)
> About your question on OpenBLAS, yes it does. For the 'zlatms' example:
> 0000000001c7bfa0 T LAPACKE_zlatms
> 0000000001c7c1a0 T LAPACKE_zlatms_work
> 0000000001acf870 T zlatms_

And zlatms_ is in liblapacke.so.3 or in liblapack.so.3 for openblas?
Comment 33 Egbert Eich 2023-02-14 13:42:57 UTC
(In reply to Richard Biener from comment #32)
> (In reply to Egbert Eich from comment #31)
> > About your question on OpenBLAS, yes it does. For the 'zlatms' example:
> > 0000000001c7bfa0 T LAPACKE_zlatms
> > 0000000001c7c1a0 T LAPACKE_zlatms_work
> > 0000000001acf870 T zlatms_
> 
> And zlatms_ is in liblapacke.so.3 or in liblapack.so.3 for openblas?

It is in libopenblas_[serial|pthreads|openmp].so.0. In OpenBLAS there is just one lib, not 4 (or 5) separate ones. Sorry I was not clearer about this.
Comment 34 Richard Biener 2023-02-14 14:00:52 UTC
(In reply to Egbert Eich from comment #33)
> (In reply to Richard Biener from comment #32)
> > (In reply to Egbert Eich from comment #31)
> > > About your question on OpenBLAS, yes it does. For the 'zlatms' example:
> > > 0000000001c7bfa0 T LAPACKE_zlatms
> > > 0000000001c7c1a0 T LAPACKE_zlatms_work
> > > 0000000001acf870 T zlatms_
> > 
> > And zlatms_ is in liblapacke.so.3 or in liblapack.so.3 for openblas?
> 
> It is in libopenblas_[serial|pthreads|openmp].so.0. In OpenBLAS there is
> just one lib, not 4 (or 5) separate ones. Sorry I was not clearer about this.

Are the TMGLIB functions _not_ part of the LAPACKE API also included in libopenblas?
Comment 35 Egbert Eich 2023-02-14 16:05:29 UTC
(In reply to Richard Biener from comment #34) 
> Are the TMGLIB functions _not_ part of the LAPACKE API also included in
> libopenblas?

I've only checked a subset, all these were, yes.
Comment 36 Egbert Eich 2023-02-15 08:57:03 UTC
I've created a version of lapack with tmglib in my OBS home (home:eeich:branches:science).
Thinking about this some more, if we decide to add this, I'd opt for including the MATGEN functions in the lapack library: It it is getting more and more cumbersome to keep lapack and openblas in sync regarding their update-alternatives and keeping things in sync across all update paths.
Comment 37 OBSbugzilla Bot 2023-02-16 18:15:05 UTC
This is an autogenerated message for OBS integration:
This bug (1207989) was mentioned in
https://build.opensuse.org/request/show/1066236 Factory / lapack
Comment 38 Egbert Eich 2023-02-16 18:27:28 UTC
An update has been submitted to Factory.
Comment 44 Maintenance Automation 2023-03-17 08:30:15 UTC
SUSE-FU-2023:0789-1: An update that contains one feature and has six feature fixes can now be installed.

Category: feature (important)
Bug References: 1087426, 1166619, 1184786, 1207358, 1207563, 1207989
Jira References: PED-3628
Sources used:
openSUSE Leap 15.4 (src): lapack-3.9.0-150000.4.13.2, lapack-man-3.9.0-150000.4.13.2
Basesystem Module 15-SP4 (src): lapack-3.9.0-150000.4.13.2
Development Tools Module 15-SP4 (src): lapack-3.9.0-150000.4.13.2
SUSE Linux Enterprise High Performance Computing 15 SP1 LTSS 15-SP1 (src): lapack-3.9.0-150000.4.13.2
SUSE Linux Enterprise High Performance Computing 15 SP2 LTSS 15-SP2 (src): lapack-3.9.0-150000.4.13.2
SUSE Linux Enterprise High Performance Computing ESPOS 15 SP3 (src): lapack-3.9.0-150000.4.13.2
SUSE Linux Enterprise High Performance Computing LTSS 15 SP3 (src): lapack-3.9.0-150000.4.13.2
SUSE Linux Enterprise Real Time 15 SP3 (src): lapack-3.9.0-150000.4.13.2
SUSE Linux Enterprise Server 15 SP1 LTSS 15-SP1 (src): lapack-3.9.0-150000.4.13.2
SUSE Linux Enterprise Server 15 SP2 LTSS 15-SP2 (src): lapack-3.9.0-150000.4.13.2
SUSE Linux Enterprise Server 15 SP3 LTSS 15-SP3 (src): lapack-3.9.0-150000.4.13.2
SUSE Linux Enterprise Server for SAP Applications 15 SP1 (src): lapack-3.9.0-150000.4.13.2
SUSE Linux Enterprise Server for SAP Applications 15 SP2 (src): lapack-3.9.0-150000.4.13.2
SUSE Linux Enterprise Server for SAP Applications 15 SP3 (src): lapack-3.9.0-150000.4.13.2
SUSE Manager Proxy 4.2 (src): lapack-3.9.0-150000.4.13.2
SUSE Manager Retail Branch Server 4.2 (src): lapack-3.9.0-150000.4.13.2
SUSE Manager Server 4.2 (src): lapack-3.9.0-150000.4.13.2
SUSE Enterprise Storage 7.1 (src): lapack-3.9.0-150000.4.13.2
SUSE Enterprise Storage 7 (src): lapack-3.9.0-150000.4.13.2
SUSE CaaS Platform 4.0 (src): lapack-3.9.0-150000.4.13.2

NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination.