[Numpy-discussion] querying backend information

[Stefan Seefeld]

Hello,

is there a way to query Numpy for information about backends (BLAS,
LAPACK, etc.) that it was compiled against, including compiler / linker
flags that were used ?
Consider the use-case where instead of calling a function such as
numpy.dot() I may want to call the appropriate backend directly using
the C API as an optimization technique. Is there a straight-forward way
to do that ?

In a somewhat related line of thought: Is there a way to see what
backends are available during Numpy compile-time ? I'm looking for a
list of flags to pick ATLAS/OpenBLAS/LAPACK/MKL or any other backend
that might be available, combined with variables (compiler and linker
flags, notably) I might have to set. Is that information available at all ?

Thanks,
Stefan

[Nathaniel Smith]

Post by Stefan Seefeld
Hello,
is there a way to query Numpy for information about backends (BLAS,
LAPACK, etc.) that it was compiled against, including compiler / linker
flags that were used ?
Consider the use-case where instead of calling a function such as
numpy.dot() I may want to call the appropriate backend directly using
the C API as an optimization technique. Is there a straight-forward way
to do that ?
In a somewhat related line of thought: Is there a way to see what
backends are available during Numpy compile-time ? I'm looking for a
list of flags to pick ATLAS/OpenBLAS/LAPACK/MKL or any other backend
that might be available, combined with variables (compiler and linker
flags, notably) I might have to set. Is that information available at all ?

NumPy does reveal some information about its configuration and
numpy.distutils does provide helper methods, but I'm not super
familiar with it so I'll let others answer that part.

Regarding the idea of "cutting out the middleman" and calling directly
into the appropriate backend via the C API, NumPy doesn't currently
expose any interface for doing this. There are some discussions with
Antoine from a few months back about this (and given that you work at
the same place I'm guessing the motivation is the same? :-)). For some
reason I'm failing to find the archives now, but the summary from off
the top of my head is: SciPy does expose an interface for this (via
cython and its PyCapsule tricks -- see [1]), NumPy is unlikely to
because we're wary of adding extra public interfaces and can't
guarantee that we even have a full BLAS/LAPACK available (sometimes we
fall back on a minimal vendored subset that's just enough for our
needs), you probably don't want to try and get into the business of
dynamically hunting down BLAS/LAPACK because it will be brittle and
expose you to all kinds of cross-platform linker issues, and if you
want to pull the clever stuff that scipy is doing out of scipy and put
it into its own dedicated blas/lapack package, then well, we need one
of those anyway [2].

-n

[1] https://github.com/scipy-conference/scipy_proceedings_2015/blob/master/papers/ian_henriksen/cython_blas_lapack_api.rst
[2] e.g. https://mail.scipy.org/pipermail/numpy-discussion/2015-January/072123.html

[Ralf Gommers]

Post by Nathaniel Smith

Post by Stefan Seefeld
Hello,
is there a way to query Numpy for information about backends (BLAS,
LAPACK, etc.) that it was compiled against, including compiler / linker
flags that were used ?
Consider the use-case where instead of calling a function such as
numpy.dot() I may want to call the appropriate backend directly using
the C API as an optimization technique. Is there a straight-forward way
to do that ?
In a somewhat related line of thought: Is there a way to see what
backends are available during Numpy compile-time ? I'm looking for a
list of flags to pick ATLAS/OpenBLAS/LAPACK/MKL or any other backend
that might be available, combined with variables (compiler and linker
flags, notably) I might have to set. Is that information available at

all ?
NumPy does reveal some information about its configuration and
numpy.distutils does provide helper methods, but I'm not super
familiar with it so I'll let others answer that part.

np.show_config()

Gives:

lapack_opt_info:
libraries = ['lapack', 'f77blas', 'cblas', 'atlas']
library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base']
define_macros = [('NO_ATLAS_INFO', -1)]
language = f77
include_dirs = ['/usr/include/atlas']
openblas_lapack_info:
NOT AVAILABLE
....
<etc>

It's a function with no docstring and not in the html docs (I think), so
that can certainly be improved.

Ralf

[Eric Moore]

Post by Ralf Gommers

Post by Nathaniel Smith

Post by Stefan Seefeld
Hello,
is there a way to query Numpy for information about backends (BLAS,
LAPACK, etc.) that it was compiled against, including compiler / linker
flags that were used ?
Consider the use-case where instead of calling a function such as
numpy.dot() I may want to call the appropriate backend directly using
the C API as an optimization technique. Is there a straight-forward way
to do that ?
In a somewhat related line of thought: Is there a way to see what
backends are available during Numpy compile-time ? I'm looking for a
list of flags to pick ATLAS/OpenBLAS/LAPACK/MKL or any other backend
that might be available, combined with variables (compiler and linker
flags, notably) I might have to set. Is that information available at

all ?
NumPy does reveal some information about its configuration and
numpy.distutils does provide helper methods, but I'm not super
familiar with it so I'll let others answer that part.

np.show_config()
libraries = ['lapack', 'f77blas', 'cblas', 'atlas']
library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base']
define_macros = [('NO_ATLAS_INFO', -1)]
language = f77
include_dirs = ['/usr/include/atlas']
NOT AVAILABLE
....
<etc>
It's a function with no docstring and not in the html docs (I think), so
that can certainly be improved.
Ralf

I don't think that show_config is what you want. Those are built time
values that aren't necessarily true at run time. For instance, numpy from
conda references directories that are not on my machine.

Eric