Discussion:
[Numpy-discussion] Using OpenBLAS for manylinux wheels
Matthew Brett
9 years ago
Permalink
Hi,

Olivier Grisel and I are working on building and testing manylinux
wheels for numpy and scipy.

We first thought that we should use ATLAS BLAS, but Olivier found that
my build of these could be very slow [1]. I set up a testing grid [2]
which found test errors for numpy and scipy using ATLAS wheels.

On the other hand, the same testing grid finds no errors or failures
[3] using latest OpenBLAS (0.2.17) and running tests for:

numpy
scipy
scikit-learn
numexpr
pandas
statsmodels

This is on the travis-ci ubuntu VMs.

Please do test on your own machines with something like this script [4]:

source test_manylinux.sh

We have worried in the past about the reliability of OpenBLAS, but I
find these tests reassuring.

Are there any other tests of OpenBLAS that we should run to assure
ourselves that it is safe to use?

Matthew

[1] https://github.com/matthew-brett/manylinux-builds/issues/4#issue-143530908
[2] https://travis-ci.org/matthew-brett/manylinux-testing/builds/118780781
[3] I disabled a few pandas tests which were failing for reasons not
related to BLAS. Some of the statsmodels test runs time out.
[4] https://gist.github.com/matthew-brett/2fd9d9a29e022c297634
Olivier Grisel
9 years ago
Permalink
I just tested those new openblas-based wheels on the linux virtualbox
setup that I used to report the following segfault back in February:

https://mail.scipy.org/pipermail/numpy-discussion/2016-February/074866.html
https://mail.scipy.org/pipermail/numpy-discussion/2016-February/074870.html

I cannot reproduce this problem anymore with this new version of
OpenBLAS. All scikit-learn tests pass (I had originally encountered
the segfault when runing the scikit-learn tests). I also ran the numpy
and scipy tests successfully on that machine.

What I find reassuring is that the upstream OpenBLAS has set up a
buildbot based CI to test OpenBLAS on many CPU architectures and is
running the scipy test continuously to detect regressions early on:

https://github.com/xianyi/OpenBLAS/issues/785
http://build.openblas.net/waterfall
https://github.com/xianyi/OpenBLAS-CI/
--
Olivier Grisel
Jonathan Helmus
9 years ago
Permalink
Matthew,

I ran the tests after installing the wheels on my machine running
Ubuntu 14.04. Three numpy tests failed with the GFORTRAN_1.4 error you
mentioned in post to the wheel-builds list recently. All other tests
passed. I can reproduce these failing tests in a Docker container if it
is helpful.

# python -c 'import numpy; numpy.test("full")'
Running unit tests for numpy
NumPy version 1.11.0
NumPy relaxed strides checking option: False
NumPy is installed in /usr/local/lib/python2.7/dist-packages/numpy
Python version 2.7.6 (default, Jun 22 2015, 17:58:13) [GCC 4.8.2]
nose version 1.3.7
...
======================================================================
ERROR: test_kind.TestKind.test_all
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/nose/case.py", line 381,
in setUp
try_run(self.inst, ('setup', 'setUp'))
File "/usr/local/lib/python2.7/dist-packages/nose/util.py", line 471,
in try_run
return func()
File
"/usr/local/lib/python2.7/dist-packages/numpy/f2py/tests/util.py", line
358, in setUp
module_name=self.module_name)
File
"/usr/local/lib/python2.7/dist-packages/numpy/f2py/tests/util.py", line
78, in wrapper
memo[key] = func(*a, **kw)
File
"/usr/local/lib/python2.7/dist-packages/numpy/f2py/tests/util.py", line
149, in build_module
__import__(module_name)
ImportError:
/usr/local/lib/python2.7/dist-packages/numpy/core/../.libs/libgfortran.so.3:
version `GFORTRAN_1.4' not found (required by
/tmp/tmptYznnz/_test_ext_module_5405.so)

======================================================================
ERROR: test_mixed.TestMixed.test_all
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/nose/case.py", line 381,
in setUp
try_run(self.inst, ('setup', 'setUp'))
File "/usr/local/lib/python2.7/dist-packages/nose/util.py", line 471,
in try_run
return func()
File
"/usr/local/lib/python2.7/dist-packages/numpy/f2py/tests/util.py", line
358, in setUp
module_name=self.module_name)
File
"/usr/local/lib/python2.7/dist-packages/numpy/f2py/tests/util.py", line
78, in wrapper
memo[key] = func(*a, **kw)
File
"/usr/local/lib/python2.7/dist-packages/numpy/f2py/tests/util.py", line
149, in build_module
__import__(module_name)
ImportError:
/usr/local/lib/python2.7/dist-packages/numpy/core/../.libs/libgfortran.so.3:
version `GFORTRAN_1.4' not found (required by
/tmp/tmptYznnz/_test_ext_module_5405.so)

======================================================================
ERROR: test_mixed.TestMixed.test_docstring
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/nose/case.py", line 381,
in setUp
try_run(self.inst, ('setup', 'setUp'))
File "/usr/local/lib/python2.7/dist-packages/nose/util.py", line 471,
in try_run
return func()
File
"/usr/local/lib/python2.7/dist-packages/numpy/f2py/tests/util.py", line
358, in setUp
module_name=self.module_name)
File
"/usr/local/lib/python2.7/dist-packages/numpy/f2py/tests/util.py", line
84, in wrapper
raise ret
ImportError:
/usr/local/lib/python2.7/dist-packages/numpy/core/../.libs/libgfortran.so.3:
version `GFORTRAN_1.4' not found (required by
/tmp/tmptYznnz/_test_ext_module_5405.so)

----------------------------------------------------------------------
Ran 6322 tests in 136.678s

FAILED (KNOWNFAIL=6, SKIP=11, errors=3)


Cheers,

- Jonathan Helmus
Olivier Grisel
9 years ago
Permalink
The problem with the gfortran failures will be tackled by renaming the
vendored libgfortran.so library, see:

https://github.com/pypa/auditwheel/issues/24

This is orthogonal to the ATLAS vs OpenBLAS decision though.
--
Olivier
Freddy Rietdijk
9 years ago
Permalink
On Nix/NixOS we've been using OpenBLAS 0.2.14 for some time now because we
had some segmentation faults with 0.2.15 and scipy/scikitlearn. I've tested
the packages you listed, and more, with OpenBLAS 0.2.17 and encountered no
problems.
Post by Olivier Grisel
The problem with the gfortran failures will be tackled by renaming the
https://github.com/pypa/auditwheel/issues/24
This is orthogonal to the ATLAS vs OpenBLAS decision though.
--
Olivier
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Nathaniel Smith
9 years ago
Permalink
If OpenBLAS is looking like the easiest to support solution, then no
objections here. (If 0.2.17 is genuinely working well, then maybe we want
to switch to it on Windows too. I know Xianyi disabled some of the
problematic kernels for us -- maybe that's enough. Mostly I just don't want
to end up in the situation where we're trying to support a bunch of
differently broken builds on different platforms.)
...
Carl Kleffner
9 years ago
Permalink
I would like to see OpenBLAS support for numpy on windows. The latest
OpenBLAS windows builds numpy support for are on
https://bitbucket.org/carlkl/mingw-w64-for-python/downloads now. Scipy
wheels should work regardless if numpy was build with MSVC or with mingwpy.
It is only mandantory to agree about the BLAS implemenation used.

Carl
...
Matthew Brett
9 years ago
Permalink
Post by Carl Kleffner
I would like to see OpenBLAS support for numpy on windows. The latest
OpenBLAS windows builds numpy support for are on
https://bitbucket.org/carlkl/mingw-w64-for-python/downloads now. Scipy
wheels should work regardless if numpy was build with MSVC or with mingwpy.
It is only mandantory to agree about the BLAS implemenation used.
How do the tests look for `numpy.test("full")` and `scipy.test("full")
with OpenBLAS 0.2.17 on Windows?

Sorry if you said before.

Thanks,

Matthew
Matthew Brett
9 years ago
Permalink
...
Here is an update on progress:

We've now done a lot of testing on the Linux OpenBLAS wheels. They
pass all tests on Linux, with Intel kernels:

https://travis-ci.org/matthew-brett/manylinux-testing/builds/120825485
http://nipy.bic.berkeley.edu/builders/manylinux-2.7-debian/builds/22
http://nipy.bic.berkeley.edu/builders/manylinux-2.7-fedora/builds/10

Xianyi, the maintainer of OpenBLAS, is very helpfully running the
OpenBLAS buildbot nightly tests with numpy and scipy:

http://build.openblas.net/builders

There is still one BLAS-related failure on these tests on AMD chips:

https://github.com/xianyi/OpenBLAS-CI/issues/10

I propose to hold off distributing the OpenBLAS wheels until the
OpenBLAS tests are clean on the OpenBLAS buildbots - any objections?

Cheers,

Matthew
Olivier Grisel
9 years ago
Permalink
Post by Matthew Brett
Xianyi, the maintainer of OpenBLAS, is very helpfully running the
http://build.openblas.net/builders
https://github.com/xianyi/OpenBLAS-CI/issues/10
I propose to hold off distributing the OpenBLAS wheels until the
OpenBLAS tests are clean on the OpenBLAS buildbots - any objections?
I agree. If someone can understand the fortran code of that scipy test
failure to
extract a minimalistic reproduction case that only use a few BLAS calls
that would help.
--
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel
Nathaniel Smith
9 years ago
Permalink
...
Alternatively, would it make sense to add a local patch to our openblas
builds to blacklist the piledriver kernel and then distribute them now?
(I'm not immediately sure what would be involved in doing this but it seems
unlikely that it would require anything tricky?)

-n
Olivier Grisel
9 years ago
Permalink
Post by Nathaniel Smith
Post by Matthew Brett
I propose to hold off distributing the OpenBLAS wheels until the
OpenBLAS tests are clean on the OpenBLAS buildbots - any objections?
Alternatively, would it make sense to add a local patch to our openblas
builds to blacklist the piledriver kernel and then distribute them now? (I'm
not immediately sure what would be involved in doing this but it seems
unlikely that it would require anything tricky?)
I tried to force use the NEHALEM or the BARCELONA driver on a PILEDRIVER
AMD box and while it fixes the original test failure in isolve it
causes this other
scipy test to fail:

======================================================================
FAIL: test_nanmedian_all_axis (test_stats.TestNanFunc)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/scipy/stats/tests/test_stats.py",
line 242, in test_nanmedian_all_axis
assert_equal(len(w), 4)
File "/usr/local/lib/python2.7/site-packages/numpy/testing/utils.py",
line 375, in assert_equal
raise AssertionError(msg)
AssertionError:
Items are not equal:
ACTUAL: 1
DESIRED: 4
--
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel
Nathaniel Smith
9 years ago
Permalink
...
I'm reading this email next to
https://github.com/xianyi/OpenBLAS-CI/issues/10#issuecomment-206195714
and I'm confused :-).

-n
--
Nathaniel J. Smith -- https://vorpus.org
Olivier Grisel
9 years ago
Permalink
Yes sorry I forgot to update the thread. Actually I am no longer sure
how I go this error. I am re-running the full test suite because I
cannot reproduce it when running the test_stats.py module alone.
--
Olivier
Olivier Grisel
9 years ago
Permalink
I updated the issue:

https://github.com/xianyi/OpenBLAS-CI/issues/10#issuecomment-206195714

The random test_nanmedian_all_axis failure is unrelated to openblas
and should be ignored.
--
Olivier
Matthew Brett
9 years ago
Permalink
Post by Nathaniel Smith
https://github.com/xianyi/OpenBLAS-CI/issues/10#issuecomment-206195714
The random test_nanmedian_all_axis failure is unrelated to openblas
and should be ignored.
It looks like all is well now, at least for the OpenBLAS buildbots :
http://build.openblas.net/builders

Matthew

Loading...