Discussion:
linux wheels coming soon
(too old to reply)
Nathaniel Smith
2016-03-15 23:33:38 UTC
Permalink
Hi all,

Just a heads-up that we're planning to upload Linux wheels for numpy
to PyPI soon. Unless there's some objection, these will be using
ATLAS, just like the current Windows wheels, for the same reasons --
moving to something faster like OpenBLAS would be good, but given the
concerns about OpenBLAS's reliability we want to get something working
first and then worry about making it fast. (Plus it doesn't make sense
to ship different BLAS libraries on Windows versus Linux -- that just
multiplies our support burden for no reason.)

-n
--
Nathaniel J. Smith -- https://vorpus.org
Charles R Harris
2016-03-16 00:54:38 UTC
Permalink
Post by Nathaniel Smith
Hi all,
Just a heads-up that we're planning to upload Linux wheels for numpy
to PyPI soon. Unless there's some objection, these will be using
ATLAS, just like the current Windows wheels, for the same reasons --
moving to something faster like OpenBLAS would be good, but given the
concerns about OpenBLAS's reliability we want to get something working
first and then worry about making it fast. (Plus it doesn't make sense
to ship different BLAS libraries on Windows versus Linux -- that just
multiplies our support burden for no reason.)
Good news, thanks to all who have worked on this.

Question: what to do with the prerelease uploads on pypi after they are
outdated? I'm inclined to delete them, as there may be four of five of them
per release and that seems unnecessary clutter.

Chuck
Nathaniel Smith
2016-03-16 01:10:13 UTC
Permalink
Post by Charles R Harris
Post by Nathaniel Smith
Hi all,
Just a heads-up that we're planning to upload Linux wheels for numpy
to PyPI soon. Unless there's some objection, these will be using
ATLAS, just like the current Windows wheels, for the same reasons --
moving to something faster like OpenBLAS would be good, but given the
concerns about OpenBLAS's reliability we want to get something working
first and then worry about making it fast. (Plus it doesn't make sense
to ship different BLAS libraries on Windows versus Linux -- that just
multiplies our support burden for no reason.)
Good news, thanks to all who have worked on this.
Question: what to do with the prerelease uploads on pypi after they are
outdated? I'm inclined to delete them, as there may be four of five of them
per release and that seems unnecessary clutter.

I'd just leave them? Pypi doesn't care, and who knows, they might be useful
for archival purposes to someone. Plus this is less work :-)

-n
Charles R Harris
2016-03-16 01:36:29 UTC
Permalink
Post by Charles R Harris
Post by Charles R Harris
Post by Nathaniel Smith
Hi all,
Just a heads-up that we're planning to upload Linux wheels for numpy
to PyPI soon. Unless there's some objection, these will be using
ATLAS, just like the current Windows wheels, for the same reasons --
moving to something faster like OpenBLAS would be good, but given the
concerns about OpenBLAS's reliability we want to get something working
first and then worry about making it fast. (Plus it doesn't make sense
to ship different BLAS libraries on Windows versus Linux -- that just
multiplies our support burden for no reason.)
Good news, thanks to all who have worked on this.
Question: what to do with the prerelease uploads on pypi after they are
outdated? I'm inclined to delete them, as there may be four of five of them
per release and that seems unnecessary clutter.
I'd just leave them? Pypi doesn't care, and who knows, they might be
useful for archival purposes to someone. Plus this is less work :-)
Less work than hitting the delete button? Oh, my aching finger ;)

Chuck
Peter Cock
2016-03-24 15:04:23 UTC
Permalink
Hi Nathaniel,

Will you be providing portable Linux wheels aka manylinux1?
https://www.python.org/dev/peps/pep-0513/

Does this also open up the door to releasing wheels for SciPy
too?

While speeding up "pip install" would be of benefit in itself,
I am particularly keen to see this for use within automated
testing frameworks like TravisCI where currently having to
install NumPy (and SciPy) from source is an unreasonable
overhead.

Many thanks to everyone working on this,

Peter
Post by Nathaniel Smith
Hi all,
Just a heads-up that we're planning to upload Linux wheels for numpy
to PyPI soon. Unless there's some objection, these will be using
ATLAS, just like the current Windows wheels, for the same reasons --
moving to something faster like OpenBLAS would be good, but given the
concerns about OpenBLAS's reliability we want to get something working
first and then worry about making it fast. (Plus it doesn't make sense
to ship different BLAS libraries on Windows versus Linux -- that just
multiplies our support burden for no reason.)
-n
--
Nathaniel J. Smith -- https://vorpus.org
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Ralf Gommers
2016-03-24 17:35:40 UTC
Permalink
Post by Peter Cock
Hi Nathaniel,
Will you be providing portable Linux wheels aka manylinux1?
https://www.python.org/dev/peps/pep-0513/
Does this also open up the door to releasing wheels for SciPy
too?
That should work just fine.
Post by Peter Cock
While speeding up "pip install" would be of benefit in itself,
I am particularly keen to see this for use within automated
testing frameworks like TravisCI where currently having to
install NumPy (and SciPy) from source is an unreasonable
overhead.
There's already http://travis-dev-wheels.scipy.org/ (latest dev versions of
numpy and scipy) and http://travis-wheels.scikit-image.org/ (releases,
there are multiple sources for this one) for TravisCI setups to reuse.

Ralf
Post by Peter Cock
Many thanks to everyone working on this,
Peter
Post by Nathaniel Smith
Hi all,
Just a heads-up that we're planning to upload Linux wheels for numpy
to PyPI soon. Unless there's some objection, these will be using
ATLAS, just like the current Windows wheels, for the same reasons --
moving to something faster like OpenBLAS would be good, but given the
concerns about OpenBLAS's reliability we want to get something working
first and then worry about making it fast. (Plus it doesn't make sense
to ship different BLAS libraries on Windows versus Linux -- that just
multiplies our support burden for no reason.)
-n
--
Nathaniel J. Smith -- https://vorpus.org
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Nathaniel Smith
2016-03-24 18:37:33 UTC
Permalink
Post by Peter Cock
Hi Nathaniel,
Will you be providing portable Linux wheels aka manylinux1?
https://www.python.org/dev/peps/pep-0513/
Matthew Brett will (probably) do the actual work, but yeah, that's the idea
exactly. Note the author list on that PEP ;-)

-n
Peter Cock
2016-03-24 18:44:44 UTC
Permalink
Post by Nathaniel Smith
Post by Peter Cock
Hi Nathaniel,
Will you be providing portable Linux wheels aka manylinux1?
https://www.python.org/dev/peps/pep-0513/
Matthew Brett will (probably) do the actual work, but yeah, that's the idea
exactly. Note the author list on that PEP ;-)
-n
Yep - I was partly double checking, but also aware many folk
skim the NumPy list and might not be aware of PEP-513 and
the standardisation efforts going on.

Also in addition to http://travis-dev-wheels.scipy.org/ and
http://travis-wheels.scikit-image.org/ mentioned by Ralf there
is http://wheels.scipy.org/ which I presume will get the new
Linux wheels once they go live.

Is it possible to add a README to these listings explaining
what they are intended to be used for?

P.S. To save anyone else Googling, you can do things like this:

pip install -r requirements.txt --timeout 60 --trusted-host
travis-wheels.scikit-image.org -f
http://travis-wheels.scikit-image.org/

Thanks,

Peter
Nathaniel Smith
2016-03-25 02:46:07 UTC
Permalink
Post by Peter Cock
Post by Nathaniel Smith
Post by Peter Cock
Hi Nathaniel,
Will you be providing portable Linux wheels aka manylinux1?
https://www.python.org/dev/peps/pep-0513/
Matthew Brett will (probably) do the actual work, but yeah, that's the idea
exactly. Note the author list on that PEP ;-)
-n
Yep - I was partly double checking, but also aware many folk
skim the NumPy list and might not be aware of PEP-513 and
the standardisation efforts going on.
Also in addition to http://travis-dev-wheels.scipy.org/ and
http://travis-wheels.scikit-image.org/ mentioned by Ralf there
is http://wheels.scipy.org/ which I presume will get the new
Linux wheels once they go live.
The new wheels will go up on pypi, and I guess once everyone has
wheels on pypi then these ad-hoc wheel servers that existed only as a
way to distribute Linux wheels will become obsolete.

(travis-dev-wheels will remain useful, though, because its purpose is
to hold up-to-the-minute builds of project master branches to allow
downstream projects to get early warning of breaking changes -- we
don't plan to upload to pypi after every commit :-).)

-n
--
Nathaniel J. Smith -- https://vorpus.org
Robert T. McGibbon
2016-03-25 03:02:03 UTC
Permalink
I suspect that many of the maintainers of major scipy-ecosystem projects
are aware of these (or other similar) travis wheel caches, but would guess
that the pool of travis-ci python users who weren't aware of these wheel
caches is much much larger. So there will still be a lot of travis-ci clock
cycles saved by manylinux wheels.

-Robert
Post by Nathaniel Smith
Post by Peter Cock
Post by Nathaniel Smith
Post by Peter Cock
Hi Nathaniel,
Will you be providing portable Linux wheels aka manylinux1?
https://www.python.org/dev/peps/pep-0513/
Matthew Brett will (probably) do the actual work, but yeah, that's the
idea
Post by Peter Cock
Post by Nathaniel Smith
exactly. Note the author list on that PEP ;-)
-n
Yep - I was partly double checking, but also aware many folk
skim the NumPy list and might not be aware of PEP-513 and
the standardisation efforts going on.
Also in addition to http://travis-dev-wheels.scipy.org/ and
http://travis-wheels.scikit-image.org/ mentioned by Ralf there
is http://wheels.scipy.org/ which I presume will get the new
Linux wheels once they go live.
The new wheels will go up on pypi, and I guess once everyone has
wheels on pypi then these ad-hoc wheel servers that existed only as a
way to distribute Linux wheels will become obsolete.
(travis-dev-wheels will remain useful, though, because its purpose is
to hold up-to-the-minute builds of project master branches to allow
downstream projects to get early warning of breaking changes -- we
don't plan to upload to pypi after every commit :-).)
-n
--
Nathaniel J. Smith -- https://vorpus.org
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
--
-Robert
Peter Cock
2016-03-25 13:39:45 UTC
Permalink
I suspect that many of the maintainers of major scipy-ecosystem projects are
aware of these (or other similar) travis wheel caches, but would guess that
the pool of travis-ci python users who weren't aware of these wheel caches
is much much larger. So there will still be a lot of travis-ci clock cycles
saved by manylinux wheels.
-Robert
Yes exactly. Availability of NumPy Linux wheels on PyPI is definitely something
I would suggest adding to the release notes. Hopefully this will help trigger
a general availability of wheels in the numpy-ecosystem :)

In the case of Travis CI, their VM images for Python already have a version
of NumPy installed, but having the latest version of NumPy and SciPy etc
available as Linux wheels would be very nice.

Peter

P.S.

As an aside, PyPI seems to be having trouble displaying the main NumPy
page https://pypi.python.org/pypi/numpy at the moment (Error 404 page):

https://bitbucket.org/pypa/pypi/issues/423/version-less-page-for-numpy-broken-error
Matthew Brett
2016-04-03 01:11:51 UTC
Permalink
Post by Peter Cock
I suspect that many of the maintainers of major scipy-ecosystem projects are
aware of these (or other similar) travis wheel caches, but would guess that
the pool of travis-ci python users who weren't aware of these wheel caches
is much much larger. So there will still be a lot of travis-ci clock cycles
saved by manylinux wheels.
-Robert
Yes exactly. Availability of NumPy Linux wheels on PyPI is definitely something
I would suggest adding to the release notes. Hopefully this will help trigger
a general availability of wheels in the numpy-ecosystem :)
In the case of Travis CI, their VM images for Python already have a version
of NumPy installed, but having the latest version of NumPy and SciPy etc
available as Linux wheels would be very nice.
We're very nearly there now.

The latest versions of numpy, scipy, scikit-image, pandas, numexpr,
statsmodels wheels for testing at
http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com/

Please do test with:

python -m install --upgrade pip

pip install --trusted-host=ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com
--find-links=http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com
numpy scipy scikit-learn numexpr

python -c 'import numpy; numpy.test("full")'
python -c 'import scipy; scipy.test("full")'

We would love to get any feedback as to whether these work on your machines.

Cheers,

Matthew
Olivier Grisel
2016-04-03 11:37:49 UTC
Permalink
typo:

python -m install --upgrade pip

should read:

python -m pip install --upgrade pip
--
Olivier
Olivier Grisel
2016-04-03 14:20:11 UTC
Permalink
I ran some tests on an image of the future ubuntu xenial that ships a
version of pip recent enough to install manylinux1 wheels by default
and everything looks fine.

Just to clarify, those wheels use openblas 0.2.17 that have proven to
be both fast and very stable on various CPU architectures while we
could not achieve similar results with atlas 3.10.
--
Olivier Grisel
Peter Cock
2016-04-04 16:02:08 UTC
Permalink
Post by Matthew Brett
Post by Peter Cock
I suspect that many of the maintainers of major scipy-ecosystem projects are
aware of these (or other similar) travis wheel caches, but would guess that
the pool of travis-ci python users who weren't aware of these wheel caches
is much much larger. So there will still be a lot of travis-ci clock cycles
saved by manylinux wheels.
-Robert
Yes exactly. Availability of NumPy Linux wheels on PyPI is definitely something
I would suggest adding to the release notes. Hopefully this will help trigger
a general availability of wheels in the numpy-ecosystem :)
In the case of Travis CI, their VM images for Python already have a version
of NumPy installed, but having the latest version of NumPy and SciPy etc
available as Linux wheels would be very nice.
We're very nearly there now.
The latest versions of numpy, scipy, scikit-image, pandas, numexpr,
statsmodels wheels for testing at
http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com/
...
We would love to get any feedback as to whether these work on your machines.
Hi Matthew,

Testing on a 64bit CentOS 6 machine with Python 3.5 compiled
from source under my home directory:


$ python3.5 -m pip install --upgrade pip
Requirement already up-to-date: pip in ./lib/python3.5/site-packages

$ python3.5 -m pip install
--trusted-host=ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com
--find-links=http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com
numpy scipy
Requirement already satisfied (use --upgrade to upgrade): numpy in
./lib/python3.5/site-packages
Requirement already satisfied (use --upgrade to upgrade): scipy in
./lib/python3.5/site-packages

$ python3.5 -m pip install
--trusted-host=ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com
--find-links=http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com
numpy scipy --upgrade
Collecting numpy
Downloading http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com/numpy-1.11.0-cp35-cp35m-manylinux1_x86_64.whl
(15.5MB)
100% |████████████████████████████████| 15.5MB 42.1MB/s
Collecting scipy
Downloading http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com/scipy-0.17.0-cp35-cp35m-manylinux1_x86_64.whl
(40.8MB)
100% |████████████████████████████████| 40.8MB 53.6MB/s
Installing collected packages: numpy, scipy
Found existing installation: numpy 1.10.4
Uninstalling numpy-1.10.4:
Successfully uninstalled numpy-1.10.4
Found existing installation: scipy 0.16.0
Uninstalling scipy-0.16.0:
Successfully uninstalled scipy-0.16.0
Successfully installed numpy-1.11.0 scipy-0.17.0


$ python3.5 -c 'import numpy; numpy.test("full")'
Running unit tests for numpy
NumPy version 1.11.0
NumPy relaxed strides checking option: False
NumPy is installed in /home/xxx/lib/python3.5/site-packages/numpy
Python version 3.5.0 (default, Sep 28 2015, 11:25:31) [GCC 4.4.7
20120313 (Red Hat 4.4.7-16)]
nose version 1.3.7
.............................................................................................................................................................................................................................S....................................................................................................................................................................KKK....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................S....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K.................................................................................................................................................................................................................................................................................................................................................................................................................................................K.......................................................................................................................................................................................................................................................................................................................................................................................................................................................K......................K............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 6332 tests in 243.029s

OK (KNOWNFAIL=7, SKIP=2)



So far so good, but there are a lot of deprecation warnings etc from SciPy,


$ python3.5 -c 'import scipy; scipy.test("full")'
Running unit tests for scipy
NumPy version 1.11.0
NumPy relaxed strides checking option: False
NumPy is installed in /home/xxx/lib/python3.5/site-packages/numpy
SciPy version 0.17.0
SciPy is installed in /home/xxx/lib/python3.5/site-packages/scipy
Python version 3.5.0 (default, Sep 28 2015, 11:25:31) [GCC 4.4.7
20120313 (Red Hat 4.4.7-16)]
nose version 1.3.7
[snip]
/home/xxx/lib/python3.5/site-packages/numpy/lib/utils.py:99:
DeprecationWarning: `rand` is deprecated!
numpy.testing.rand is deprecated in numpy 1.11. Use numpy.random.rand instead.
warnings.warn(depdoc, DeprecationWarning)
[snip]
/home/xxx/lib/python3.5/site-packages/scipy/io/arff/tests/test_arffread.py:254:
DeprecationWarning: parsing timezone aware datetimes is deprecated;
this will raise an error in the future
], dtype='datetime64[m]')
/home/xxx/lib/python3.5/site-packages/scipy/io/arff/arffread.py:638:
PendingDeprecationWarning: generator '_loadarff.<locals>.generator'
raised StopIteration
[snip]
/home/xxx/lib/python3.5/site-packages/scipy/sparse/tests/test_base.py:2425:
DeprecationWarning: This function is deprecated. Please call
randint(-5, 5 + 1) instead
I = np.random.random_integers(-M + 1, M - 1, size=NUM_SAMPLES)
[snip]
0-th dimension must be fixed to 3 but got 15
[snip]
----------------------------------------------------------------------
Ran 21407 tests in 741.602s

OK (KNOWNFAIL=130, SKIP=1775)


Hopefully I didn't miss anything important in hand editing the scipy output.

Peter
Matthew Brett
2016-04-04 17:47:03 UTC
Permalink
Hi,
Post by Peter Cock
Post by Matthew Brett
Post by Peter Cock
I suspect that many of the maintainers of major scipy-ecosystem projects are
aware of these (or other similar) travis wheel caches, but would guess that
the pool of travis-ci python users who weren't aware of these wheel caches
is much much larger. So there will still be a lot of travis-ci clock cycles
saved by manylinux wheels.
-Robert
Yes exactly. Availability of NumPy Linux wheels on PyPI is definitely something
I would suggest adding to the release notes. Hopefully this will help trigger
a general availability of wheels in the numpy-ecosystem :)
In the case of Travis CI, their VM images for Python already have a version
of NumPy installed, but having the latest version of NumPy and SciPy etc
available as Linux wheels would be very nice.
We're very nearly there now.
The latest versions of numpy, scipy, scikit-image, pandas, numexpr,
statsmodels wheels for testing at
http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com/
...
We would love to get any feedback as to whether these work on your machines.
Hi Matthew,
Testing on a 64bit CentOS 6 machine with Python 3.5 compiled
$ python3.5 -m pip install --upgrade pip
Requirement already up-to-date: pip in ./lib/python3.5/site-packages
$ python3.5 -m pip install
--trusted-host=ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com
--find-links=http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com
numpy scipy
Requirement already satisfied (use --upgrade to upgrade): numpy in
./lib/python3.5/site-packages
Requirement already satisfied (use --upgrade to upgrade): scipy in
./lib/python3.5/site-packages
$ python3.5 -m pip install
--trusted-host=ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com
--find-links=http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com
numpy scipy --upgrade
Collecting numpy
Downloading http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com/numpy-1.11.0-cp35-cp35m-manylinux1_x86_64.whl
(15.5MB)
100% |████████████████████████████████| 15.5MB 42.1MB/s
Collecting scipy
Downloading http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com/scipy-0.17.0-cp35-cp35m-manylinux1_x86_64.whl
(40.8MB)
100% |████████████████████████████████| 40.8MB 53.6MB/s
Installing collected packages: numpy, scipy
Found existing installation: numpy 1.10.4
Successfully uninstalled numpy-1.10.4
Found existing installation: scipy 0.16.0
Successfully uninstalled scipy-0.16.0
Successfully installed numpy-1.11.0 scipy-0.17.0
$ python3.5 -c 'import numpy; numpy.test("full")'
Running unit tests for numpy
NumPy version 1.11.0
NumPy relaxed strides checking option: False
NumPy is installed in /home/xxx/lib/python3.5/site-packages/numpy
Python version 3.5.0 (default, Sep 28 2015, 11:25:31) [GCC 4.4.7
20120313 (Red Hat 4.4.7-16)]
nose version 1.3.7
.............................................................................................................................................................................................................................S....................................................................................................................................................................KKK....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................S....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K.................................................................................................................................................................................................................................................................................................................................................................................................................................................K.......................................................................................................................................................................................................................................................................................................................................................................................................................................................K......................K............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 6332 tests in 243.029s
OK (KNOWNFAIL=7, SKIP=2)
So far so good, but there are a lot of deprecation warnings etc from SciPy,
$ python3.5 -c 'import scipy; scipy.test("full")'
Running unit tests for scipy
NumPy version 1.11.0
NumPy relaxed strides checking option: False
NumPy is installed in /home/xxx/lib/python3.5/site-packages/numpy
SciPy version 0.17.0
SciPy is installed in /home/xxx/lib/python3.5/site-packages/scipy
Python version 3.5.0 (default, Sep 28 2015, 11:25:31) [GCC 4.4.7
20120313 (Red Hat 4.4.7-16)]
nose version 1.3.7
[snip]
DeprecationWarning: `rand` is deprecated!
numpy.testing.rand is deprecated in numpy 1.11. Use numpy.random.rand instead.
warnings.warn(depdoc, DeprecationWarning)
[snip]
DeprecationWarning: parsing timezone aware datetimes is deprecated;
this will raise an error in the future
], dtype='datetime64[m]')
PendingDeprecationWarning: generator '_loadarff.<locals>.generator'
raised StopIteration
[snip]
DeprecationWarning: This function is deprecated. Please call
randint(-5, 5 + 1) instead
I = np.random.random_integers(-M + 1, M - 1, size=NUM_SAMPLES)
[snip]
0-th dimension must be fixed to 3 but got 15
[snip]
----------------------------------------------------------------------
Ran 21407 tests in 741.602s
OK (KNOWNFAIL=130, SKIP=1775)
Hopefully I didn't miss anything important in hand editing the scipy output.
Thanks a lot for testing.

I believe the deprecation warnings are expected, because numpy 1.11.0
introduced a new deprecation warning when using `random_integers`.
Scipy 0.17.0 is using `random_integers` in a few places.

Best,

Matthew
G Young
2016-04-04 18:26:26 UTC
Permalink
Matthew, you are correct. A lot of things happened with random integer
generation recently (including deprecating random_integers), but I believe
those warnings should be squashed in the up and coming version of SciPy
from what I remember.
Hi,
Post by Peter Cock
Post by Matthew Brett
On Fri, Mar 25, 2016 at 3:02 AM, Robert T. McGibbon <
Post by Robert T. McGibbon
I suspect that many of the maintainers of major scipy-ecosystem
projects are
Post by Peter Cock
Post by Matthew Brett
Post by Robert T. McGibbon
aware of these (or other similar) travis wheel caches, but would
guess that
Post by Peter Cock
Post by Matthew Brett
Post by Robert T. McGibbon
the pool of travis-ci python users who weren't aware of these wheel
caches
Post by Peter Cock
Post by Matthew Brett
Post by Robert T. McGibbon
is much much larger. So there will still be a lot of travis-ci clock
cycles
Post by Peter Cock
Post by Matthew Brett
Post by Robert T. McGibbon
saved by manylinux wheels.
-Robert
Yes exactly. Availability of NumPy Linux wheels on PyPI is definitely
something
Post by Peter Cock
Post by Matthew Brett
I would suggest adding to the release notes. Hopefully this will help
trigger
Post by Peter Cock
Post by Matthew Brett
a general availability of wheels in the numpy-ecosystem :)
In the case of Travis CI, their VM images for Python already have a
version
Post by Peter Cock
Post by Matthew Brett
of NumPy installed, but having the latest version of NumPy and SciPy
etc
Post by Peter Cock
Post by Matthew Brett
available as Linux wheels would be very nice.
We're very nearly there now.
The latest versions of numpy, scipy, scikit-image, pandas, numexpr,
statsmodels wheels for testing at
http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com/
Post by Peter Cock
Post by Matthew Brett
...
We would love to get any feedback as to whether these work on your
machines.
Post by Peter Cock
Hi Matthew,
Testing on a 64bit CentOS 6 machine with Python 3.5 compiled
$ python3.5 -m pip install --upgrade pip
Requirement already up-to-date: pip in ./lib/python3.5/site-packages
$ python3.5 -m pip install
--trusted-host=
ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com
Post by Peter Cock
--find-links=
http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com
Post by Peter Cock
numpy scipy
Requirement already satisfied (use --upgrade to upgrade): numpy in
./lib/python3.5/site-packages
Requirement already satisfied (use --upgrade to upgrade): scipy in
./lib/python3.5/site-packages
$ python3.5 -m pip install
--trusted-host=
ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com
Post by Peter Cock
--find-links=
http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com
Post by Peter Cock
numpy scipy --upgrade
Collecting numpy
Downloading
http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com/numpy-1.11.0-cp35-cp35m-manylinux1_x86_64.whl
Post by Peter Cock
(15.5MB)
100% |████████████████████████████████| 15.5MB 42.1MB/s
Collecting scipy
Downloading
http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com/scipy-0.17.0-cp35-cp35m-manylinux1_x86_64.whl
Post by Peter Cock
(40.8MB)
100% |████████████████████████████████| 40.8MB 53.6MB/s
Installing collected packages: numpy, scipy
Found existing installation: numpy 1.10.4
Successfully uninstalled numpy-1.10.4
Found existing installation: scipy 0.16.0
Successfully uninstalled scipy-0.16.0
Successfully installed numpy-1.11.0 scipy-0.17.0
$ python3.5 -c 'import numpy; numpy.test("full")'
Running unit tests for numpy
NumPy version 1.11.0
NumPy relaxed strides checking option: False
NumPy is installed in /home/xxx/lib/python3.5/site-packages/numpy
Python version 3.5.0 (default, Sep 28 2015, 11:25:31) [GCC 4.4.7
20120313 (Red Hat 4.4.7-16)]
nose version 1.3.7
.............................................................................................................................................................................................................................S....................................................................................................................................................................KKK....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................S....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................K.................................................................................................................................................................................................................................................................................................................................................................................................................................................K.......................................................................................................................................................................................................................................................................................................................................................................................................................................................K......................K............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Post by Peter Cock
----------------------------------------------------------------------
Ran 6332 tests in 243.029s
OK (KNOWNFAIL=7, SKIP=2)
So far so good, but there are a lot of deprecation warnings etc from
SciPy,
Post by Peter Cock
$ python3.5 -c 'import scipy; scipy.test("full")'
Running unit tests for scipy
NumPy version 1.11.0
NumPy relaxed strides checking option: False
NumPy is installed in /home/xxx/lib/python3.5/site-packages/numpy
SciPy version 0.17.0
SciPy is installed in /home/xxx/lib/python3.5/site-packages/scipy
Python version 3.5.0 (default, Sep 28 2015, 11:25:31) [GCC 4.4.7
20120313 (Red Hat 4.4.7-16)]
nose version 1.3.7
[snip]
DeprecationWarning: `rand` is deprecated!
numpy.testing.rand is deprecated in numpy 1.11. Use numpy.random.rand
instead.
Post by Peter Cock
warnings.warn(depdoc, DeprecationWarning)
[snip]
DeprecationWarning: parsing timezone aware datetimes is deprecated;
this will raise an error in the future
], dtype='datetime64[m]')
PendingDeprecationWarning: generator '_loadarff.<locals>.generator'
raised StopIteration
[snip]
DeprecationWarning: This function is deprecated. Please call
randint(-5, 5 + 1) instead
I = np.random.random_integers(-M + 1, M - 1, size=NUM_SAMPLES)
[snip]
0-th dimension must be fixed to 3 but got 15
[snip]
----------------------------------------------------------------------
Ran 21407 tests in 741.602s
OK (KNOWNFAIL=130, SKIP=1775)
Hopefully I didn't miss anything important in hand editing the scipy
output.
Thanks a lot for testing.
I believe the deprecation warnings are expected, because numpy 1.11.0
introduced a new deprecation warning when using `random_integers`.
Scipy 0.17.0 is using `random_integers` in a few places.
Best,
Matthew
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Matthew Brett
2016-04-13 02:15:19 UTC
Permalink
Hi,
Post by Matthew Brett
Post by Peter Cock
I suspect that many of the maintainers of major scipy-ecosystem projects are
aware of these (or other similar) travis wheel caches, but would guess that
the pool of travis-ci python users who weren't aware of these wheel caches
is much much larger. So there will still be a lot of travis-ci clock cycles
saved by manylinux wheels.
-Robert
Yes exactly. Availability of NumPy Linux wheels on PyPI is definitely something
I would suggest adding to the release notes. Hopefully this will help trigger
a general availability of wheels in the numpy-ecosystem :)
In the case of Travis CI, their VM images for Python already have a version
of NumPy installed, but having the latest version of NumPy and SciPy etc
available as Linux wheels would be very nice.
We're very nearly there now.
The latest versions of numpy, scipy, scikit-image, pandas, numexpr,
statsmodels wheels for testing at
http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com/
python -m pip install --upgrade pip
pip install --trusted-host=ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com
--find-links=http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com
numpy scipy scikit-learn numexpr
python -c 'import numpy; numpy.test("full")'
python -c 'import scipy; scipy.test("full")'
We would love to get any feedback as to whether these work on your machines.
I've just rebuilt these wheels with the just-released OpenBLAS 0.2.18.

OpenBLAS is now passing all its own tests and tests on numpy / scipy /
scikit-learn at http://build.openblas.net/builders

Our tests of the wheels look good too:

http://nipy.bic.berkeley.edu/builders/manylinux-2.7-debian
http://nipy.bic.berkeley.edu/builders/manylinux-2.7-debian
https://travis-ci.org/matthew-brett/manylinux-testing

So I think these are ready to go. I propose uploading these wheels
for numpy and scipy to pypi tomorrow unless anyone has an objection.

Cheers,

Matthew
Matthew Brett
2016-04-13 19:15:30 UTC
Permalink
Post by Matthew Brett
Hi,
Post by Matthew Brett
Post by Peter Cock
I suspect that many of the maintainers of major scipy-ecosystem projects are
aware of these (or other similar) travis wheel caches, but would guess that
the pool of travis-ci python users who weren't aware of these wheel caches
is much much larger. So there will still be a lot of travis-ci clock cycles
saved by manylinux wheels.
-Robert
Yes exactly. Availability of NumPy Linux wheels on PyPI is definitely something
I would suggest adding to the release notes. Hopefully this will help trigger
a general availability of wheels in the numpy-ecosystem :)
In the case of Travis CI, their VM images for Python already have a version
of NumPy installed, but having the latest version of NumPy and SciPy etc
available as Linux wheels would be very nice.
We're very nearly there now.
The latest versions of numpy, scipy, scikit-image, pandas, numexpr,
statsmodels wheels for testing at
http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com/
python -m pip install --upgrade pip
pip install --trusted-host=ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com
--find-links=http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com
numpy scipy scikit-learn numexpr
python -c 'import numpy; numpy.test("full")'
python -c 'import scipy; scipy.test("full")'
We would love to get any feedback as to whether these work on your machines.
I've just rebuilt these wheels with the just-released OpenBLAS 0.2.18.
OpenBLAS is now passing all its own tests and tests on numpy / scipy /
scikit-learn at http://build.openblas.net/builders
http://nipy.bic.berkeley.edu/builders/manylinux-2.7-debian
http://nipy.bic.berkeley.edu/builders/manylinux-2.7-debian
https://travis-ci.org/matthew-brett/manylinux-testing
So I think these are ready to go. I propose uploading these wheels
for numpy and scipy to pypi tomorrow unless anyone has an objection.
Done. If y'all are on linux, and you have pip >= 8.11, you should
now see this kind of thing:

$ pip install numpy scipy
Collecting numpy
Downloading numpy-1.11.0-cp27-cp27mu-manylinux1_x86_64.whl (15.3MB)
100% |████████████████████████████████| 15.3MB 61kB/s
Collecting scipy
Downloading scipy-0.17.0-cp27-cp27mu-manylinux1_x86_64.whl (39.5MB)
100% |████████████████████████████████| 39.5MB 24kB/s
Installing collected packages: numpy, scipy
Successfully installed numpy-1.11.0 scipy-0.17.0

Cheers,

Matthew
Olivier Grisel
2016-04-13 20:19:40 UTC
Permalink
\o/

Thank you very much Matthew. I will upload the scikit-learn wheels soon.
--
Olivier
Nathaniel Smith
2016-04-13 20:22:53 UTC
Permalink
Woot! \o/
Post by Matthew Brett
Hi,
Post by Matthew Brett
On Fri, Mar 25, 2016 at 3:02 AM, Robert T. McGibbon <
Post by Robert T. McGibbon
I suspect that many of the maintainers of major scipy-ecosystem
projects are
Post by Matthew Brett
Post by Matthew Brett
Post by Robert T. McGibbon
aware of these (or other similar) travis wheel caches, but would
guess that
Post by Matthew Brett
Post by Matthew Brett
Post by Robert T. McGibbon
the pool of travis-ci python users who weren't aware of these wheel
caches
Post by Matthew Brett
Post by Matthew Brett
Post by Robert T. McGibbon
is much much larger. So there will still be a lot of travis-ci clock
cycles
Post by Matthew Brett
Post by Matthew Brett
Post by Robert T. McGibbon
saved by manylinux wheels.
-Robert
Yes exactly. Availability of NumPy Linux wheels on PyPI is definitely
something
Post by Matthew Brett
Post by Matthew Brett
I would suggest adding to the release notes. Hopefully this will help
trigger
Post by Matthew Brett
Post by Matthew Brett
a general availability of wheels in the numpy-ecosystem :)
In the case of Travis CI, their VM images for Python already have a
version
Post by Matthew Brett
Post by Matthew Brett
of NumPy installed, but having the latest version of NumPy and SciPy
etc
Post by Matthew Brett
Post by Matthew Brett
available as Linux wheels would be very nice.
We're very nearly there now.
The latest versions of numpy, scipy, scikit-image, pandas, numexpr,
statsmodels wheels for testing at
http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com/
Post by Matthew Brett
Post by Matthew Brett
python -m pip install --upgrade pip
pip install --trusted-host=
ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com
Post by Matthew Brett
Post by Matthew Brett
--find-links=
http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com
Post by Matthew Brett
Post by Matthew Brett
numpy scipy scikit-learn numexpr
python -c 'import numpy; numpy.test("full")'
python -c 'import scipy; scipy.test("full")'
We would love to get any feedback as to whether these work on your
machines.
Post by Matthew Brett
I've just rebuilt these wheels with the just-released OpenBLAS 0.2.18.
OpenBLAS is now passing all its own tests and tests on numpy / scipy /
scikit-learn at http://build.openblas.net/builders
http://nipy.bic.berkeley.edu/builders/manylinux-2.7-debian
http://nipy.bic.berkeley.edu/builders/manylinux-2.7-debian
https://travis-ci.org/matthew-brett/manylinux-testing
So I think these are ready to go. I propose uploading these wheels
for numpy and scipy to pypi tomorrow unless anyone has an objection.
Done. If y'all are on linux, and you have pip >= 8.11, you should
$ pip install numpy scipy
Collecting numpy
Downloading numpy-1.11.0-cp27-cp27mu-manylinux1_x86_64.whl (15.3MB)
100% |████████████████████████████████| 15.3MB 61kB/s
Collecting scipy
Downloading scipy-0.17.0-cp27-cp27mu-manylinux1_x86_64.whl (39.5MB)
100% |████████████████████████████████| 39.5MB 24kB/s
Installing collected packages: numpy, scipy
Successfully installed numpy-1.11.0 scipy-0.17.0
Cheers,
Matthew
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
--
Nathaniel J. Smith -- https://vorpus.org <http://vorpus.org>
Oscar Benjamin
2016-04-13 20:29:40 UTC
Permalink
Post by Matthew Brett
Done. If y'all are on linux, and you have pip >= 8.11, you should
That's fantastic. Thanks Matt!

I just test installed this and ran numpy.test(). All tests passed but
then I got a segfault at the end by (semi-accidentally) hitting Ctrl-C
at the prompt:

$ python
Python 2.7.9 (default, Apr 2 2015, 15:33:21)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Post by Matthew Brett
Post by Matthew Brett
import numpy
numpy.test()
Running unit tests for numpy
<snip>
Ran 5781 tests in 72.238s

OK (KNOWNFAIL=6, SKIP=15)
<nose.result.TextTestResult run=5781 errors=0 failures=0>
Post by Matthew Brett
Post by Matthew Brett
Segmentation fault (core dumped)
It was stopped at the prompt and then I did Ctrl-C and then the
seg-fault message.

$ uname -a
Linux vnwulf 3.19.0-15-generic #15-Ubuntu SMP Thu Apr 16 23:32:37 UTC
2015 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 15.04
Release: 15.04
Codename: vivid

--
Oscar
Matthew Brett
2016-04-13 20:47:29 UTC
Permalink
On Wed, Apr 13, 2016 at 1:29 PM, Oscar Benjamin
Post by Oscar Benjamin
Post by Matthew Brett
Done. If y'all are on linux, and you have pip >= 8.11, you should
That's fantastic. Thanks Matt!
I just test installed this and ran numpy.test(). All tests passed but
then I got a segfault at the end by (semi-accidentally) hitting Ctrl-C
$ python
Python 2.7.9 (default, Apr 2 2015, 15:33:21)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Post by Matthew Brett
Post by Matthew Brett
import numpy
numpy.test()
Running unit tests for numpy
<snip>
Ran 5781 tests in 72.238s
OK (KNOWNFAIL=6, SKIP=15)
<nose.result.TextTestResult run=5781 errors=0 failures=0>
Post by Matthew Brett
Post by Matthew Brett
Segmentation fault (core dumped)
It was stopped at the prompt and then I did Ctrl-C and then the
seg-fault message.
$ uname -a
Linux vnwulf 3.19.0-15-generic #15-Ubuntu SMP Thu Apr 16 23:32:37 UTC
2015 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 15.04
Release: 15.04
Codename: vivid
Thanks so much for testing - that's very useful.

I get the same thing on my Debian Sid machine.

Actually I also get the same thing with a local compile against Debian
Post by Oscar Benjamin
Post by Matthew Brett
import numpy; numpy.test()
# Ctrl-C
https://gist.github.com/f6d8fb42f24689b39536a2416d717056

Do you get this as well?

Cheers,

Matthew
Oscar Benjamin
2016-04-13 22:11:03 UTC
Permalink
Post by Matthew Brett
On Wed, Apr 13, 2016 at 1:29 PM, Oscar Benjamin
Post by Oscar Benjamin
Post by Matthew Brett
Done. If y'all are on linux, and you have pip >= 8.11, you should
That's fantastic. Thanks Matt!
I just test installed this and ran numpy.test(). All tests passed but
then I got a segfault at the end by (semi-accidentally) hitting Ctrl-C
$ python
Python 2.7.9 (default, Apr 2 2015, 15:33:21)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Post by Matthew Brett
Post by Matthew Brett
import numpy
numpy.test()
Running unit tests for numpy
<snip>
Ran 5781 tests in 72.238s
OK (KNOWNFAIL=6, SKIP=15)
<nose.result.TextTestResult run=5781 errors=0 failures=0>
Post by Matthew Brett
Post by Matthew Brett
Segmentation fault (core dumped)
It was stopped at the prompt and then I did Ctrl-C and then the
seg-fault message.
$ uname -a
Linux vnwulf 3.19.0-15-generic #15-Ubuntu SMP Thu Apr 16 23:32:37 UTC
2015 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 15.04
Release: 15.04
Codename: vivid
Thanks so much for testing - that's very useful.
I get the same thing on my Debian Sid machine.
Actually I also get the same thing with a local compile against Debian
Post by Oscar Benjamin
Post by Matthew Brett
import numpy; numpy.test()
# Ctrl-C
https://gist.github.com/f6d8fb42f24689b39536a2416d717056
Do you get this as well?
It's late here but I'll test again tomorrow. What do I need to do to get
comparable output?

--
Oscar
Nathaniel Smith
2016-04-13 22:38:40 UTC
Permalink
I can reproduce in self-compiled 1.9, so it's not a new bug.

I think something's going wrong with NPY_SIGINT_ON / NPY_SIGINT_OFF,
where our special sigint handler is getting left in place even after
our code finishes running.

Skimming the code, my best guess is that this is due to a race
condition in how we save/restore the original signal handler, when
multiple threads are running numpy fftpack code at the same time (and
thus using NPY_SIGINT_{ON,OFF} from multiple threads).

-n
Post by Matthew Brett
On Wed, Apr 13, 2016 at 1:29 PM, Oscar Benjamin
Post by Oscar Benjamin
Post by Matthew Brett
Done. If y'all are on linux, and you have pip >= 8.11, you should
That's fantastic. Thanks Matt!
I just test installed this and ran numpy.test(). All tests passed but
then I got a segfault at the end by (semi-accidentally) hitting Ctrl-C
$ python
Python 2.7.9 (default, Apr 2 2015, 15:33:21)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Post by Matthew Brett
Post by Matthew Brett
import numpy
numpy.test()
Running unit tests for numpy
<snip>
Ran 5781 tests in 72.238s
OK (KNOWNFAIL=6, SKIP=15)
<nose.result.TextTestResult run=5781 errors=0 failures=0>
Post by Matthew Brett
Post by Matthew Brett
Segmentation fault (core dumped)
It was stopped at the prompt and then I did Ctrl-C and then the
seg-fault message.
$ uname -a
Linux vnwulf 3.19.0-15-generic #15-Ubuntu SMP Thu Apr 16 23:32:37 UTC
2015 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 15.04
Release: 15.04
Codename: vivid
Thanks so much for testing - that's very useful.
I get the same thing on my Debian Sid machine.
Actually I also get the same thing with a local compile against Debian
Post by Oscar Benjamin
Post by Matthew Brett
import numpy; numpy.test()
# Ctrl-C
https://gist.github.com/f6d8fb42f24689b39536a2416d717056
Do you get this as well?
Cheers,
Matthew
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
--
Nathaniel J. Smith -- https://vorpus.org
Nathaniel Smith
2016-04-14 00:46:31 UTC
Permalink
https://github.com/numpy/numpy/issues/7545
Post by Nathaniel Smith
I can reproduce in self-compiled 1.9, so it's not a new bug.
I think something's going wrong with NPY_SIGINT_ON / NPY_SIGINT_OFF,
where our special sigint handler is getting left in place even after
our code finishes running.
Skimming the code, my best guess is that this is due to a race
condition in how we save/restore the original signal handler, when
multiple threads are running numpy fftpack code at the same time (and
thus using NPY_SIGINT_{ON,OFF} from multiple threads).
-n
Post by Matthew Brett
On Wed, Apr 13, 2016 at 1:29 PM, Oscar Benjamin
Post by Oscar Benjamin
Post by Matthew Brett
Done. If y'all are on linux, and you have pip >= 8.11, you should
That's fantastic. Thanks Matt!
I just test installed this and ran numpy.test(). All tests passed but
then I got a segfault at the end by (semi-accidentally) hitting Ctrl-C
$ python
Python 2.7.9 (default, Apr 2 2015, 15:33:21)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Post by Matthew Brett
Post by Matthew Brett
import numpy
numpy.test()
Running unit tests for numpy
<snip>
Ran 5781 tests in 72.238s
OK (KNOWNFAIL=6, SKIP=15)
<nose.result.TextTestResult run=5781 errors=0 failures=0>
Post by Matthew Brett
Post by Matthew Brett
Segmentation fault (core dumped)
It was stopped at the prompt and then I did Ctrl-C and then the
seg-fault message.
$ uname -a
Linux vnwulf 3.19.0-15-generic #15-Ubuntu SMP Thu Apr 16 23:32:37 UTC
2015 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 15.04
Release: 15.04
Codename: vivid
Thanks so much for testing - that's very useful.
I get the same thing on my Debian Sid machine.
Actually I also get the same thing with a local compile against Debian
Post by Oscar Benjamin
Post by Matthew Brett
import numpy; numpy.test()
# Ctrl-C
https://gist.github.com/f6d8fb42f24689b39536a2416d717056
Do you get this as well?
Cheers,
Matthew
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
--
Nathaniel J. Smith -- https://vorpus.org
--
Nathaniel J. Smith -- https://vorpus.org
Jens Nielsen
2016-04-14 15:02:17 UTC
Permalink
I have tried testing the wheels in a project that runs tests on Travis's
Trusty infrastructure which. The wheels work great for python 3.5 and saves
us several minuts of runtime.

However, I am having trouble using the wheels on python 2.7 on the same
Trusty machines. It seems to be because the wheels are tagged as
cp27-cp27mu (numpy-1.11.0-cp27-cp27mu-manylinux1_x86_64.whl) where as
pip.pep425tags.get_abi_tag()
returns cp27m on this particular python version. (Stock python 2.7
installed on Travis 14.04 VMs) Any chance of a cp27m compatible wheel build?

best
Jens
Post by Nathaniel Smith
https://github.com/numpy/numpy/issues/7545
Post by Nathaniel Smith
I can reproduce in self-compiled 1.9, so it's not a new bug.
I think something's going wrong with NPY_SIGINT_ON / NPY_SIGINT_OFF,
where our special sigint handler is getting left in place even after
our code finishes running.
Skimming the code, my best guess is that this is due to a race
condition in how we save/restore the original signal handler, when
multiple threads are running numpy fftpack code at the same time (and
thus using NPY_SIGINT_{ON,OFF} from multiple threads).
-n
Post by Matthew Brett
On Wed, Apr 13, 2016 at 1:29 PM, Oscar Benjamin
Post by Oscar Benjamin
Post by Matthew Brett
Done. If y'all are on linux, and you have pip >= 8.11, you should
That's fantastic. Thanks Matt!
I just test installed this and ran numpy.test(). All tests passed but
then I got a segfault at the end by (semi-accidentally) hitting Ctrl-C
$ python
Python 2.7.9 (default, Apr 2 2015, 15:33:21)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Post by Matthew Brett
Post by Matthew Brett
import numpy
numpy.test()
Running unit tests for numpy
<snip>
Ran 5781 tests in 72.238s
OK (KNOWNFAIL=6, SKIP=15)
<nose.result.TextTestResult run=5781 errors=0 failures=0>
Post by Matthew Brett
Post by Matthew Brett
Segmentation fault (core dumped)
It was stopped at the prompt and then I did Ctrl-C and then the
seg-fault message.
$ uname -a
Linux vnwulf 3.19.0-15-generic #15-Ubuntu SMP Thu Apr 16 23:32:37 UTC
2015 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 15.04
Release: 15.04
Codename: vivid
Thanks so much for testing - that's very useful.
I get the same thing on my Debian Sid machine.
Actually I also get the same thing with a local compile against Debian
Post by Oscar Benjamin
Post by Matthew Brett
import numpy; numpy.test()
# Ctrl-C
https://gist.github.com/f6d8fb42f24689b39536a2416d717056
Do you get this as well?
Cheers,
Matthew
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
--
Nathaniel J. Smith -- https://vorpus.org
--
Nathaniel J. Smith -- https://vorpus.org
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Matthew Brett
2016-04-14 18:04:12 UTC
Permalink
Hi,
Post by Jens Nielsen
I have tried testing the wheels in a project that runs tests on Travis's
Trusty infrastructure which. The wheels work great for python 3.5 and saves
us several minuts of runtime.
However, I am having trouble using the wheels on python 2.7 on the same
Trusty machines. It seems to be because the wheels are tagged as cp27-cp27mu
(numpy-1.11.0-cp27-cp27mu-manylinux1_x86_64.whl) where as
pip.pep425tags.get_abi_tag() returns cp27m on this particular python
version. (Stock python 2.7 installed on Travis 14.04 VMs) Any chance of a
cp27m compatible wheel build?
Ouch - do you know where travis-ci's Python 2.7 comes from? I see
that the standard apt-get install -y python is a wide (mu) build...

Cheers,

Matthew
Benjamin Root
2016-04-14 18:11:17 UTC
Permalink
Are we going to have to have documentation somewhere making it clear that
the numpy wheel shouldn't be used in a conda environment? Not that I would
expect this issue to come up all that often, but I could imagine a scenario
where a non-scientist is simply using a base conda distribution because
that is what IT put on their system. Then they do "pip install ipython"
that indirectly brings in numpy (through the matplotlib dependency), and
end up with an incompatible numpy because they would have been linked
against different pythons?

Or is this not an issue?

Ben Root
Post by Matthew Brett
Hi,
Post by Jens Nielsen
I have tried testing the wheels in a project that runs tests on Travis's
Trusty infrastructure which. The wheels work great for python 3.5 and
saves
Post by Jens Nielsen
us several minuts of runtime.
However, I am having trouble using the wheels on python 2.7 on the same
Trusty machines. It seems to be because the wheels are tagged as
cp27-cp27mu
Post by Jens Nielsen
(numpy-1.11.0-cp27-cp27mu-manylinux1_x86_64.whl) where as
pip.pep425tags.get_abi_tag() returns cp27m on this particular python
version. (Stock python 2.7 installed on Travis 14.04 VMs) Any chance of a
cp27m compatible wheel build?
Ouch - do you know where travis-ci's Python 2.7 comes from? I see
that the standard apt-get install -y python is a wide (mu) build...
Cheers,
Matthew
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Matthew Brett
2016-04-14 18:26:27 UTC
Permalink
Hi,
Post by Benjamin Root
Are we going to have to have documentation somewhere making it clear that
the numpy wheel shouldn't be used in a conda environment? Not that I would
expect this issue to come up all that often, but I could imagine a scenario
where a non-scientist is simply using a base conda distribution because that
is what IT put on their system. Then they do "pip install ipython" that
indirectly brings in numpy (through the matplotlib dependency), and end up
with an incompatible numpy because they would have been linked against
different pythons?
Or is this not an issue?
I'm afraid I don't know conda at all, but I'm guessing that pip will
not install numpy when it is installed via conda.

So the potential difference is that, pre-wheel, if numpy was not
installed in your conda environment, then pip would build numpy from
source, whereas now you'll get a binary install.

I _think_ that Python's binary API specification
(pip.pep425tags.get_abi_tag()) should prevent pip from installing an
incompatible wheel. Are there any conda experts out there who can
give more detail, or more convincing assurance?

Cheers,

Matthew
G Young
2016-04-14 18:51:24 UTC
Permalink
Actually, conda pip will install the wheels that you put up. The good news
is: they all (by which I mean *numpy* and *scipy* both on 2.7 and 3.5) pass!
Post by Benjamin Root
Hi,
Post by Benjamin Root
Are we going to have to have documentation somewhere making it clear that
the numpy wheel shouldn't be used in a conda environment? Not that I
would
Post by Benjamin Root
expect this issue to come up all that often, but I could imagine a
scenario
Post by Benjamin Root
where a non-scientist is simply using a base conda distribution because
that
Post by Benjamin Root
is what IT put on their system. Then they do "pip install ipython" that
indirectly brings in numpy (through the matplotlib dependency), and end
up
Post by Benjamin Root
with an incompatible numpy because they would have been linked against
different pythons?
Or is this not an issue?
I'm afraid I don't know conda at all, but I'm guessing that pip will
not install numpy when it is installed via conda.
So the potential difference is that, pre-wheel, if numpy was not
installed in your conda environment, then pip would build numpy from
source, whereas now you'll get a binary install.
I _think_ that Python's binary API specification
(pip.pep425tags.get_abi_tag()) should prevent pip from installing an
incompatible wheel. Are there any conda experts out there who can
give more detail, or more convincing assurance?
Cheers,
Matthew
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Jonathan Helmus
2016-04-14 19:25:10 UTC
Permalink
Post by Matthew Brett
Hi,
Post by Benjamin Root
Are we going to have to have documentation somewhere making it clear that
the numpy wheel shouldn't be used in a conda environment? Not that I would
expect this issue to come up all that often, but I could imagine a scenario
where a non-scientist is simply using a base conda distribution because that
is what IT put on their system. Then they do "pip install ipython" that
indirectly brings in numpy (through the matplotlib dependency), and end up
with an incompatible numpy because they would have been linked against
different pythons?
Or is this not an issue?
I'm afraid I don't know conda at all, but I'm guessing that pip will
not install numpy when it is installed via conda.
Correct, pip will not (or at least should not, and did not in my tests)
install numpy over top of an existing conda installed numpy.
Unfortunately from my testing, conda will install a conda version of
numpy over top of a pip installed version. This may be the expected
behavior as conda maintains its own list of installed packages.
Post by Matthew Brett
So the potential difference is that, pre-wheel, if numpy was not
installed in your conda environment, then pip would build numpy from
source, whereas now you'll get a binary install.
I _think_ that Python's binary API specification
(pip.pep425tags.get_abi_tag()) should prevent pip from installing an
incompatible wheel. Are there any conda experts out there who can
give more detail, or more convincing assurance?
I tested "pip install numpy" in conda environments (conda's equivalent
to virtualenvs) which did not have numpy installed previously for Python
2.7, 3.4 and 3.5 in a Ubuntu 14.04 Docker container. In all cases numpy
was installed from the whl file and appeared to be functional. Running
the numpy test suite found three failing tests for Python 2.7 and 3.5
and 21 errors in Python 3.4. The 2.7 and 3.5 failures do not look
concerning but the 3.4 errors are a bit strange.
Logs are in
https://gist.github.com/jjhelmus/a433a66d56fb0e39b8ebde248ad3fe36


Cheers,

- Jonathan Helmus
Post by Matthew Brett
Cheers,
Matthew
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Matthew Brett
2016-04-14 19:57:35 UTC
Permalink
Post by Jonathan Helmus
Post by Matthew Brett
Hi,
Post by Benjamin Root
Are we going to have to have documentation somewhere making it clear that
the numpy wheel shouldn't be used in a conda environment? Not that I would
expect this issue to come up all that often, but I could imagine a scenario
where a non-scientist is simply using a base conda distribution because that
is what IT put on their system. Then they do "pip install ipython" that
indirectly brings in numpy (through the matplotlib dependency), and end up
with an incompatible numpy because they would have been linked against
different pythons?
Or is this not an issue?
I'm afraid I don't know conda at all, but I'm guessing that pip will
not install numpy when it is installed via conda.
Correct, pip will not (or at least should not, and did not in my tests)
install numpy over top of an existing conda installed numpy. Unfortunately
from my testing, conda will install a conda version of numpy over top of a
pip installed version. This may be the expected behavior as conda maintains
its own list of installed packages.
Post by Matthew Brett
So the potential difference is that, pre-wheel, if numpy was not
installed in your conda environment, then pip would build numpy from
source, whereas now you'll get a binary install.
I _think_ that Python's binary API specification
(pip.pep425tags.get_abi_tag()) should prevent pip from installing an
incompatible wheel. Are there any conda experts out there who can
give more detail, or more convincing assurance?
I tested "pip install numpy" in conda environments (conda's equivalent to
virtualenvs) which did not have numpy installed previously for Python 2.7,
3.4 and 3.5 in a Ubuntu 14.04 Docker container. In all cases numpy was
installed from the whl file and appeared to be functional. Running the
numpy test suite found three failing tests for Python 2.7 and 3.5 and 21
errors in Python 3.4. The 2.7 and 3.5 failures do not look concerning but
the 3.4 errors are a bit strange.
Logs are in
https://gist.github.com/jjhelmus/a433a66d56fb0e39b8ebde248ad3fe36
Thanks for testing. For:

docker run -ti --rm ubuntu:14.04 /bin/bash

apt-get update && apt-get install -y curl
curl -LO https://bootstrap.pypa.io/get-pip.py
python3 get-pip.py
pip install numpy nose
python3 -c "import numpy; numpy.test()"

I get:

FAILED (KNOWNFAIL=7, SKIP=17, errors=21)

This is stock Python 3.4 - so not a conda issue. It is definitely a
problem with the wheel because a compiled numpy wheel on the same
docker image:

apt-get update && apt-get install -y curl python3-dev
curl -LO https://bootstrap.pypa.io/get-pip.py
python3 get-pip.py
pip install --no-binary=:all: numpy nose
python3 -c "import numpy; numpy.test()"

gives no test errors.

It looks like we have some more work to do...

Cheers,

Matthew
Matthew Brett
2016-04-14 20:11:18 UTC
Permalink
Post by Matthew Brett
Post by Jonathan Helmus
Post by Matthew Brett
Hi,
Post by Benjamin Root
Are we going to have to have documentation somewhere making it clear that
the numpy wheel shouldn't be used in a conda environment? Not that I would
expect this issue to come up all that often, but I could imagine a scenario
where a non-scientist is simply using a base conda distribution because that
is what IT put on their system. Then they do "pip install ipython" that
indirectly brings in numpy (through the matplotlib dependency), and end up
with an incompatible numpy because they would have been linked against
different pythons?
Or is this not an issue?
I'm afraid I don't know conda at all, but I'm guessing that pip will
not install numpy when it is installed via conda.
Correct, pip will not (or at least should not, and did not in my tests)
install numpy over top of an existing conda installed numpy. Unfortunately
from my testing, conda will install a conda version of numpy over top of a
pip installed version. This may be the expected behavior as conda maintains
its own list of installed packages.
Post by Matthew Brett
So the potential difference is that, pre-wheel, if numpy was not
installed in your conda environment, then pip would build numpy from
source, whereas now you'll get a binary install.
I _think_ that Python's binary API specification
(pip.pep425tags.get_abi_tag()) should prevent pip from installing an
incompatible wheel. Are there any conda experts out there who can
give more detail, or more convincing assurance?
I tested "pip install numpy" in conda environments (conda's equivalent to
virtualenvs) which did not have numpy installed previously for Python 2.7,
3.4 and 3.5 in a Ubuntu 14.04 Docker container. In all cases numpy was
installed from the whl file and appeared to be functional. Running the
numpy test suite found three failing tests for Python 2.7 and 3.5 and 21
errors in Python 3.4. The 2.7 and 3.5 failures do not look concerning but
the 3.4 errors are a bit strange.
Logs are in
https://gist.github.com/jjhelmus/a433a66d56fb0e39b8ebde248ad3fe36
docker run -ti --rm ubuntu:14.04 /bin/bash
apt-get update && apt-get install -y curl
curl -LO https://bootstrap.pypa.io/get-pip.py
python3 get-pip.py
pip install numpy nose
python3 -c "import numpy; numpy.test()"
FAILED (KNOWNFAIL=7, SKIP=17, errors=21)
This is stock Python 3.4 - so not a conda issue. It is definitely a
problem with the wheel because a compiled numpy wheel on the same
apt-get update && apt-get install -y curl python3-dev
curl -LO https://bootstrap.pypa.io/get-pip.py
python3 get-pip.py
pip install --no-binary=:all: numpy nose
python3 -c "import numpy; numpy.test()"
gives no test errors.
It looks like we have some more work to do...
Actually, I can solve these errors by first doing:

apt-get install gcc

I think these must be bugs in the numpy tests where numpy is assuming
a functional compiler.

Does the conda numpy give test errors when there is no compiler?

Cheers,

Matthew
Jonathan Helmus
2016-04-14 20:47:19 UTC
Permalink
Post by Matthew Brett
Post by Matthew Brett
Post by Jonathan Helmus
Post by Matthew Brett
Hi,
Post by Benjamin Root
Are we going to have to have documentation somewhere making it clear that
the numpy wheel shouldn't be used in a conda environment? Not that I would
expect this issue to come up all that often, but I could imagine a scenario
where a non-scientist is simply using a base conda distribution because that
is what IT put on their system. Then they do "pip install ipython" that
indirectly brings in numpy (through the matplotlib dependency), and end up
with an incompatible numpy because they would have been linked against
different pythons?
Or is this not an issue?
I'm afraid I don't know conda at all, but I'm guessing that pip will
not install numpy when it is installed via conda.
Correct, pip will not (or at least should not, and did not in my tests)
install numpy over top of an existing conda installed numpy. Unfortunately
from my testing, conda will install a conda version of numpy over top of a
pip installed version. This may be the expected behavior as conda maintains
its own list of installed packages.
Post by Matthew Brett
So the potential difference is that, pre-wheel, if numpy was not
installed in your conda environment, then pip would build numpy from
source, whereas now you'll get a binary install.
I _think_ that Python's binary API specification
(pip.pep425tags.get_abi_tag()) should prevent pip from installing an
incompatible wheel. Are there any conda experts out there who can
give more detail, or more convincing assurance?
I tested "pip install numpy" in conda environments (conda's equivalent to
virtualenvs) which did not have numpy installed previously for Python 2.7,
3.4 and 3.5 in a Ubuntu 14.04 Docker container. In all cases numpy was
installed from the whl file and appeared to be functional. Running the
numpy test suite found three failing tests for Python 2.7 and 3.5 and 21
errors in Python 3.4. The 2.7 and 3.5 failures do not look concerning but
the 3.4 errors are a bit strange.
Logs are in
https://gist.github.com/jjhelmus/a433a66d56fb0e39b8ebde248ad3fe36
docker run -ti --rm ubuntu:14.04 /bin/bash
apt-get update && apt-get install -y curl
curl -LO https://bootstrap.pypa.io/get-pip.py
python3 get-pip.py
pip install numpy nose
python3 -c "import numpy; numpy.test()"
FAILED (KNOWNFAIL=7, SKIP=17, errors=21)
This is stock Python 3.4 - so not a conda issue. It is definitely a
problem with the wheel because a compiled numpy wheel on the same
apt-get update && apt-get install -y curl python3-dev
curl -LO https://bootstrap.pypa.io/get-pip.py
python3 get-pip.py
pip install --no-binary=:all: numpy nose
python3 -c "import numpy; numpy.test()"
gives no test errors.
It looks like we have some more work to do...
apt-get install gcc
I think these must be bugs in the numpy tests where numpy is assuming
a functional compiler.
Does the conda numpy give test errors when there is no compiler?
Cheers,
Matthew
Yes, both the wheel and conda numpy packages give errors when there is
not a compiler. These errors clear when gcc is installed. Looks like
the wheels are fine, just forgot about a compiler.

Cheers,

- Jonathan Helmus
Matthew Brett
2016-04-14 21:32:08 UTC
Permalink
Post by Matthew Brett
Post by Matthew Brett
Post by Jonathan Helmus
Post by Matthew Brett
Hi,
Post by Benjamin Root
Are we going to have to have documentation somewhere making it clear that
the numpy wheel shouldn't be used in a conda environment? Not that I would
expect this issue to come up all that often, but I could imagine a scenario
where a non-scientist is simply using a base conda distribution because
that
is what IT put on their system. Then they do "pip install ipython" that
indirectly brings in numpy (through the matplotlib dependency), and
end
up
with an incompatible numpy because they would have been linked against
different pythons?
Or is this not an issue?
I'm afraid I don't know conda at all, but I'm guessing that pip will
not install numpy when it is installed via conda.
Correct, pip will not (or at least should not, and did not in my tests)
install numpy over top of an existing conda installed numpy. Unfortunately
from my testing, conda will install a conda version of numpy over top of a
pip installed version. This may be the expected behavior as conda maintains
its own list of installed packages.
Post by Matthew Brett
So the potential difference is that, pre-wheel, if numpy was not
installed in your conda environment, then pip would build numpy from
source, whereas now you'll get a binary install.
I _think_ that Python's binary API specification
(pip.pep425tags.get_abi_tag()) should prevent pip from installing an
incompatible wheel. Are there any conda experts out there who can
give more detail, or more convincing assurance?
I tested "pip install numpy" in conda environments (conda's equivalent to
virtualenvs) which did not have numpy installed previously for Python 2.7,
3.4 and 3.5 in a Ubuntu 14.04 Docker container. In all cases numpy was
installed from the whl file and appeared to be functional. Running the
numpy test suite found three failing tests for Python 2.7 and 3.5 and 21
errors in Python 3.4. The 2.7 and 3.5 failures do not look concerning but
the 3.4 errors are a bit strange.
Logs are in
https://gist.github.com/jjhelmus/a433a66d56fb0e39b8ebde248ad3fe36
docker run -ti --rm ubuntu:14.04 /bin/bash
apt-get update && apt-get install -y curl
curl -LO https://bootstrap.pypa.io/get-pip.py
python3 get-pip.py
pip install numpy nose
python3 -c "import numpy; numpy.test()"
FAILED (KNOWNFAIL=7, SKIP=17, errors=21)
This is stock Python 3.4 - so not a conda issue. It is definitely a
problem with the wheel because a compiled numpy wheel on the same
apt-get update && apt-get install -y curl python3-dev
curl -LO https://bootstrap.pypa.io/get-pip.py
python3 get-pip.py
pip install --no-binary=:all: numpy nose
python3 -c "import numpy; numpy.test()"
gives no test errors.
It looks like we have some more work to do...
apt-get install gcc
I think these must be bugs in the numpy tests where numpy is assuming
a functional compiler.
Does the conda numpy give test errors when there is no compiler?
Cheers,
Matthew
Yes, both the wheel and conda numpy packages give errors when there is not a
compiler. These errors clear when gcc is installed. Looks like the wheels
are fine, just forgot about a compiler.
Thanks for checking. I think the problem is fixed here:

https://github.com/numpy/numpy/pull/7549

Cheers,

Matthew
Nathaniel Smith
2016-04-14 19:07:54 UTC
Permalink
Post by Benjamin Root
Are we going to have to have documentation somewhere making it clear that
the numpy wheel shouldn't be used in a conda environment? Not that I would
expect this issue to come up all that often, but I could imagine a scenario
where a non-scientist is simply using a base conda distribution because
that is what IT put on their system. Then they do "pip install ipython"
that indirectly brings in numpy (through the matplotlib dependency), and
end up with an incompatible numpy because they would have been linked
against different pythons?
Post by Benjamin Root
Or is this not an issue?
There are always issues when you have two different package managers
maintaining separate and out-of-sync metadata about what they think is
installed, but that's true for any mixed use of conda and pip.

But:
- pip won't install a numpy that is incompatible with your python, unless
Anaconda is actively breaking cpython's standard abi (they aren't) or
there's a bug in pip (possible, but no reports yet).
- conda packages for python packages like numpy do generally include the
.egg-info / .dist-info directories that pip uses to store its installation
metadata, so pip can "see" packages installed by conda (but not
vice-versa). So "pip install matplotlib" won't drag in a pypi numpy if
there's already a conda numpy installed.

AFAIK the one case that's nasty is if you first install a conda X, and then
install a pypi X, and then try to use conda to (explicitly, or implicitly
via dependencies) upgrade X. And maybe this is particularly nasty for
X=numpy just because numpy is so low in the stack, but it's not really
numpy specific. (NB I'm not an expert on the internals of conda though :-).)

Actually, from the numpy developer point of view, one of the major
advantages of having wheels is that we can ask people to test prereleases
with 'pip install -U --pre numpy'. If you're a conda user you should only
do this in a temporary environment (like any use of pip really), but I
definitely hope that some conda users will do exactly that to test things
:-).

Also note that there's nothing Linux specific about this scenario. We've
been shipping osx wheels for ages, and AFAIK it hasn't caused any disaster.

-n
Paul Hobson
2016-04-14 19:59:39 UTC
Permalink
Post by Benjamin Root
Are we going to have to have documentation somewhere making it clear
that the numpy wheel shouldn't be used in a conda environment? Not that I
would expect this issue to come up all that often, but I could imagine a
scenario where a non-scientist is simply using a base conda distribution
because that is what IT put on their system. Then they do "pip install
ipython" that indirectly brings in numpy (through the matplotlib
dependency), and end up with an incompatible numpy because they would have
been linked against different pythons?
Post by Benjamin Root
Or is this not an issue?
There are always issues when you have two different package managers
maintaining separate and out-of-sync metadata about what they think is
installed, but that's true for any mixed use of conda and pip.
- pip won't install a numpy that is incompatible with your python, unless
Anaconda is actively breaking cpython's standard abi (they aren't) or
there's a bug in pip (possible, but no reports yet).
- conda packages for python packages like numpy do generally include the
.egg-info / .dist-info directories that pip uses to store its installation
metadata, so pip can "see" packages installed by conda (but not
vice-versa). So "pip install matplotlib" won't drag in a pypi numpy if
there's already a conda numpy installed.
Minor clarification:. I believe conda can see pip-installed packages.

If I execute "conda list" in an environment, I can see packaged installed
by both pip, conda, and locally (i.e., "pip install . -e").

-paul
Benjamin Root
2016-04-14 20:04:52 UTC
Permalink
I am honestly surprised that these worked (I haven't gotten around to
testing for myself). I could have sworn there was a difference in how
Continuum compiled python such that any binaries built against a stock
python would not work in a conda environment. I ran into issues a couple
years ago where a modwsgi package provided through yum wouldn't work with
miniconda because of link-time differences.

I cannot for the life of me remember the error message, though.

Ben Root
Post by Paul Hobson
Post by Benjamin Root
Are we going to have to have documentation somewhere making it clear
that the numpy wheel shouldn't be used in a conda environment? Not that I
would expect this issue to come up all that often, but I could imagine a
scenario where a non-scientist is simply using a base conda distribution
because that is what IT put on their system. Then they do "pip install
ipython" that indirectly brings in numpy (through the matplotlib
dependency), and end up with an incompatible numpy because they would have
been linked against different pythons?
Post by Benjamin Root
Or is this not an issue?
There are always issues when you have two different package managers
maintaining separate and out-of-sync metadata about what they think is
installed, but that's true for any mixed use of conda and pip.
- pip won't install a numpy that is incompatible with your python, unless
Anaconda is actively breaking cpython's standard abi (they aren't) or
there's a bug in pip (possible, but no reports yet).
- conda packages for python packages like numpy do generally include the
.egg-info / .dist-info directories that pip uses to store its installation
metadata, so pip can "see" packages installed by conda (but not
vice-versa). So "pip install matplotlib" won't drag in a pypi numpy if
there's already a conda numpy installed.
Minor clarification:. I believe conda can see pip-installed packages.
If I execute "conda list" in an environment, I can see packaged installed
by both pip, conda, and locally (i.e., "pip install . -e").
-paul
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Matthew Brett
2016-04-17 03:02:18 UTC
Permalink
Hi,
Post by Matthew Brett
Hi,
Post by Jens Nielsen
I have tried testing the wheels in a project that runs tests on Travis's
Trusty infrastructure which. The wheels work great for python 3.5 and saves
us several minuts of runtime.
However, I am having trouble using the wheels on python 2.7 on the same
Trusty machines. It seems to be because the wheels are tagged as cp27-cp27mu
(numpy-1.11.0-cp27-cp27mu-manylinux1_x86_64.whl) where as
pip.pep425tags.get_abi_tag() returns cp27m on this particular python
version. (Stock python 2.7 installed on Travis 14.04 VMs) Any chance of a
cp27m compatible wheel build?
Ouch - do you know where travis-ci's Python 2.7 comes from? I see
that the standard apt-get install -y python is a wide (mu) build...
I built some narrow unicode builds
(numpy-1.11.0-cp27-cp27m-manylinux1_x86_64.whl etc) here:

http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com/

Would you mind testing them to see if they work on travis-ci?

Thanks,

Matthew
Matthew Brett
2016-04-17 03:23:47 UTC
Permalink
Post by Matthew Brett
Hi,
Post by Matthew Brett
Hi,
Post by Jens Nielsen
I have tried testing the wheels in a project that runs tests on Travis's
Trusty infrastructure which. The wheels work great for python 3.5 and saves
us several minuts of runtime.
However, I am having trouble using the wheels on python 2.7 on the same
Trusty machines. It seems to be because the wheels are tagged as cp27-cp27mu
(numpy-1.11.0-cp27-cp27mu-manylinux1_x86_64.whl) where as
pip.pep425tags.get_abi_tag() returns cp27m on this particular python
version. (Stock python 2.7 installed on Travis 14.04 VMs) Any chance of a
cp27m compatible wheel build?
Ouch - do you know where travis-ci's Python 2.7 comes from? I see
that the standard apt-get install -y python is a wide (mu) build...
I built some narrow unicode builds
http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com/
Would you mind testing them to see if they work on travis-ci?
I tried testing on trusty with travis-ci, but it appears to pick up
the mu builds as on precise...

https://travis-ci.org/matthew-brett/manylinux-testing/jobs/123652670#L161

Cheers,

Matthew
Olivier Grisel
2016-04-17 10:05:10 UTC
Permalink
I tried on trusty and is also picked
numpy-1.11.0-cp27-cp27mu-manylinux1_x86_64.whl using the system python
import pip
pip.pep425tags.get_abi_tag()
'cp27mu'

Outside of the virtualenv I still have the pip version from ubuntu
trusty and it does cannot detect ABI tags:

$ /usr/bin/pip --version
pip 1.5.4 from /usr/lib/python2.7/dist-packages (python 2.7)
import pip
pip.pep425tags.get_abi_tag()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'get_abi_tag'

But we don't really care because manylinux1 wheels can only be
installed by pip 8.1 and later. Previous versions of pip should just
ignore those wheels and try to install from the source tarball
instead.
--
Olivier
Jens Nielsen
2016-04-17 16:48:48 UTC
Permalink
I have tested the new cp27m wheels and they seem to work great.

@Matthew I am using the:

```
sudo: required
dist: trusty

images mentioned here https://docs.travis-ci.com/user/ci-environment/. As
far as I can see you are doing:
sudo: false
dist: trusty

I had no idea such an image exist since it's not documented on
https://docs.travis-ci.com/user/ci-environment/

Anyway your tests runs with python 2.7.9 where as the sudo: requires ships
python 2.7.10 so it's clearly a different python version:

@Olivier Grisel this only applies to Travis's own home build versions of
python 2.7 on the Trusty running on google compute engine.
It ships it's own prebuild python version. I don't have any issues with the
stock versions on Ubuntu which pip tells me are indeed cp27mu.

It seems like the new cp27m wheels works as expected. Thanks a lot
Doing:

```
python -c "from pip import pep425tags;
print(pep425tags.is_manylinux1_compatible());
print(pep425tags.have_compatible_glibc(2, 5));
print(pep425tags.get_abi_tag())"
pip install --timeout=60 --no-index --trusted-host "
ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com"
--find-links "
http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com/"
numpy scipy --upgrade
```
results in:

```
True
True
cp27m
Ignoring indexes: https://pypi.python.org/simple
Collecting numpy
Downloading
http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com/numpy-1.11.0-cp27-cp27m-manylinux1_x86_64.whl
(15.3MB)
100% |████████████████████████████████| 15.3MB 49.0MB/s
Collecting scipy
Downloading
http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com/scipy-0.17.0-cp27-cp27m-manylinux1_x86_64.whl
(39.5MB)
100% |████████████████████████████████| 39.5MB 21.1MB/s
Installing collected packages: numpy, scipy
Found existing installation: numpy 1.10.1
Uninstalling numpy-1.10.1:
Successfully uninstalled numpy-1.10.1
Successfully installed numpy-1.11.0 scipy-0.17.0
```
And all my tests pass as expected.
Thanks a lot for all the work.
Best
Jens
Post by Olivier Grisel
I tried on trusty and is also picked
numpy-1.11.0-cp27-cp27mu-manylinux1_x86_64.whl using the system python
import pip
pip.pep425tags.get_abi_tag()
'cp27mu'
Outside of the virtualenv I still have the pip version from ubuntu
$ /usr/bin/pip --version
pip 1.5.4 from /usr/lib/python2.7/dist-packages (python 2.7)
import pip
pip.pep425tags.get_abi_tag()
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'get_abi_tag'
But we don't really care because manylinux1 wheels can only be
installed by pip 8.1 and later. Previous versions of pip should just
ignore those wheels and try to install from the source tarball
instead.
--
Olivier
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Olivier Grisel
2016-04-17 17:46:36 UTC
Permalink
Thanks for the clarification, I read your original report too quickly.

I wonder why the travis maintainers built Python 2.7 with a
non-standard unicode option.

Edit (after googling): this is a known issue. The image with Python
2.7.11 will be fixed:

https://github.com/travis-ci/travis-ci/issues/5107
--
Olivier
Benjamin Root
2016-04-17 19:03:02 UTC
Permalink
Yeah! That's the bug I encountered! So, that would explain why this seems
to work fine now (I tried it out a bit on Friday on a CentOS6 system, but
didn't run the test suite).

Cheers!
Ben Root
Post by Olivier Grisel
Thanks for the clarification, I read your original report too quickly.
I wonder why the travis maintainers built Python 2.7 with a
non-standard unicode option.
Edit (after googling): this is a known issue. The image with Python
https://github.com/travis-ci/travis-ci/issues/5107
--
Olivier
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Nathaniel Smith
2016-04-17 19:25:34 UTC
Permalink
Post by Olivier Grisel
Thanks for the clarification, I read your original report too quickly.
I wonder why the travis maintainers built Python 2.7 with a
non-standard unicode option.
Because for some reason cpython's configure script (in the now somewhat
ancient versions we're talking about) defaults to non-standard broken
Unicode support, and you have to explicitly override it if you want working
standard Unicode support.

I guess this made sense in like the 90s before people realized how unicode
was going to go down.

Same issue affects pyenv users (or used to, I think they might have just
fixed it [0]) and Enthought Canopy.

-n

[0] https://github.com/yyuu/pyenv/issues/257
Matthew Brett
2016-04-18 21:49:50 UTC
Permalink
Post by Jens Nielsen
I have tested the new cp27m wheels and they seem to work great.
```
sudo: required
dist: trusty
images mentioned here https://docs.travis-ci.com/user/ci-environment/. As
sudo: false
dist: trusty
I had no idea such an image exist since it's not documented on
https://docs.travis-ci.com/user/ci-environment/
Anyway your tests runs with python 2.7.9 where as the sudo: requires ships
@Olivier Grisel this only applies to Travis's own home build versions of
python 2.7 on the Trusty running on google compute engine.
It ships it's own prebuild python version. I don't have any issues with the
stock versions on Ubuntu which pip tells me are indeed cp27mu.
It seems like the new cp27m wheels works as expected. Thanks a lot
```
python -c "from pip import pep425tags;
print(pep425tags.is_manylinux1_compatible());
print(pep425tags.have_compatible_glibc(2, 5));
print(pep425tags.get_abi_tag())"
pip install --timeout=60 --no-index --trusted-host
"ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com"
--find-links
"http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com/"
numpy scipy --upgrade
```
```
True
True
cp27m
Ignoring indexes: https://pypi.python.org/simple
Collecting numpy
Downloading
http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com/numpy-1.11.0-cp27-cp27m-manylinux1_x86_64.whl
(15.3MB)
100% |████████████████████████████████| 15.3MB 49.0MB/s
Collecting scipy
Downloading
http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com/scipy-0.17.0-cp27-cp27m-manylinux1_x86_64.whl
(39.5MB)
100% |████████████████████████████████| 39.5MB 21.1MB/s
Installing collected packages: numpy, scipy
Found existing installation: numpy 1.10.1
Successfully uninstalled numpy-1.10.1
Successfully installed numpy-1.11.0 scipy-0.17.0
```
And all my tests pass as expected.
Thanks for testing.

I set up a buildbot test to run against a narrow unicode build of Python:

http://nipy.bic.berkeley.edu/builders/manylinux-2.7-debian-narrow/builds/1

All tests pass for me too, so I've done the pypi upload for the narrow
unicode numpy, scipy, cython wheels.

Cheers,

Matthew
Matthew Brett
2016-04-19 07:17:40 UTC
Permalink
Hi,
Post by Matthew Brett
Post by Jens Nielsen
I have tested the new cp27m wheels and they seem to work great.
```
sudo: required
dist: trusty
images mentioned here https://docs.travis-ci.com/user/ci-environment/. As
sudo: false
dist: trusty
I had no idea such an image exist since it's not documented on
https://docs.travis-ci.com/user/ci-environment/
Anyway your tests runs with python 2.7.9 where as the sudo: requires ships
@Olivier Grisel this only applies to Travis's own home build versions of
python 2.7 on the Trusty running on google compute engine.
It ships it's own prebuild python version. I don't have any issues with the
stock versions on Ubuntu which pip tells me are indeed cp27mu.
It seems like the new cp27m wheels works as expected. Thanks a lot
```
python -c "from pip import pep425tags;
print(pep425tags.is_manylinux1_compatible());
print(pep425tags.have_compatible_glibc(2, 5));
print(pep425tags.get_abi_tag())"
pip install --timeout=60 --no-index --trusted-host
"ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com"
--find-links
"http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com/"
numpy scipy --upgrade
```
```
True
True
cp27m
Ignoring indexes: https://pypi.python.org/simple
Collecting numpy
Downloading
http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com/numpy-1.11.0-cp27-cp27m-manylinux1_x86_64.whl
(15.3MB)
100% |████████████████████████████████| 15.3MB 49.0MB/s
Collecting scipy
Downloading
http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com/scipy-0.17.0-cp27-cp27m-manylinux1_x86_64.whl
(39.5MB)
100% |████████████████████████████████| 39.5MB 21.1MB/s
Installing collected packages: numpy, scipy
Found existing installation: numpy 1.10.1
Successfully uninstalled numpy-1.10.1
Successfully installed numpy-1.11.0 scipy-0.17.0
```
And all my tests pass as expected.
Thanks for testing.
I've also tested a range of numpy and scipy wheels built with the
manylinux docker image.

Built numpy and scipy wheels here:

http://nipy.bic.berkeley.edu/manylinux/

Test script and output here:

http://nipy.bic.berkeley.edu/manylinux/tests/

There are some test failures in the logs there, but I think they are
all known failures from old numpy / scipy versions, particularly

https://github.com/scipy/scipy/issues/5370

Y'all can test for yourselves with something like:

python -m pip install -U pip
pip install -f https://nipy.bic.berkeley.edu/manylinux numpy==1.6.2
scipy==0.16.0

I propose to upload these historical wheels to pypi to make it easier
to test against older versions of numpy / scipy.

Any objections?

Cheers,

Matthew
Olivier Grisel
2016-04-19 08:12:54 UTC
Permalink
I think that would be very useful, e.g. for downstream projects to
check that they work properly with old versions using a simple pip
install command on their CI workers.
--
Olivier
Matthew Brett
2016-04-20 05:08:54 UTC
Permalink
On Tue, Apr 19, 2016 at 1:12 AM, Olivier Grisel
Post by Olivier Grisel
I think that would be very useful, e.g. for downstream projects to
check that they work properly with old versions using a simple pip
install command on their CI workers.
Done for numpy 1.6.0 through 1.10.4, scipy 0.9 through scipy 0.16.1

Please let me know of any problems,

Matthew
Olivier Grisel
2016-04-20 08:59:31 UTC
Permalink
Thanks,

I think next we could upgrade the travis configuration of numpy and
scipy to build and upload manylinux1 wheels to
http://travis-dev-wheels.scipy.org/ for downstream project to test
against the master branch of numpy and scipy whithout having to build
those from source.

However that would require publishing an official pre-built
libopenblas.so (+headers) archive or RPM package. That archive would
server as the reference libary to build scipy stack manylinux1 wheels.
--
Olivier
Jens Nielsen
2016-04-20 10:33:56 UTC
Permalink
Thanks

I can confirm that the new narrow unicode build wheels of Scipy works as
expected for my project.
@Oliver Grisel Thanks for finding the Travis issue it's probably worth
considering switching the Travis build to 2.7.11 to avoid other similar
issues.

The old versions of numpy are very handy for downstream testing. I have
verified that they work as expected in the Matplotlib tests here:
https://github.com/jenshnielsen/matplotlib/tree/travisnowheelhouse where we
are testing against numpy 1.6 as the earliest. This branch switches
matplotlib from the scikit image wheelhouse to manylinux wheels which seems
to work great.

best
Jens
Post by Peter Cock
Thanks,
I think next we could upgrade the travis configuration of numpy and
scipy to build and upload manylinux1 wheels to
http://travis-dev-wheels.scipy.org/ for downstream project to test
against the master branch of numpy and scipy whithout having to build
those from source.
However that would require publishing an official pre-built
libopenblas.so (+headers) archive or RPM package. That archive would
server as the reference libary to build scipy stack manylinux1 wheels.
--
Olivier
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Matthew Brett
2016-04-20 18:41:49 UTC
Permalink
Hi,
Post by Peter Cock
Thanks
I can confirm that the new narrow unicode build wheels of Scipy works as
expected for my project.
@Oliver Grisel Thanks for finding the Travis issue it's probably worth
considering switching the Travis build to 2.7.11 to avoid other similar
issues.
The old versions of numpy are very handy for downstream testing. I have
https://github.com/jenshnielsen/matplotlib/tree/travisnowheelhouse where we
are testing against numpy 1.6 as the earliest. This branch switches
matplotlib from the scikit image wheelhouse to manylinux wheels which seems
to work great.
Jens - any interest in working together on a good matplotlib build recipe?

Matthew
Matthew Brett
2016-04-20 14:57:50 UTC
Permalink
On Wed, Apr 20, 2016 at 1:59 AM, Olivier Grisel
Post by Peter Cock
Thanks,
I think next we could upgrade the travis configuration of numpy and
scipy to build and upload manylinux1 wheels to
http://travis-dev-wheels.scipy.org/ for downstream project to test
against the master branch of numpy and scipy whithout having to build
those from source.
However that would require publishing an official pre-built
libopenblas.so (+headers) archive or RPM package. That archive would
server as the reference libary to build scipy stack manylinux1 wheels.
There's an OpenBLAS archive up at :
http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com/openblas_0.2.18.tgz
- is that the right place for it? It gets uploaded by the
manylinux-builds travis run.

Cheers,

Matthew
Olivier Grisel
2016-04-21 08:47:50 UTC
Permalink
Post by Matthew Brett
On Wed, Apr 20, 2016 at 1:59 AM, Olivier Grisel
Post by Peter Cock
Thanks,
I think next we could upgrade the travis configuration of numpy and
scipy to build and upload manylinux1 wheels to
http://travis-dev-wheels.scipy.org/ for downstream project to test
against the master branch of numpy and scipy whithout having to build
those from source.
However that would require publishing an official pre-built
libopenblas.so (+headers) archive or RPM package. That archive would
server as the reference libary to build scipy stack manylinux1 wheels.
http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com/openblas_0.2.18.tgz
Thanks.
Post by Matthew Brett
- is that the right place for it? It gets uploaded by the
manylinux-builds travis run.
The only problem with rackspace cloud files is that as of now there is
no way to put a short domain name (CNAME) with https. Maybe we could
use the github "release" system on a github repo under the numpy
github organization to host it. Or alternatively use an external
binary file host that use github credentials for upload rigths, for
instance bintray (I have no experience with this yet though).
--
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel
Matthew Brett
2016-04-22 18:17:11 UTC
Permalink
On Thu, Apr 21, 2016 at 1:47 AM, Olivier Grisel
Post by Olivier Grisel
Post by Matthew Brett
On Wed, Apr 20, 2016 at 1:59 AM, Olivier Grisel
Post by Peter Cock
Thanks,
I think next we could upgrade the travis configuration of numpy and
scipy to build and upload manylinux1 wheels to
http://travis-dev-wheels.scipy.org/ for downstream project to test
against the master branch of numpy and scipy whithout having to build
those from source.
However that would require publishing an official pre-built
libopenblas.so (+headers) archive or RPM package. That archive would
server as the reference libary to build scipy stack manylinux1 wheels.
http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com/openblas_0.2.18.tgz
Thanks.
Post by Matthew Brett
- is that the right place for it? It gets uploaded by the
manylinux-builds travis run.
The only problem with rackspace cloud files is that as of now there is
no way to put a short domain name (CNAME) with https. Maybe we could
use the github "release" system on a github repo under the numpy
github organization to host it. Or alternatively use an external
binary file host that use github credentials for upload rigths, for
instance bintray (I have no experience with this yet though).
The github releases idea sounds intriguing. Do you have any
experience with that? Are there good examples other than the API
documentation?

https://developer.github.com/v3/repos/releases/

Cheers,

Matthew
Olivier Grisel
2016-04-22 18:27:58 UTC
Permalink
Post by Matthew Brett
The github releases idea sounds intriguing. Do you have any
experience with that? Are there good examples other than the API
documentation?
https://developer.github.com/v3/repos/releases/
I never used it by I assume we could create a numpy-openblas repo to
host official builds suitable for embedding numpy wheels for stable
each releases of OpenBLAS:

There is also a travis deployment target.

https://docs.travis-ci.com/user/deployment/releases

I have not sure that the travis timeout is long enough to build
openblas. I believe so but I have not tried myself yet.
--
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel
Matthew Brett
2016-04-22 18:35:24 UTC
Permalink
On Fri, Apr 22, 2016 at 11:27 AM, Olivier Grisel
Post by Olivier Grisel
Post by Matthew Brett
The github releases idea sounds intriguing. Do you have any
experience with that? Are there good examples other than the API
documentation?
https://developer.github.com/v3/repos/releases/
I never used it by I assume we could create a numpy-openblas repo to
host official builds suitable for embedding numpy wheels for stable
There is also a travis deployment target.
https://docs.travis-ci.com/user/deployment/releases
Ah - thanks - that's good resource.
Post by Olivier Grisel
I have not sure that the travis timeout is long enough to build
openblas. I believe so but I have not tried myself yet.
Yes, the manylinux-builds repo currently builds openblas for each
entry in the build matrix, so it's easily within time:

https://travis-ci.org/matthew-brett/manylinux-builds/builds/123643313

It would be good to think of a way of supporting a set of libraries,
such as libpng, freetype, openblas. We might need to support both
64-bit and 32-bit versions as well. Then, some automated build script
would by default pick up the latest of these for numpy, matplotlib
etc.

Matthew

Matthew Brett
2016-04-14 20:22:01 UTC
Permalink
Post by Jens Nielsen
I have tried testing the wheels in a project that runs tests on Travis's
Trusty infrastructure which. The wheels work great for python 3.5 and saves
us several minuts of runtime.
However, I am having trouble using the wheels on python 2.7 on the same
Trusty machines. It seems to be because the wheels are tagged as cp27-cp27mu
(numpy-1.11.0-cp27-cp27mu-manylinux1_x86_64.whl) where as
pip.pep425tags.get_abi_tag() returns cp27m on this particular python
version. (Stock python 2.7 installed on Travis 14.04 VMs) Any chance of a
cp27m compatible wheel build?
Nathaniel / other pip experts - I can't remember the history of these tags.

Is there any danger that an older pip will install a cp27m wheel on a
cp27mu system?

Matthew
Nathaniel Smith
2016-04-15 01:00:12 UTC
Permalink
Post by Matthew Brett
Post by Jens Nielsen
I have tried testing the wheels in a project that runs tests on Travis's
Trusty infrastructure which. The wheels work great for python 3.5 and saves
us several minuts of runtime.
However, I am having trouble using the wheels on python 2.7 on the same
Trusty machines. It seems to be because the wheels are tagged as cp27-cp27mu
(numpy-1.11.0-cp27-cp27mu-manylinux1_x86_64.whl) where as
pip.pep425tags.get_abi_tag() returns cp27m on this particular python
version. (Stock python 2.7 installed on Travis 14.04 VMs) Any chance of a
cp27m compatible wheel build?
Nathaniel / other pip experts - I can't remember the history of these tags.
Is there any danger that an older pip will install a cp27m wheel on a
cp27mu system?
No, support for cp27m/cp27mu tags went in before support for manylinux
tags. And in any case, a pip that doesn't know about cp27m/cp27mu will
just not install such wheels.

The dangerous case is if you were to use an old version of bdist_wheel
that generated a wheel with the "none" abi tag instead of a
cp27m/cp27mu abi tag -- this will mess up all versions of pip, new and
old. But the manylinux docker image definitely has a new enough
version of the wheel package that this is not a problem. ...I guess
the other dangerous case is if you generate a wheel that simply has
the wrong name -- this happened to the gevent packager due to some
distutils brokenness involving using the same source directory to
build both wheels. So don't do that :-). (IIRC there's an open bug
against auditwheel to check for all these problems -- belt *and*
suspenders -- but that hasn't been implemented yet.)

-n
--
Nathaniel J. Smith -- https://vorpus.org
Charles R Harris
2016-04-13 21:52:29 UTC
Permalink
Post by Matthew Brett
Hi,
Post by Matthew Brett
On Fri, Mar 25, 2016 at 3:02 AM, Robert T. McGibbon <
Post by Robert T. McGibbon
I suspect that many of the maintainers of major scipy-ecosystem
projects are
Post by Matthew Brett
Post by Matthew Brett
Post by Robert T. McGibbon
aware of these (or other similar) travis wheel caches, but would
guess that
Post by Matthew Brett
Post by Matthew Brett
Post by Robert T. McGibbon
the pool of travis-ci python users who weren't aware of these wheel
caches
Post by Matthew Brett
Post by Matthew Brett
Post by Robert T. McGibbon
is much much larger. So there will still be a lot of travis-ci clock
cycles
Post by Matthew Brett
Post by Matthew Brett
Post by Robert T. McGibbon
saved by manylinux wheels.
-Robert
Yes exactly. Availability of NumPy Linux wheels on PyPI is definitely
something
Post by Matthew Brett
Post by Matthew Brett
I would suggest adding to the release notes. Hopefully this will help
trigger
Post by Matthew Brett
Post by Matthew Brett
a general availability of wheels in the numpy-ecosystem :)
In the case of Travis CI, their VM images for Python already have a
version
Post by Matthew Brett
Post by Matthew Brett
of NumPy installed, but having the latest version of NumPy and SciPy
etc
Post by Matthew Brett
Post by Matthew Brett
available as Linux wheels would be very nice.
We're very nearly there now.
The latest versions of numpy, scipy, scikit-image, pandas, numexpr,
statsmodels wheels for testing at
http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com/
Post by Matthew Brett
Post by Matthew Brett
python -m pip install --upgrade pip
pip install --trusted-host=
ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com
Post by Matthew Brett
Post by Matthew Brett
--find-links=
http://ccdd0ebb5a931e58c7c5-aae005c4999d7244ac63632f8b80e089.r77.cf2.rackcdn.com
Post by Matthew Brett
Post by Matthew Brett
numpy scipy scikit-learn numexpr
python -c 'import numpy; numpy.test("full")'
python -c 'import scipy; scipy.test("full")'
We would love to get any feedback as to whether these work on your
machines.
Post by Matthew Brett
I've just rebuilt these wheels with the just-released OpenBLAS 0.2.18.
OpenBLAS is now passing all its own tests and tests on numpy / scipy /
scikit-learn at http://build.openblas.net/builders
http://nipy.bic.berkeley.edu/builders/manylinux-2.7-debian
http://nipy.bic.berkeley.edu/builders/manylinux-2.7-debian
https://travis-ci.org/matthew-brett/manylinux-testing
So I think these are ready to go. I propose uploading these wheels
for numpy and scipy to pypi tomorrow unless anyone has an objection.
Done. If y'all are on linux, and you have pip >= 8.11, you should
$ pip install numpy scipy
Collecting numpy
Downloading numpy-1.11.0-cp27-cp27mu-manylinux1_x86_64.whl (15.3MB)
100% |████████████████████████████████| 15.3MB 61kB/s
Collecting scipy
Downloading scipy-0.17.0-cp27-cp27mu-manylinux1_x86_64.whl (39.5MB)
100% |████████████████████████████████| 39.5MB 24kB/s
Installing collected packages: numpy, scipy
Successfully installed numpy-1.11.0 scipy-0.17.0
Great work. It is nice that we are finally getting the Windows thing
squared away after all these years.

Chuck
Continue reading on narkive:
Loading...