Discussion:
[Numpy-discussion] Clarification sought on Scipy Numpy version requirements.
Charles R Harris
2015-06-19 20:08:10 UTC
Permalink
Hi All,

I'm looking to change some numpy deprecations into errors as well as remove
some deprecated functions. The problem I see is that
SciPy claims to support Numpy >= 1.5 and Numpy 1.5 is really, really, old.
So the question is, does "support" mean compiles with earlier versions
of Numpy ? If that is the case there is very little that can be done about
deprecation. OTOH, if it means Scipy can be compiled with more recent numpy
versions but used with earlier Numpy versions (which is a good feat), I'd
like to know. I'd also like to know what the interface requirements are, as
I'd like to remove old_defines.h

Chuck
Sturla Molden
2015-06-19 21:05:24 UTC
Permalink
Post by Charles R Harris
I'm looking to change some numpy deprecations into errors as well as remove
some deprecated functions. The problem I see is that
SciPy claims to support Numpy >= 1.5 and Numpy 1.5 is really, really, old.
So the question is, does "support" mean compiles with earlier versions
of Numpy ?
It means there is a Travis CI build with NumPy 1.6.2. So any change to the
SciPy source code must compile with NumPy 1.6 and any later version of
NumPy.

There is no Travis CI build with NumPy 1.5. I don't think we know for sure
if it is really compatible with the current SciPy.

Sturla
Charles R Harris
2015-06-19 21:52:06 UTC
Permalink
Post by Charles R Harris
Post by Charles R Harris
I'm looking to change some numpy deprecations into errors as well as
remove
Post by Charles R Harris
some deprecated functions. The problem I see is that
SciPy claims to support Numpy >= 1.5 and Numpy 1.5 is really, really,
old.
Post by Charles R Harris
So the question is, does "support" mean compiles with earlier versions
of Numpy ?
It means there is a Travis CI build with NumPy 1.6.2. So any change to the
SciPy source code must compile with NumPy 1.6 and any later version of
NumPy.
There is no Travis CI build with NumPy 1.5. I don't think we know for sure
if it is really compatible with the current SciPy.
I guess this also raises the question of what versions of Scipy Numpy needs
to support. I'm thinking of removing the noprefix.h, but it doesn't cost to
leave it in as it must be explicitly included by anyone who needs it. Hmm,
maybe best to leave it be, although I suspect anyone using it could just as
well use an earlier version of Numpy.

Chuck
Ralf Gommers
2015-06-21 13:14:22 UTC
Permalink
On Fri, Jun 19, 2015 at 11:52 PM, Charles R Harris <
Post by Charles R Harris
Post by Charles R Harris
Post by Charles R Harris
I'm looking to change some numpy deprecations into errors as well as
remove
Post by Charles R Harris
some deprecated functions. The problem I see is that
SciPy claims to support Numpy >= 1.5 and Numpy 1.5 is really, really,
old.
Post by Charles R Harris
So the question is, does "support" mean compiles with earlier versions
of Numpy ?
It means there is a Travis CI build with NumPy 1.6.2. So any change to the
SciPy source code must compile with NumPy 1.6 and any later version of
NumPy.
There is no Travis CI build with NumPy 1.5. I don't think we know for sure
if it is really compatible with the current SciPy.
I guess this also raises the question of what versions of Scipy Numpy
needs to support.
I'd treat Scipy like any other popular package that depends on Numpy. If a
change in Numpy would break a Scipy version released in say the last 1.5
years, then that's a problem. If it's a quite old Scipy, then it may be OK.

I'm thinking of removing the noprefix.h, but it doesn't cost to leave it in
Post by Charles R Harris
as it must be explicitly included by anyone who needs it. Hmm, maybe best
to leave it be, although I suspect anyone using it could just as well use
an earlier version of Numpy.
Does noprefix.h even give you a deprecation warning now? Doesn't look like
it to me, which means it should be left alone for quite a while.

Ralf
Post by Charles R Harris
Chuck
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Charles R Harris
2015-06-21 15:13:11 UTC
Permalink
Post by Ralf Gommers
On Fri, Jun 19, 2015 at 11:52 PM, Charles R Harris <
Post by Charles R Harris
Post by Charles R Harris
Post by Charles R Harris
I'm looking to change some numpy deprecations into errors as well as
remove
Post by Charles R Harris
some deprecated functions. The problem I see is that
SciPy claims to support Numpy >= 1.5 and Numpy 1.5 is really, really,
old.
Post by Charles R Harris
So the question is, does "support" mean compiles with earlier versions
of Numpy ?
It means there is a Travis CI build with NumPy 1.6.2. So any change to the
SciPy source code must compile with NumPy 1.6 and any later version of
NumPy.
There is no Travis CI build with NumPy 1.5. I don't think we know for sure
if it is really compatible with the current SciPy.
There is still a reference to 1.5 in Scipy, I forget where.
Post by Ralf Gommers
Post by Charles R Harris
I guess this also raises the question of what versions of Scipy Numpy
needs to support.
I'd treat Scipy like any other popular package that depends on Numpy. If a
change in Numpy would break a Scipy version released in say the last 1.5
years, then that's a problem. If it's a quite old Scipy, then it may be OK.
So that would be Scipy 0.13, looks like.
Post by Ralf Gommers
I'm thinking of removing the noprefix.h, but it doesn't cost to leave it
Post by Charles R Harris
in as it must be explicitly included by anyone who needs it. Hmm, maybe
best to leave it be, although I suspect anyone using it could just as well
use an earlier version of Numpy.
Does noprefix.h even give you a deprecation warning now? Doesn't look like
it to me, which means it should be left alone for quite a while.
Yeah, it's probably best to just leave it be.

Chuck
Ralf Gommers
2015-06-21 15:31:24 UTC
Permalink
Post by Charles R Harris
Post by Ralf Gommers
On Fri, Jun 19, 2015 at 11:52 PM, Charles R Harris <
Post by Charles R Harris
Post by Charles R Harris
I'm looking to change some numpy deprecations into errors as well as
remove
Post by Charles R Harris
some deprecated functions. The problem I see is that
SciPy claims to support Numpy >= 1.5 and Numpy 1.5 is really, really,
old.
Post by Charles R Harris
So the question is, does "support" mean compiles with earlier versions
of Numpy ?
It means there is a Travis CI build with NumPy 1.6.2. So any change to the
SciPy source code must compile with NumPy 1.6 and any later version of
NumPy.
There is no Travis CI build with NumPy 1.5. I don't think we know for sure
if it is really compatible with the current SciPy.
There is still a reference to 1.5 in Scipy, I forget where.
In INSTALL.rst.txt, will fix that now.

Ralf
Charles R Harris
2015-06-21 15:45:25 UTC
Permalink
On Sun, Jun 21, 2015 at 5:13 PM, Charles R Harris <
Post by Charles R Harris
Post by Ralf Gommers
On Fri, Jun 19, 2015 at 11:52 PM, Charles R Harris <
Post by Charles R Harris
Post by Charles R Harris
I'm looking to change some numpy deprecations into errors as well as
remove
Post by Charles R Harris
some deprecated functions. The problem I see is that
SciPy claims to support Numpy >= 1.5 and Numpy 1.5 is really,
really, old.
Post by Charles R Harris
So the question is, does "support" mean compiles with earlier
versions
Post by Charles R Harris
of Numpy ?
It means there is a Travis CI build with NumPy 1.6.2. So any change to the
SciPy source code must compile with NumPy 1.6 and any later version of
NumPy.
There is no Travis CI build with NumPy 1.5. I don't think we know for sure
if it is really compatible with the current SciPy.
There is still a reference to 1.5 in Scipy, I forget where.
In INSTALL.rst.txt, will fix that now.
Ralf, I cannot compile Scipy 0.13.3 on my system, it seems to fail here

Error compiling Cython file:
------------------------------------------------------------
...
# and object. In this file, only NULL is passed to these parameters.
cdef extern from *:
cnp.ndarray PyArray_CheckFromAny(object, void*, int, int, int, void*)
cnp.ndarray PyArray_FromArray(cnp.ndarray, void*, int)

from . cimport cython_blas as blas_pointers
^
------------------------------------------------------------

_decomp_update.pyx:60:0: 'cython_blas.pxd' not found


Although it is hard to tell, the traceback doesn't give much useful
information. I suspect this is due to a cython version mismatch, as it
seems to be looking for cython_blas.pxd but only cython_blas.c is
available. Have you seen this before?

Chuck
Ralf Gommers
2015-06-21 15:57:04 UTC
Permalink
Post by Charles R Harris
On Sun, Jun 21, 2015 at 5:13 PM, Charles R Harris <
Post by Charles R Harris
Post by Ralf Gommers
On Fri, Jun 19, 2015 at 11:52 PM, Charles R Harris <
On Fri, Jun 19, 2015 at 3:05 PM, Sturla Molden <
Post by Charles R Harris
Post by Charles R Harris
I'm looking to change some numpy deprecations into errors as well
as remove
Post by Charles R Harris
some deprecated functions. The problem I see is that
SciPy claims to support Numpy >= 1.5 and Numpy 1.5 is really,
really, old.
Post by Charles R Harris
So the question is, does "support" mean compiles with earlier
versions
Post by Charles R Harris
of Numpy ?
It means there is a Travis CI build with NumPy 1.6.2. So any change to the
SciPy source code must compile with NumPy 1.6 and any later version of
NumPy.
There is no Travis CI build with NumPy 1.5. I don't think we know for sure
if it is really compatible with the current SciPy.
There is still a reference to 1.5 in Scipy, I forget where.
In INSTALL.rst.txt, will fix that now.
Ralf, I cannot compile Scipy 0.13.3 on my system, it seems to fail here
------------------------------------------------------------
...
# and object. In this file, only NULL is passed to these parameters.
cnp.ndarray PyArray_CheckFromAny(object, void*, int, int, int, void*)
cnp.ndarray PyArray_FromArray(cnp.ndarray, void*, int)
from . cimport cython_blas as blas_pointers
^
------------------------------------------------------------
_decomp_update.pyx:60:0: 'cython_blas.pxd' not found
Although it is hard to tell, the traceback doesn't give much useful
information. I suspect this is due to a cython version mismatch, as it
seems to be looking for cython_blas.pxd but only cython_blas.c is
available. Have you seen this before?
That's code that was only introduced for 0.16.x; a ``git clean -xdf``
should fix this for you.

Next obstacle: I think it'll fail with Cython 0.22, you'll need a lower
Cython version (probably around 0.19.x).

Ralf
Charles R Harris
2015-06-21 17:49:47 UTC
Permalink
On Sun, Jun 21, 2015 at 5:45 PM, Charles R Harris <
Post by Charles R Harris
On Sun, Jun 21, 2015 at 5:13 PM, Charles R Harris <
Post by Charles R Harris
Post by Ralf Gommers
On Fri, Jun 19, 2015 at 11:52 PM, Charles R Harris <
On Fri, Jun 19, 2015 at 3:05 PM, Sturla Molden <
Post by Charles R Harris
Post by Charles R Harris
I'm looking to change some numpy deprecations into errors as well
as remove
Post by Charles R Harris
some deprecated functions. The problem I see is that
SciPy claims to support Numpy >= 1.5 and Numpy 1.5 is really,
really, old.
Post by Charles R Harris
So the question is, does "support" mean compiles with earlier
versions
Post by Charles R Harris
of Numpy ?
It means there is a Travis CI build with NumPy 1.6.2. So any change to the
SciPy source code must compile with NumPy 1.6 and any later version of
NumPy.
There is no Travis CI build with NumPy 1.5. I don't think we know for sure
if it is really compatible with the current SciPy.
There is still a reference to 1.5 in Scipy, I forget where.
In INSTALL.rst.txt, will fix that now.
Ralf, I cannot compile Scipy 0.13.3 on my system, it seems to fail here
------------------------------------------------------------
...
# and object. In this file, only NULL is passed to these parameters.
cnp.ndarray PyArray_CheckFromAny(object, void*, int, int, int, void*)
cnp.ndarray PyArray_FromArray(cnp.ndarray, void*, int)
from . cimport cython_blas as blas_pointers
^
------------------------------------------------------------
_decomp_update.pyx:60:0: 'cython_blas.pxd' not found
Although it is hard to tell, the traceback doesn't give much useful
information. I suspect this is due to a cython version mismatch, as it
seems to be looking for cython_blas.pxd but only cython_blas.c is
available. Have you seen this before?
That's code that was only introduced for 0.16.x; a ``git clean -xdf``
should fix this for you.
Next obstacle: I think it'll fail with Cython 0.22, you'll need a lower
Cython version (probably around 0.19.x).
Looks like Scipy 0.13.3 is OK against master apart from a bunch of runtime
errors due to deprecation warnings, precision changes, TypeErrors due to
default casting rule changes, and new runtime warnings about empty slices.
I wouldn't recommend it for use with Numpy 1.10, but it is probably not
fatal to do so. Nothing changes with the deprecation removals added.

Chuck
Charles R Harris
2015-06-21 18:13:50 UTC
Permalink
On Sun, Jun 21, 2015 at 11:49 AM, Charles R Harris <
Post by Charles R Harris
On Sun, Jun 21, 2015 at 5:45 PM, Charles R Harris <
Post by Charles R Harris
On Sun, Jun 21, 2015 at 5:13 PM, Charles R Harris <
Post by Charles R Harris
Post by Ralf Gommers
On Fri, Jun 19, 2015 at 11:52 PM, Charles R Harris <
On Fri, Jun 19, 2015 at 3:05 PM, Sturla Molden <
Post by Charles R Harris
Post by Charles R Harris
I'm looking to change some numpy deprecations into errors as well
as remove
Post by Charles R Harris
some deprecated functions. The problem I see is that
SciPy claims to support Numpy >= 1.5 and Numpy 1.5 is really,
really, old.
Post by Charles R Harris
So the question is, does "support" mean compiles with earlier
versions
Post by Charles R Harris
of Numpy ?
It means there is a Travis CI build with NumPy 1.6.2. So any change to the
SciPy source code must compile with NumPy 1.6 and any later version of
NumPy.
There is no Travis CI build with NumPy 1.5. I don't think we know for sure
if it is really compatible with the current SciPy.
There is still a reference to 1.5 in Scipy, I forget where.
In INSTALL.rst.txt, will fix that now.
Ralf, I cannot compile Scipy 0.13.3 on my system, it seems to fail here
------------------------------------------------------------
...
# and object. In this file, only NULL is passed to these parameters.
cnp.ndarray PyArray_CheckFromAny(object, void*, int, int, int, void*)
cnp.ndarray PyArray_FromArray(cnp.ndarray, void*, int)
from . cimport cython_blas as blas_pointers
^
------------------------------------------------------------
_decomp_update.pyx:60:0: 'cython_blas.pxd' not found
Although it is hard to tell, the traceback doesn't give much useful
information. I suspect this is due to a cython version mismatch, as it
seems to be looking for cython_blas.pxd but only cython_blas.c is
available. Have you seen this before?
That's code that was only introduced for 0.16.x; a ``git clean -xdf``
should fix this for you.
Next obstacle: I think it'll fail with Cython 0.22, you'll need a lower
Cython version (probably around 0.19.x).
Looks like Scipy 0.13.3 is OK against master apart from a bunch of runtime
errors due to deprecation warnings, precision changes, TypeErrors due to
default casting rule changes, and new runtime warnings about empty slices.
I wouldn't recommend it for use with Numpy 1.10, but it is probably not
fatal to do so. Nothing changes with the deprecation removals added.
Scipy 0.14.1 is clean except for InvalidValue warnings and is probably the
earliest I'd recommend as "safe". It was released 6 months ago. Scipy
0.14.0 actually has fewer errors, those resulting from the changes to
default casting rules, so is probably usable also, it was released about a
year ago.

Ralf, thoughts?

Chuck
Ralf Gommers
2015-06-21 19:47:18 UTC
Permalink
Post by Charles R Harris
On Sun, Jun 21, 2015 at 11:49 AM, Charles R Harris <
Post by Charles R Harris
Looks like Scipy 0.13.3 is OK against master apart from a bunch of
runtime errors due to deprecation warnings,
Note that you only get RuntimeWarnings with numpy master, not with a
released version (due to switching tests to 'release' mode).
Post by Charles R Harris
precision changes,
That can always happen, those are usually harmless.
Post by Charles R Harris
TypeErrors due to default casting rule changes,
That's the casting='same_kind' I assume? We did that on purpose and thought
about it quite hard, so that's OK. If there are other, unintended casting
rule changes then I'm not sure.
Post by Charles R Harris
and new runtime warnings about empty slices.
Also not an issue, because they were added on purpose. I think those
warnings are a bit too intrusive at the moment, but that's unrelated to
Scipy 0.13.3
Post by Charles R Harris
I wouldn't recommend it for use with Numpy 1.10, but it is probably not
Post by Charles R Harris
fatal to do so. Nothing changes with the deprecation removals added.
Scipy 0.14.1 is clean except for InvalidValue warnings and is probably the
earliest I'd recommend as "safe". It was released 6 months ago. Scipy
0.14.0 actually has fewer errors, those resulting from the changes to
default casting rules, so is probably usable also, it was released about a
year ago.
Ralf, thoughts?
Sounds like we managed to not break anything seriously in numpy master
recently, so branching 1.10.x seems OK from this point of view.

Ralf

Ralf Gommers
2015-06-21 19:37:22 UTC
Permalink
Post by Charles R Harris
Post by Ralf Gommers
That's code that was only introduced for 0.16.x; a ``git clean -xdf``
should fix this for you.
Next obstacle: I think it'll fail with Cython 0.22, you'll need a lower
Cython version (probably around 0.19.x).
Looks like Scipy 0.13.3 is OK against master apart from a bunch of runtime
errors due to deprecation warnings, precision changes, TypeErrors due to
default casting rule changes, and new runtime warnings about empty slices.
I wouldn't recommend it for use with Numpy 1.10, but it is probably not
fatal to do so.
Thanks for checking.
Post by Charles R Harris
Nothing changes with the deprecation removals added.
You mean the deprecations in your open PR, not old_defines.h right?
Without old_defines.h Scipy 0.13.3 doesn't build.

Ralf
Sturla Molden
2015-06-21 17:42:25 UTC
Permalink
Post by Charles R Harris
Ralf, I cannot compile Scipy 0.13.3 on my system, it seems to fail her
_decomp_update.pyx:60:0: 'cython_blas.pxd' not found
Do you have a clean SciPy 0.13.3 source tree?

cython_blas.pxd is introduced in 0.16, and should be in 0.13 at all.


Sturla
j***@gmail.com
2015-06-19 21:51:22 UTC
Permalink
Post by Charles R Harris
Hi All,
I'm looking to change some numpy deprecations into errors as well as
remove some deprecated functions. The problem I see is that
SciPy claims to support Numpy >= 1.5 and Numpy 1.5 is really, really, old.
So the question is, does "support" mean compiles with earlier versions
of Numpy ? If that is the case there is very little that can be done about
deprecation. OTOH, if it means Scipy can be compiled with more recent numpy
versions but used with earlier Numpy versions (which is a good feat), I'd
like to know. I'd also like to know what the interface requirements are, as
I'd like to remove old_defines.h
numpy 1.6 I think is still accurate
https://github.com/scipy/scipy/pull/4265

As far as I know, you can never compile against a newer and run with an
older version.

We had the discussion recently about backwards versus forwards binary
compatibility

Josef
Post by Charles R Harris
Chuck
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Ralf Gommers
2015-06-21 13:23:10 UTC
Permalink
On Fri, Jun 19, 2015 at 10:08 PM, Charles R Harris <
Post by Charles R Harris
Hi All,
I'm looking to change some numpy deprecations into errors as well as
remove some deprecated functions. The problem I see is that
SciPy claims to support Numpy >= 1.5 and Numpy 1.5 is really, really, old.
Lowest supported Scipy version in master and 0.16.x is 1.6.2. This can be
seen in the main setup.py, scipy/__init__.py and the 0.16.0 release notes.
Post by Charles R Harris
So the question is, does "support" mean compiles with earlier versions
of Numpy ?
Indeed.
Post by Charles R Harris
If that is the case there is very little that can be done about
deprecation.
They can be fixed in Scipy, see for example
https://github.com/scipy/scipy/pull/4378
Post by Charles R Harris
OTOH, if it means Scipy can be compiled with more recent numpy versions
but used with earlier Numpy versions (which is a good feat), I'd like to
know.
That's never a good idea, and in most cases raises errors on import if you
try.
Post by Charles R Harris
I'd also like to know what the interface requirements are, as I'd like to
remove old_defines.h
This can be fixed in Scipy (see PR above); there's still a lot to do there
though. More importantly, I think Cython still relies on this API and
therefore also needs to be updated. This description of changes made in
Theano might be helpful:
http://mail.scipy.org/pipermail/numpy-discussion/2013-November/068209.html

Ralf
Post by Charles R Harris
Chuck
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Loading...