[Numpy-discussion] Integers to negative integer powers, time for a decision.

[Charles R Harris]

Hi All,

The time for NumPy 1.12.0 approaches and I like to have a final decision on
the treatment of integers to negative integer powers with the `**`
operator. The two alternatives looked to be


*Raise an error for arrays and numpy scalars, including 1 and -1 to
negative powers.*
*Pluses*

- Backward compatible
- Allows common powers to be integer, e.g., arange(3)**2
- Consistent with inplace operators
- Fixes current wrong behavior.
- Preserves type


*Minuses*

- Integer overflow
- Computational inconvenience
- Inconsistent with Python integers


*Always return a float *

*Pluses*

- Computational convenience


*Minuses*

- Loss of type
- Possible backward incompatibilities
- Not applicable to inplace operators



Thoughts?

Chuck

[j***@gmail.com]

On Fri, Oct 7, 2016 at 9:12 PM, Charles R Harris

Post by Charles R Harris
Hi All,
The time for NumPy 1.12.0 approaches and I like to have a final decision on
the treatment of integers to negative integer powers with the `**` operator.
The two alternatives looked to be
Raise an error for arrays and numpy scalars, including 1 and -1 to negative
powers.
Pluses
Backward compatible
Allows common powers to be integer, e.g., arange(3)**2
Consistent with inplace operators
Fixes current wrong behavior.
Preserves type
Minuses
Integer overflow
Computational inconvenience
Inconsistent with Python integers
Always return a float
Pluses
Computational convenience
Minuses
Loss of type
Possible backward incompatibilities
Not applicable to inplace operators
Thoughts?

2: +1
I'm still in favor of number 2: less buggy code and less mental
gymnastics (watch out for that int, or which int do I need)

(upcasting is not applicable for any inplace operators, AFAIU <int> *=0.5 ?
zz = np.arange(5)
zz**(-1)
zz *= 0.5

tried in

Post by Charles R Harris

np.__version__

'1.9.2rc1'

Post by Charles R Harris

np.__version__

'1.10.4'
backwards compatibility ?
)


Josef

Post by Charles R Harris
Chuck
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Alan Isaac]

Post by Charles R Harris
*Always return a float *
/Pluses/
* Computational convenience

Is the behavior of C++11 of any relevance to the choice?
http://www.cplusplus.com/reference/cmath/pow/

Alan Isaac

[V. Armando Sole]

Hi all,

Just to have the options clear. Is the operator '**' going to be handled
in any different manner than pow?

Thanks.

Armando

[Nathaniel Smith]

On Fri, Oct 7, 2016 at 6:12 PM, Charles R Harris

Post by Charles R Harris
Hi All,
The time for NumPy 1.12.0 approaches and I like to have a final decision on
the treatment of integers to negative integer powers with the `**` operator.
The two alternatives looked to be
Raise an error for arrays and numpy scalars, including 1 and -1 to negative
powers.
Pluses
Backward compatible
Allows common powers to be integer, e.g., arange(3)**2
Consistent with inplace operators
Fixes current wrong behavior.
Preserves type
Minuses
Integer overflow
Computational inconvenience
Inconsistent with Python integers
Always return a float
Pluses
Computational convenience
Minuses
Loss of type
Possible backward incompatibilities
Not applicable to inplace operators

I guess I could be wrong, but I think the backwards incompatibilities
are going to be *way* too severe to make option 2 possible in
practice.

-n

[Charles R Harris]

Post by Nathaniel Smith
On Fri, Oct 7, 2016 at 6:12 PM, Charles R Harris

Post by Charles R Harris
Hi All,
The time for NumPy 1.12.0 approaches and I like to have a final decision

on

Post by Charles R Harris
the treatment of integers to negative integer powers with the `**`

operator.

Post by Charles R Harris
The two alternatives looked to be
Raise an error for arrays and numpy scalars, including 1 and -1 to

negative

Post by Charles R Harris
powers.
Pluses
Backward compatible
Allows common powers to be integer, e.g., arange(3)**2
Consistent with inplace operators
Fixes current wrong behavior.
Preserves type
Minuses
Integer overflow
Computational inconvenience
Inconsistent with Python integers
Always return a float
Pluses
Computational convenience
Minuses
Loss of type
Possible backward incompatibilities
Not applicable to inplace operators

I guess I could be wrong, but I think the backwards incompatibilities
are going to be *way* too severe to make option 2 possible in
practice.

Backwards compatibility is also a major concern for me. Here are my
current thoughts


- Add an fpow ufunc that always converts to float, it would not accept
object arrays.
- Raise errors in current power ufunc (**), for ints to negative ints.

The power ufunc will change in the following ways


- +1, -1 to negative ints will error, currently they work
- n > 1 ints to negative ints will error, currently warn and return zero
- 0 to negative ints will error, they currently return the minimum
integer

The `**` operator currently calls the power ufunc, leave that as is for
backward almost compatibility. The remaining question is numpy scalars,
which we can make either compatible with Python, or with NumPy arrays. I'm
leaning towards NumPy array compatibility mostly on account of type
preservation and the close relationship between zero dimensionaly arrays
and scalars.


The fpow function could be backported to NumPy 1.11 if that would be
helpful going forward.

Chuck

[Nathaniel Smith]

On Sat, Oct 8, 2016 at 6:59 AM, Charles R Harris

Post by Nathaniel Smith
On Fri, Oct 7, 2016 at 6:12 PM, Charles R Harris

Post by Charles R Harris
Hi All,
The time for NumPy 1.12.0 approaches and I like to have a final decision on
the treatment of integers to negative integer powers with the `**` operator.
The two alternatives looked to be
Raise an error for arrays and numpy scalars, including 1 and -1 to negative
powers.
Pluses
Backward compatible
Allows common powers to be integer, e.g., arange(3)**2
Consistent with inplace operators
Fixes current wrong behavior.
Preserves type
Minuses
Integer overflow
Computational inconvenience
Inconsistent with Python integers
Always return a float
Pluses
Computational convenience
Minuses
Loss of type
Possible backward incompatibilities
Not applicable to inplace operators

I guess I could be wrong, but I think the backwards incompatibilities
are going to be *way* too severe to make option 2 possible in
practice.

Backwards compatibility is also a major concern for me. Here are my current
thoughts
Add an fpow ufunc that always converts to float, it would not accept object
arrays.

Maybe call it `fpower` or even `float_power`, for consistency with `power`?

Raise errors in current power ufunc (**), for ints to negative ints.
The power ufunc will change in the following ways
+1, -1 to negative ints will error, currently they work
n > 1 ints to negative ints will error, currently warn and return zero
0 to negative ints will error, they currently return the minimum integer
The `**` operator currently calls the power ufunc, leave that as is for
backward almost compatibility. The remaining question is numpy scalars,
which we can make either compatible with Python, or with NumPy arrays. I'm
leaning towards NumPy array compatibility mostly on account of type
preservation and the close relationship between zero dimensionaly arrays and
scalars.

Sounds good to me. I agree that we should prioritize within-numpy
consistency over consistency with Python.

The fpow function could be backported to NumPy 1.11 if that would be helpful
going forward.

I'm not a big fan of this kind of backport. Violating the
"bug-fixes-only" rule makes it hard for people to understand our
release versions. And it creates the situation where people can write
code that they think requires numpy 1.11 (because it works with their
numpy 1.11!), but then breaks on other people's computers (because
those users have 1.11.(x-1)). And if there's some reason why people
aren't willing to upgrade to 1.12 for new features, then probably
better to spend energy addressing those instead of on putting together
1.11-and-a-half releases.

-n

[Charles R Harris]

Post by Nathaniel Smith
On Sat, Oct 8, 2016 at 6:59 AM, Charles R Harris

Post by Charles R Harris

Post by Nathaniel Smith
On Fri, Oct 7, 2016 at 6:12 PM, Charles R Harris

Post by Charles R Harris
Hi All,
The time for NumPy 1.12.0 approaches and I like to have a final

decision

Post by Charles R Harris

Post by Nathaniel Smith

Post by Charles R Harris
on
the treatment of integers to negative integer powers with the `**` operator.
The two alternatives looked to be
Raise an error for arrays and numpy scalars, including 1 and -1 to negative
powers.
Pluses
Backward compatible
Allows common powers to be integer, e.g., arange(3)**2
Consistent with inplace operators
Fixes current wrong behavior.
Preserves type
Minuses
Integer overflow
Computational inconvenience
Inconsistent with Python integers
Always return a float
Pluses
Computational convenience
Minuses
Loss of type
Possible backward incompatibilities
Not applicable to inplace operators

I guess I could be wrong, but I think the backwards incompatibilities
are going to be *way* too severe to make option 2 possible in
practice.

Backwards compatibility is also a major concern for me. Here are my

current

Post by Charles R Harris
thoughts
Add an fpow ufunc that always converts to float, it would not accept

object

Post by Charles R Harris
arrays.

Maybe call it `fpower` or even `float_power`, for consistency with `power`?

Post by Charles R Harris
Raise errors in current power ufunc (**), for ints to negative ints.
The power ufunc will change in the following ways
+1, -1 to negative ints will error, currently they work
n > 1 ints to negative ints will error, currently warn and return zero
0 to negative ints will error, they currently return the minimum integer
The `**` operator currently calls the power ufunc, leave that as is for
backward almost compatibility. The remaining question is numpy scalars,
which we can make either compatible with Python, or with NumPy arrays.

I'm

Post by Charles R Harris
leaning towards NumPy array compatibility mostly on account of type
preservation and the close relationship between zero dimensionaly arrays

and

Post by Charles R Harris
scalars.

Sounds good to me. I agree that we should prioritize within-numpy
consistency over consistency with Python.

Post by Charles R Harris
The fpow function could be backported to NumPy 1.11 if that would be

helpful

Post by Charles R Harris
going forward.

I'm not a big fan of this kind of backport. Violating the
"bug-fixes-only" rule makes it hard for people to understand our
release versions. And it creates the situation where people can write
code that they think requires numpy 1.11 (because it works with their
numpy 1.11!), but then breaks on other people's computers (because
those users have 1.11.(x-1)). And if there's some reason why people
aren't willing to upgrade to 1.12 for new features, then probably
better to spend energy addressing those instead of on putting together
1.11-and-a-half releases.

The power ufunc is updated in https://github.com/numpy/numpy/pull/8127.

[Ralf Gommers]

Post by Nathaniel Smith
On Sat, Oct 8, 2016 at 6:59 AM, Charles R Harris

Post by Charles R Harris

Post by Nathaniel Smith
On Fri, Oct 7, 2016 at 6:12 PM, Charles R Harris

Post by Charles R Harris
Hi All,
The time for NumPy 1.12.0 approaches and I like to have a final

decision

Post by Charles R Harris

Post by Nathaniel Smith

Post by Charles R Harris
on
the treatment of integers to negative integer powers with the `**` operator.
The two alternatives looked to be
Raise an error for arrays and numpy scalars, including 1 and -1 to negative
powers.
Pluses
Backward compatible
Allows common powers to be integer, e.g., arange(3)**2
Consistent with inplace operators
Fixes current wrong behavior.
Preserves type
Minuses
Integer overflow
Computational inconvenience
Inconsistent with Python integers
Always return a float
Pluses
Computational convenience
Minuses
Loss of type
Possible backward incompatibilities
Not applicable to inplace operators

I guess I could be wrong, but I think the backwards incompatibilities
are going to be *way* too severe to make option 2 possible in
practice.

Backwards compatibility is also a major concern for me. Here are my

current

Post by Charles R Harris
thoughts
Add an fpow ufunc that always converts to float, it would not accept

object

Post by Charles R Harris
arrays.

Maybe call it `fpower` or even `float_power`, for consistency with `power`?

Post by Charles R Harris
Raise errors in current power ufunc (**), for ints to negative ints.
The power ufunc will change in the following ways
+1, -1 to negative ints will error, currently they work
n > 1 ints to negative ints will error, currently warn and return zero
0 to negative ints will error, they currently return the minimum integer
The `**` operator currently calls the power ufunc, leave that as is for
backward almost compatibility. The remaining question is numpy scalars,
which we can make either compatible with Python, or with NumPy arrays.

I'm

Post by Charles R Harris
leaning towards NumPy array compatibility mostly on account of type
preservation and the close relationship between zero dimensionaly arrays

and

Post by Charles R Harris
scalars.

Sounds good to me. I agree that we should prioritize within-numpy
consistency over consistency with Python.

+1 sounds good to me too.

Post by Nathaniel Smith

Post by Charles R Harris
The fpow function could be backported to NumPy 1.11 if that would be

helpful

Post by Charles R Harris
going forward.

I'm not a big fan of this kind of backport. Violating the
"bug-fixes-only" rule makes it hard for people to understand our
release versions. And it creates the situation where people can write
code that they think requires numpy 1.11 (because it works with their
numpy 1.11!), but then breaks on other people's computers (because
those users have 1.11.(x-1)). And if there's some reason why people
aren't willing to upgrade to 1.12 for new features, then probably
better to spend energy addressing those instead of on putting together
1.11-and-a-half releases.

Agreed, this is not something we want to backport.

Ralf

Post by Nathaniel Smith
-n
--
Nathaniel J. Smith -- https://vorpus.org
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Krisztián Horváth]

Post by Nathaniel Smith
Sounds good to me. I agree that we should prioritize within-numpy
consistency over consistency with Python.

I agree with that. Because of numpy consitetncy, the `**` operator should

Post by Nathaniel Smith

aa = np.arange(2, 10, dtype=int)

array([2, 3, 4, 5, 6, 7, 8, 9])

Post by Nathaniel Smith

bb = np.linspace(0, 7, 8, dtype=int)

array([0, 1, 2, 3, 4, 5, 6, 7])

Post by Nathaniel Smith

1/aa

array([ 0.5 , 0.33333333, 0.25 , 0.2 , 0.16666667,
0.14285714, 0.125 , 0.11111111])

Post by Nathaniel Smith

aa**-1

array([0, 0, 0, 0, 0, 0, 0, 0])

Post by Nathaniel Smith

1/aa**2

array([ 0.25 , 0.11111111, 0.0625 , 0.04 , 0.02777778,
0.02040816, 0.015625 , 0.01234568])

Post by Nathaniel Smith

aa**-2

array([0, 0, 0, 0, 0, 0, 0, 0])

Post by Nathaniel Smith

aa**bb

array([ 1, 3, 16, 125, 1296, 16807, 262144,
4782969])

Post by Nathaniel Smith

1/aa**bb

array([ 1.00000000e+00, 3.33333333e-01, 6.25000000e-02,
8.00000000e-03, 7.71604938e-04, 5.94990183e-05,
3.81469727e-06, 2.09075158e-07])

Post by Nathaniel Smith

aa**(-bb)

array([1, 0, 0, 0, 0, 0, 0, 0])

For me this behaviour is confusing. But I am not an expert just a user. I
can live together with anything if I know what to expect. And I greatly
appreciate the work of any developer for this excellent package.

[Krisztián Horváth]

Hello,

I think it should be consistent with Python3. So, it should give back a
float.

Best regards,
Krisztian

Post by Charles R Harris
Hi All,
The time for NumPy 1.12.0 approaches and I like to have a final decision
on the treatment of integers to negative integer powers with the `**`
operator. The two alternatives looked to be
*Raise an error for arrays and numpy scalars, including 1 and -1 to
negative powers.*
*Pluses*
- Backward compatible
- Allows common powers to be integer, e.g., arange(3)**2
- Consistent with inplace operators
- Fixes current wrong behavior.
- Preserves type
*Minuses*
- Integer overflow
- Computational inconvenience
- Inconsistent with Python integers
*Always return a float *
*Pluses*
- Computational convenience
*Minuses*
- Loss of type
- Possible backward incompatibilities
- Not applicable to inplace operators
Thoughts?
Chuck
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Charles R Harris]

Post by Krisztián Horváth
Hello,
I think it should be consistent with Python3. So, it should give back a
float.
Best regards,
Krisztian

Can't do that and also return integers for positive powers. It isn't
possible to have behavior completely compatible with python for arrays:
can't have mixed type returns, can't have arbitrary precision integers.

Chuck

[Krisztián Horváth]

Sorry, I was not clear enough. I meant that the second option (always
float) would be more coherent with Python3.

Post by Krisztián Horváth
Hello,
I think it should be consistent with Python3. So, it should give back a
float.
Best regards,
Krisztian

Can't do that and also return integers for positive powers. It isn't
possible to have behavior completely compatible with python for arrays:
can't have mixed type returns, can't have arbitrary precision integers.

Chuck

[V. Armando Sole]

Well, testing under windows 64 bit, Python 3.5.2, positive powers of
integers give integers and negative powers of integers give floats. So,
do you want to raise an exception when taking a negative power of an
element of an array of integers? Because not doing so would be
inconsistent with raising the exception when applying the same operation
to the array.

Clearly things are broken now (I get zeros when calculating negative
powers of numpy arrays of integers others than 1), but that behavior was
consistent with python itself under python 2.x because the division of
two integers was an integer. That does not hold under Python 3.5 where
the division of two integers is a float.

You have offered either to raise an exception or to always return a
float (i.e. even with positive exponents). You have never offered to be
consistent with what Python does. This last option would be my favorite.
If it cannot be implemented, then I would prefer always float. At least
one would be consistent with something and we would not invent yet
another convention.

On Sat, Oct 8, 2016 at 1:31 PM, Krisztián Horváth

Post by Krisztián Horváth
Hello,
I think it should be consistent with Python3. So, it should give
back a float.
Best regards,
Krisztian

Can't do that and also return integers for positive powers. It isn't
possible to have behavior completely compatible with python for
arrays: can't have mixed type returns, can't have arbitrary precision
integers.
Chuck
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Nathaniel Smith]

Post by V. Armando Sole
Well, testing under windows 64 bit, Python 3.5.2, positive powers of
integers give integers and negative powers of integers give floats. So, do
you want to raise an exception when taking a negative power of an element of
an array of integers? Because not doing so would be inconsistent with
raising the exception when applying the same operation to the array.
Clearly things are broken now (I get zeros when calculating negative powers
of numpy arrays of integers others than 1), but that behavior was consistent
with python itself under python 2.x because the division of two integers was
an integer. That does not hold under Python 3.5 where the division of two
integers is a float.

sys.version_info

sys.version_info(major=2, minor=7, micro=12, releaselevel='final', serial=0)

Post by V. Armando Sole

2 ** -2

0.25

Post by V. Armando Sole
You have offered either to raise an exception or to always return a float
(i.e. even with positive exponents). You have never offered to be consistent
with what Python does. This last option would be my favorite. If it cannot
be implemented, then I would prefer always float. At least one would be
consistent with something and we would not invent yet another convention.

Numpy tries to be consistent with Python when it makes sense, but this
is only one of several considerations. The use cases for numpy objects
are different from the use cases for Python scalar objects, so we also
consistently deviate in cases when that makes sense -- e.g., numpy
bools are very different from Python bools (Python barely
distinguishes between bools and integers, because they don't need to;
indexing makes the distinction much more important to numpy), numpy
integers are very different from Python integers (Python's
arbitrary-width integers provide great semantics, but don't play
nicely with large fixed-size arrays), numpy pays much more attention
to type consistency between inputs and outputs than Python does (again
because of the extra constraints imposed by working with
memory-intensive type-consistent arrays), etc.

For python, 2 ** 2 -> int, 2 ** -2 -> float. But numpy can't do this,
because then 2 ** np.array([2, -2]) would have to be both int *and*
float, which it can't be. Not a problem that Python has. Or we could
say that the output is int if all the inputs are positive, and float
if any of them are negative... but then that violates the numpy
principle that output dtypes should be determined entirely by input
dtypes, without peeking at the actual values. (And this rule is very
important for avoiding nasty surprises when you run your code on new
inputs.)

And then there's backwards compatibility to consider. As mentioned, we
*could* deviate from Python by making ** always return float... but
this would almost certainly break tons and tons of people's code that
is currently doing integer ** positive integer and expecting to get an
integer back. Which is something we don't do without very careful
weighing of the trade-offs, and my intuition is that this one is so
disruptive we probably can't pull it off. Breaking working code needs
a *very* compelling reason.

-n

[Krisztián Horváth]

but then that violates the numpy

Post by Nathaniel Smith
principle that output dtypes should be determined entirely by input
dtypes, without peeking at the actual values. (And this rule is very
important for avoiding nasty surprises when you run your code on new
inputs.)

At division you get back an array of floats.

Post by Nathaniel Smith

y = np.int64([1,2,4])
y/1

array([ 1., 2., 4.])

Post by Nathaniel Smith

y/y

array([ 1., 1., 1.])

Why is it different, if you calculate the power of something?

Post by Nathaniel Smith
And then there's backwards compatibility to consider. As mentioned, we
*could* deviate from Python by making ** always return float... but
this would almost certainly break tons and tons of people's code that
is currently doing integer ** positive integer and expecting to get an
integer back. Which is something we don't do without very careful
weighing of the trade-offs, and my intuition is that this one is so
disruptive we probably can't pull it off. Breaking working code needs
a *very* compelling reason.

This is a valid reasoning. But it could be solved with raising an exception
to warn the users for the new behaviour.

[Nathaniel Smith]

Post by Krisztián Horváth

Post by Nathaniel Smith
but then that violates the numpy
principle that output dtypes should be determined entirely by input
dtypes, without peeking at the actual values. (And this rule is very
important for avoiding nasty surprises when you run your code on new
inputs.)

At division you get back an array of floats.

Post by Nathaniel Smith

y = np.int64([1,2,4])
y/1

array([ 1., 2., 4.])

Post by Nathaniel Smith

y/y

array([ 1., 1., 1.])
Why is it different, if you calculate the power of something?

The difference is that Python division always returns float. Python
int ** int sometimes returns int and sometimes returns float,
depending on which particular integers are used. We can't be
consistent with Python because Python isn't consistent with itself.

Post by Krisztián Horváth

Post by Nathaniel Smith
And then there's backwards compatibility to consider. As mentioned, we
*could* deviate from Python by making ** always return float... but
this would almost certainly break tons and tons of people's code that
is currently doing integer ** positive integer and expecting to get an
integer back. Which is something we don't do without very careful
weighing of the trade-offs, and my intuition is that this one is so
disruptive we probably can't pull it off. Breaking working code needs
a *very* compelling reason.

This is a valid reasoning. But it could be solved with raising an exception
to warn the users for the new behaviour.

That is generally the best conservative strategy for making a
backwards incompatible change like this: instead of going straight to
the new behavior, first make it raise an error, and then once people
have had time to stop depending on the old behavior, then you can add
the new behavior. But in this case if we were going to make int ** int
return float, this rule would mean that we have to make int ** int
always raise an error for a few years, i.e. remove integer power
support from numpy altogether. That's a non-starter.

-n

[Sebastian Berg]

Post by Charles R Harris
Hi All,
The time for NumPy 1.12.0 approaches and I like to have a final
decision on the treatment of integers to negative integer powers with
the `**` operator. The two alternatives looked to be
Raise an error for arrays and numpy scalars, including 1 and -1 to
negative powers.

For what its worth, I still feel it is probably the only real option to
go with error, changing to float may have weird effects. Which does not
mean it is impossible, I admit, though I would like some data on how
downstream would handle it. Also would we need an int power? The fpower
seems more straight forward/common pattern.
If errors turned out annoying in some cases, a seterr might be
plausible too (as well as a deprecation).

- Sebastian

Post by Charles R Harris
Pluses
Backward compatible
Allows common powers to be integer, e.g., arange(3)**2
Consistent with inplace operators
Fixes current wrong behavior.
Preserves type
Minuses
Integer overflow
Computational inconvenience
Inconsistent with Python integers
Always return a float 
Pluses
Computational convenience
Minuses
Loss of type
Possible backward incompatibilities
Not applicable to inplace operators
Thoughts?
Chuck
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Stephan Hoyer]

Post by Sebastian Berg
For what its worth, I still feel it is probably the only real option to
go with error, changing to float may have weird effects. Which does not
mean it is impossible, I admit, though I would like some data on how
downstream would handle it. Also would we need an int power? The fpower
seems more straight forward/common pattern.
If errors turned out annoying in some cases, a seterr might be
plausible too (as well as a deprecation).

I agree with Sebastian and Nathaniel. I don't think we can deviating from
the existing behavior (int ** int -> int) without breaking lots of existing
code, and if we did, yes, we would need a new integer power function.

I think it's better to preserve the existing behavior when it gives
sensible results, and error when it doesn't. Adding another function
float_power for the case that is currently broken seems like the right way
to go.

[Ryan May]

Post by Stephan Hoyer

Post by Sebastian Berg
For what its worth, I still feel it is probably the only real option to
go with error, changing to float may have weird effects. Which does not
mean it is impossible, I admit, though I would like some data on how
downstream would handle it. Also would we need an int power? The fpower
seems more straight forward/common pattern.
If errors turned out annoying in some cases, a seterr might be
plausible too (as well as a deprecation).

I agree with Sebastian and Nathaniel. I don't think we can deviating from
the existing behavior (int ** int -> int) without breaking lots of existing
code, and if we did, yes, we would need a new integer power function.
I think it's better to preserve the existing behavior when it gives
sensible results, and error when it doesn't. Adding another function
float_power for the case that is currently broken seems like the right way
to go.

+1

Ryan

[Peter Creasey]

Post by Stephan Hoyer
I agree with Sebastian and Nathaniel. I don't think we can deviating from
the existing behavior (int ** int -> int) without breaking lots of existing
code, and if we did, yes, we would need a new integer power function.
I think it's better to preserve the existing behavior when it gives
sensible results, and error when it doesn't. Adding another function
float_power for the case that is currently broken seems like the right way
to go.

I actually suspect that the amount of code broken by int**int->float
may be relatively small (though extremely annoying for those that it
happens to, and it would definitely be good to have statistics). I
mean, Numpy silently transitioned to int32+uint64->float64 not so long
ago which broke my code, but the world didn’t end.

If the primary argument against int**int->float seems to be the
difficulty of managing the transition, with int**int->Error being the
seen as the required yet *very* painful intermediate step for the
large fraction of the int**int users who didn’t care if it was int or
float (e.g. the output is likely to be cast to float in the next step
anyway), and fail loudly for those users who need int**int->int, then
if you are prepared to risk a less conservative transition (i.e. we
think that latter group is small enough) you could skip the error on
users and just throw a warning for a couple of releases, along the
lines of:

WARNING int**int -> int is going to be deprecated in favour of
int**int->float in Numpy 1.16. To avoid seeing this message, either
use “from numpy import __future_float_power__” or explicitly set the
type of one of your inputs to float, or use the new ipower(x,y)
function for integer powers.

Peter

[Marten van Kerkwijk]

I still strongly favour ending up at int**int -> float, and like
Peter's suggestion of raising a general warning rather than an
exception for negative powers. -- Marten

[Yaroslav Halchenko]

Post by Peter Creasey

Post by Stephan Hoyer
I agree with Sebastian and Nathaniel. I don't think we can deviating from
the existing behavior (int ** int -> int) without breaking lots of existing
code, and if we did, yes, we would need a new integer power function.
I think it's better to preserve the existing behavior when it gives
sensible results, and error when it doesn't. Adding another function
float_power for the case that is currently broken seems like the right way
to go.

I actually suspect that the amount of code broken by int**int->float
may be relatively small (though extremely annoying for those that it
happens to, and it would definitely be good to have statistics). I
mean, Numpy silently transitioned to int32+uint64->float64 not so long
ago which broke my code, but the world didn’t end.
If the primary argument against int**int->float seems to be the
difficulty of managing the transition, with int**int->Error being the
seen as the required yet *very* painful intermediate step for the
large fraction of the int**int users who didn’t care if it was int or
float (e.g. the output is likely to be cast to float in the next step
anyway), and fail loudly for those users who need int**int->int, then
if you are prepared to risk a less conservative transition (i.e. we
think that latter group is small enough) you could skip the error on
users and just throw a warning for a couple of releases, along the
WARNING int**int -> int is going to be deprecated in favour of
int**int->float in Numpy 1.16. To avoid seeing this message, either
use “from numpy import __future_float_power__” or explicitly set the
type of one of your inputs to float, or use the new ipower(x,y)
function for integer powers.

Sorry for coming too late to the discussion and after PR "addressing"
the issue by issuing an error was merged [1]. I got burnt by new
behavior while trying to build fresh pandas release on Debian (we are
freezing for release way too soon ;) ) -- some pandas tests failed since
they rely on previous non-erroring behavior and we got numpy 1.12.0~b1
which included [1] in unstable/testing (candidate release) now.

I quickly glanced over the discussion but I guess I have missed
actual description of the problem being fixed here... what was it??

previous behavior, int**int->int made sense to me as it seemed to be
consistent with casting Python's pow result to int, somewhat fulfilling
desired promise for in-place operations and being inline with built-in
pow results as far as I see it (up to casting).

Current handling and error IMHO is going against rudimentary
algebra, where numbers can be brought to negative power (integer or
not).

[1] https://github.com/numpy/numpy/pull/8231

[Stephan Hoyer]

Post by Yaroslav Halchenko
Sorry for coming too late to the discussion and after PR "addressing"
the issue by issuing an error was merged [1]. I got burnt by new
behavior while trying to build fresh pandas release on Debian (we are
freezing for release way too soon ;) ) -- some pandas tests failed since
they rely on previous non-erroring behavior and we got numpy 1.12.0~b1
which included [1] in unstable/testing (candidate release) now.
I quickly glanced over the discussion but I guess I have missed
actual description of the problem being fixed here... what was it??
previous behavior, int**int->int made sense to me as it seemed to be
consistent with casting Python's pow result to int, somewhat fulfilling
desired promise for in-place operations and being inline with built-in
pow results as far as I see it (up to casting).

I believe this is exactly the behavior we preserved. Rather, we turned some
cases that previously often gave wrong results (involving negative integer
powers) into errors.

The pandas test suite triggered this behavior, but not intentionally, and
should be fixed in the next release:
https://github.com/pandas-dev/pandas/pull/14498

[Yaroslav Halchenko]

Post by Yaroslav Halchenko
Sorry for coming too late to the discussion and after PR "addressing"
the issue by issuing an error was merged [1].A I got burnt by new
behavior while trying to build fresh pandas release on Debian (we are
freezing for release way too soon ;) ) -- some pandas tests failed since
they rely on previous non-erroring behavior and we gotA numpy 1.12.0~b1
which included [1] in unstable/testing (candidate release) now.
I quickly glanced over the discussion but I guess I have missed
actual description of the problem being fixed here...A what was it??
previous behavior, int**int->int made sense to me as it seemed to be
consistent with casting Python's pow result to int, somewhat fulfilling
desired promise for in-place operations and being inline with built-in
pow results as far as I see it (up to casting).
I believe this is exactly the behavior we preserved. Rather, we turned
some cases that previously often gave wrong results (involving negative
integer powers) into errors.

hm... testing on current master (first result is from python's pow)

$> python -c "import numpy; print('numpy version: ', numpy.__version__); a=2; b=-2; print(pow(a,b)); print(pow(numpy.array(a), b))"
('numpy version: ', '1.13.0.dev0+02e2ea8')
0.25
Traceback (most recent call last):
File "<string>", line 1, in <module>
ValueError: Integers to negative integer powers are not allowed.


testing on Debian's packaged beta

$> python -c "import numpy; print('numpy version: ', numpy.__version__); a=2; b=-2; print(pow(a,b)); print(pow(numpy.array(a), b))"
('numpy version: ', '1.12.0b1')
0.25
Traceback (most recent call last):
File "<string>", line 1, in <module>
ValueError: Integers to negative integer powers are not allowed.


testing on stable debian box with elderly numpy, where it does behave sensibly:

$> python -c "import numpy; print('numpy version: ', numpy.__version__); a=2; b=-2; print(pow(a,b)); print(pow(numpy.array(a), b))"
('numpy version: ', '1.8.2')
0.25
0

what am I missing?

Post by Yaroslav Halchenko
The pandas test suite triggered this behavior, but not intentionally, and
https://github.com/pandas-dev/pandas/pull/14498

I don't think that was the full set of cases, e.g.

(git)hopa/sid-i386:~exppsy/pandas[bf-i386]
$> nosetests -s -v pandas/tests/test_expressions.py:TestExpressions.test_mixed_arithmetic_series
test_mixed_arithmetic_series (pandas.tests.test_expressions.TestExpressions) ... ERROR

======================================================================
ERROR: test_mixed_arithmetic_series (pandas.tests.test_expressions.TestExpressions)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py", line 223, in test_mixed_arithmetic_series
self.run_series(self.mixed2[col], self.mixed2[col], binary_comp=4)
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py", line 164, in run_series
test_flex=False, **kwargs)
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py", line 93, in run_arithmetic_test
expected = op(df, other)
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/ops.py", line 715, in wrapper
result = wrap_results(safe_na_op(lvalues, rvalues))
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/ops.py", line 676, in safe_na_op
return na_op(lvalues, rvalues)
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/ops.py", line 652, in na_op
raise_on_error=True, **eval_kwargs)
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/computation/expressions.py", line 210, in evaluate
**eval_kwargs)
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/computation/expressions.py", line 63, in _evaluate_standard
return op(a, b)
ValueError: Integers to negative integer powers are not allowed.


and being paranoid, I have rebuilt exact current master of pandas with
master numpy in PYTHONPATH:

(git)hopa:~exppsy/pandas[master]git
$> PYTHONPATH=/home/yoh/proj/numpy nosetests -s -v pandas/tests/test_expressions.py:TestExpressions.test_mixed_arithmetic_series
test_mixed_arithmetic_series (pandas.tests.test_expressions.TestExpressions) ... ERROR

======================================================================
ERROR: test_mixed_arithmetic_series (pandas.tests.test_expressions.TestExpressions)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py", line 223, in test_mixed_arithmetic_series
self.run_series(self.mixed2[col], self.mixed2[col], binary_comp=4)
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py", line 164, in run_series
test_flex=False, **kwargs)
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py", line 93, in run_arithmetic_test
expected = op(df, other)
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/ops.py", line 715, in wrapper
result = wrap_results(safe_na_op(lvalues, rvalues))
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/ops.py", line 676, in safe_na_op
return na_op(lvalues, rvalues)
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/ops.py", line 652, in na_op
raise_on_error=True, **eval_kwargs)
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/computation/expressions.py", line 210, in evaluate
**eval_kwargs)
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/computation/expressions.py", line 63, in _evaluate_standard
return op(a, b)
ValueError: Integers to negative integer powers are not allowed.

----------------------------------------------------------------------
Ran 1 test in 0.015s

FAILED (errors=1)

$> git describe --tags
v0.19.0-303-gb957f6f

$> PYTHONPATH=/home/yoh/proj/numpy python -c "import numpy; print('numpy version: ', numpy.__version__); a=2; b=-2; print(pow(a,b)); print(pow(numpy.array(a), b))" ('numpy version: ', '1.13.0.dev0+02e2ea8')
0.25
Traceback (most recent call last):
File "<string>", line 1, in <module>
ValueError: Integers to negative integer powers are not allowed.

[Nathaniel Smith]

It's possible we should back off to just issuing a deprecation warning in
1.12?

Post by Yaroslav Halchenko

On Tue, Jan 3, 2017 at 9:00 AM, Yaroslav Halchenko <
Sorry for coming too late to the discussion and after PR

"addressing"

the issue by issuing an error was merged [1].A I got burnt by new
behavior while trying to build fresh pandas release on Debian (we

are

freezing for release way too soon ;) ) -- some pandas tests failed

since

they rely on previous non-erroring behavior and we gotA numpy

1.12.0~b1

which included [1] in unstable/testing (candidate release) now.
I quickly glanced over the discussion but I guess I have missed
actual description of the problem being fixed here...A what was

it??

previous behavior, int**int->int made sense to me as it seemed to be
consistent with casting Python's pow result to int, somewhat

fulfilling

desired promise for in-place operations and being inline with

built-in

pow results as far as I see it (up to casting).
I believe this is exactly the behavior we preserved. Rather, we turned
some cases that previously often gave wrong results (involving

negative

integer powers) into errors.

hm... testing on current master (first result is from python's pow)
$> python -c "import numpy; print('numpy version: ', numpy.__version__);
a=2; b=-2; print(pow(a,b)); print(pow(numpy.array(a), b))"
('numpy version: ', '1.13.0.dev0+02e2ea8')
0.25
File "<string>", line 1, in <module>
ValueError: Integers to negative integer powers are not allowed.
testing on Debian's packaged beta
$> python -c "import numpy; print('numpy version: ', numpy.__version__);
a=2; b=-2; print(pow(a,b)); print(pow(numpy.array(a), b))"
('numpy version: ', '1.12.0b1')
0.25
File "<string>", line 1, in <module>
ValueError: Integers to negative integer powers are not allowed.
$> python -c "import numpy; print('numpy version: ', numpy.__version__);
a=2; b=-2; print(pow(a,b)); print(pow(numpy.array(a), b))"
('numpy version: ', '1.8.2')
0.25
0
what am I missing?

The pandas test suite triggered this behavior, but not intentionally,

and

https://github.com/pandas-dev/pandas/pull/14498

I don't think that was the full set of cases, e.g.
(git)hopa/sid-i386:~exppsy/pandas[bf-i386]
$> nosetests -s -v pandas/tests/test_expressions.
py:TestExpressions.test_mixed_arithmetic_series
test_mixed_arithmetic_series (pandas.tests.test_expressions.TestExpressions) ... ERROR
======================================================================
ERROR: test_mixed_arithmetic_series (pandas.tests.test_
expressions.TestExpressions)
----------------------------------------------------------------------
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py",
line 223, in test_mixed_arithmetic_series
self.run_series(self.mixed2[col], self.mixed2[col], binary_comp=4)
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py",
line 164, in run_series
test_flex=False, **kwargs)
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py",
line 93, in run_arithmetic_test
expected = op(df, other)
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/ops.py", line 715, in wrapper
result = wrap_results(safe_na_op(lvalues, rvalues))
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/ops.py", line 676, in safe_na_op
return na_op(lvalues, rvalues)
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/ops.py", line 652, in na_op
raise_on_error=True, **eval_kwargs)
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/computation/expressions.py",
line 210, in evaluate
**eval_kwargs)
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/computation/expressions.py",
line 63, in _evaluate_standard
return op(a, b)
ValueError: Integers to negative integer powers are not allowed.
and being paranoid, I have rebuilt exact current master of pandas with
(git)hopa:~exppsy/pandas[master]git
$> PYTHONPATH=/home/yoh/proj/numpy nosetests -s -v
pandas/tests/test_expressions.py:TestExpressions.test_mixed_
arithmetic_series
test_mixed_arithmetic_series (pandas.tests.test_expressions.TestExpressions) ... ERROR
======================================================================
ERROR: test_mixed_arithmetic_series (pandas.tests.test_
expressions.TestExpressions)
----------------------------------------------------------------------
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py",
line 223, in test_mixed_arithmetic_series
self.run_series(self.mixed2[col], self.mixed2[col], binary_comp=4)
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py",
line 164, in run_series
test_flex=False, **kwargs)
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py",
line 93, in run_arithmetic_test
expected = op(df, other)
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/ops.py", line 715, in wrapper
result = wrap_results(safe_na_op(lvalues, rvalues))
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/ops.py", line 676, in safe_na_op
return na_op(lvalues, rvalues)
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/ops.py", line 652, in na_op
raise_on_error=True, **eval_kwargs)
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/computation/expressions.py",
line 210, in evaluate
**eval_kwargs)
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/computation/expressions.py",
line 63, in _evaluate_standard
return op(a, b)
ValueError: Integers to negative integer powers are not allowed.
----------------------------------------------------------------------
Ran 1 test in 0.015s
FAILED (errors=1)
$> git describe --tags
v0.19.0-303-gb957f6f
$> PYTHONPATH=/home/yoh/proj/numpy python -c "import numpy; print('numpy
version: ', numpy.__version__); a=2; b=-2; print(pow(a,b));
print(pow(numpy.array(a), b))"
('numpy version: ', '1.13.0.dev0+02e2ea8')
0.25
File "<string>", line 1, in <module>
ValueError: Integers to negative integer powers are not allowed.
--
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: [phone] Fax: [phone]
WWW: http://www.linkedin.com/in/yarik
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Stephan Hoyer]

Post by Nathaniel Smith
It's possible we should back off to just issuing a deprecation warning in
1.12?

Post by Yaroslav Halchenko
hm... testing on current master (first result is from python's pow)
$> python -c "import numpy; print('numpy version: ', numpy.__version__);
a=2; b=-2; print(pow(a,b)); print(pow(numpy.array(a), b))"
('numpy version: ', '1.13.0.dev0+02e2ea8')
0.25
File "<string>", line 1, in <module>
ValueError: Integers to negative integer powers are not allowed.
testing on Debian's packaged beta
$> python -c "import numpy; print('numpy version: ', numpy.__version__);
a=2; b=-2; print(pow(a,b)); print(pow(numpy.array(a), b))"
('numpy version: ', '1.12.0b1')
0.25
File "<string>", line 1, in <module>
ValueError: Integers to negative integer powers are not allowed.
$> python -c "import numpy; print('numpy version: ', numpy.__version__);
a=2; b=-2; print(pow(a,b)); print(pow(numpy.array(a), b))"
('numpy version: ', '1.8.2')
0.25
0
what am I missing?

2 ** -2 should be 0.25.

On old versions of NumPy, you see the the incorrect answer 0. We are now
preferring to give an error rather than the wrong answer.

Post by Nathaniel Smith

Post by Yaroslav Halchenko
The pandas test suite triggered this behavior, but not intentionally, and

Post by Stephan Hoyer
https://github.com/pandas-dev/pandas/pull/14498

I don't think that was the full set of cases, e.g.
(git)hopa/sid-i386:~exppsy/pandas[bf-i386]
$> nosetests -s -v pandas/tests/test_expressions.
py:TestExpressions.test_mixed_arithmetic_series
test_mixed_arithmetic_series (pandas.tests.test_expressions.TestExpressions) ... ERROR
======================================================================
ERROR: test_mixed_arithmetic_series (pandas.tests.test_expressions
.TestExpressions)
----------------------------------------------------------------------
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py",
line 223, in test_mixed_arithmetic_series
self.run_series(self.mixed2[col], self.mixed2[col], binary_comp=4)
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py",
line 164, in run_series
test_flex=False, **kwargs)
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_expressions.py",
line 93, in run_arithmetic_test
expected = op(df, other)
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/ops.py", line 715, in wrapper
result = wrap_results(safe_na_op(lvalues, rvalues))
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/ops.py", line 676, in safe_na_op
return na_op(lvalues, rvalues)
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/ops.py", line 652, in na_op
raise_on_error=True, **eval_kwargs)
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/computation/expressions.py",
line 210, in evaluate
**eval_kwargs)
File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/computation/expressions.py",
line 63, in _evaluate_standard
return op(a, b)
ValueError: Integers to negative integer powers are not allowed.

Agreed, it looks like pandas still has this issue in the test suite.
Nonetheless, I don't think this should be an issue for users -- pandas
defines all handling of arithmetic to numpy.

[Yaroslav Halchenko]

Post by Stephan Hoyer

Post by Yaroslav Halchenko
$> python -c "import numpy; print('numpy version: ', numpy.__version__);
a=2; b=-2; print(pow(a,b)); print(pow(numpy.array(a), b))"
('numpy version: ', '1.8.2')
0.25
0
what am I missing?

2 ** -2 should be 0.25.
On old versions of NumPy, you see the the incorrect answer 0. We are now
preferring to give an error rather than the wrong answer.

it is correct up to casting/truncating to an int for the desire to
maintain the int data type -- the same as

Post by Stephan Hoyer

Post by Yaroslav Halchenko
int(0.25)

0

Post by Stephan Hoyer

Post by Yaroslav Halchenko
1/4

0

or even

Post by Stephan Hoyer

Post by Yaroslav Halchenko
np.arange(5)/4

array([0, 0, 0, 0, 1])

so it is IMHO more of a documented feature and I don't see why pow needs
to get all so special. Sure thing, in the bring future, unless
in-place operation is demanded I would have voted for consistent float
output.