[Numpy-discussion] Deprecating matrices.

[Charles R Harris]

Hi All,

Just throwing this click bait out for discussion. Now that the `@` operator
is available and things seem to be moving towards Python 3, especially in
the classroom, we should consider the real possibility of deprecating the
matrix type and later removing it. No doubt there are old scripts that
require them, but older versions of numpy are available for those who need
to run old scripts.

Thoughts?

Chuck

[Ralf Gommers]

Post by Charles R Harris
Hi All,
operator is available and things seem to be moving towards Python 3,
especially in the classroom, we should consider the real possibility of
deprecating the matrix type and later removing it. No doubt there are old
scripts that require them, but older versions of numpy are available for
those who need to run old scripts.
Thoughts?

Clearly deprecate in the docs now, and warn only later imho. We can't warn
before we have a good solution for scipy.sparse matrices, which have matrix
semantics and return matrix instances.

Ralf

[j***@gmail.com]

On Tue, Jan 3, 2017 at 2:36 PM, Charles R Harris <

Post by Charles R Harris
Hi All,
operator is available and things seem to be moving towards Python 3,
especially in the classroom, we should consider the real possibility of
deprecating the matrix type and later removing it. No doubt there are old
scripts that require them, but older versions of numpy are available for
those who need to run old scripts.
Thoughts?

Clearly deprecate in the docs now, and warn only later imho. We can't warn
before we have a good solution for scipy.sparse matrices, which have matrix
semantics and return matrix instances.
Ralf

How about dropping python 2 support at the same time, then we can all be in
a @ world.

Josef

_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Nathaniel Smith]

Post by j***@gmail.com

On Tue, Jan 3, 2017 at 2:36 PM, Charles R Harris

Post by Charles R Harris
Hi All,
operator is available and things seem to be moving towards Python 3,
especially in the classroom, we should consider the real possibility of
deprecating the matrix type and later removing it. No doubt there are old
scripts that require them, but older versions of numpy are available for
those who need to run old scripts.
Thoughts?

Clearly deprecate in the docs now, and warn only later imho. We can't warn
before we have a good solution for scipy.sparse matrices, which have matrix
semantics and return matrix instances.
Ralf

How about dropping python 2 support at the same time, then we can all be in
Josef

Let's not yoke together two (mostly) unrelated controversial
discussions? I doubt we'll be able to remove either Python 2 or matrix
support before 2020 at the earliest, so the discussion now is just
about how to communicate to users that they should not be using
'matrix'.

-n

[Charles R Harris]

Post by j***@gmail.com

On Tue, Jan 3, 2017 at 2:36 PM, Charles R Harris <

Post by Charles R Harris
Hi All,
operator is available and things seem to be moving towards Python 3,
especially in the classroom, we should consider the real possibility of
deprecating the matrix type and later removing it. No doubt there are old
scripts that require them, but older versions of numpy are available for
those who need to run old scripts.
Thoughts?

Clearly deprecate in the docs now, and warn only later imho. We can't
warn before we have a good solution for scipy.sparse matrices, which have
matrix semantics and return matrix instances.
Ralf

How about dropping python 2 support at the same time, then we can all be

The "@" operator works with matrices already, what causes problems is the
combination of matrices with 1-D arrays. That can be fixed, I think. The
big problem is probably the lack of "@" in Python 2.7. I wonder if there is
any chance of getting it backported to 2.7 before support is dropped in
2020? I expect it would be a fight, but I also suspect it would not be
difficult to do if the proposal was accepted. Then at some future date
sparse could simply start returning arrays.

Chuck

[Charles R Harris]

Post by Charles R Harris

Post by j***@gmail.com

On Tue, Jan 3, 2017 at 2:36 PM, Charles R Harris <

Post by Charles R Harris
Hi All,
operator is available and things seem to be moving towards Python 3,
especially in the classroom, we should consider the real possibility of
deprecating the matrix type and later removing it. No doubt there are old
scripts that require them, but older versions of numpy are available for
those who need to run old scripts.
Thoughts?

Clearly deprecate in the docs now, and warn only later imho. We can't
warn before we have a good solution for scipy.sparse matrices, which have
matrix semantics and return matrix instances.
Ralf

How about dropping python 2 support at the same time, then we can all be

combination of matrices with 1-D arrays. That can be fixed, I think. The
any chance of getting it backported to 2.7 before support is dropped in
2020? I expect it would be a fight, but I also suspect it would not be
difficult to do if the proposal was accepted. Then at some future date
sparse could simply start returning arrays.

Hmm, matrix-scalar multiplication will be a problem.

Chuck

[Nathaniel Smith]

On Mon, Jan 2, 2017 at 7:12 PM, Charles R Harris
[...]

Post by Charles R Harris

Post by j***@gmail.com
How about dropping python 2 support at the same time, then we can all be

combination of matrices with 1-D arrays. That can be fixed, I think. The
any chance of getting it backported to 2.7 before support is dropped in
2020? I expect it would be a fight, but I also suspect it would not be
difficult to do if the proposal was accepted. Then at some future date
sparse could simply start returning arrays.

Unfortunately the chance of Python 2.7 adding support for "@" is best
expressed as a denormal.

-n

[Charles R Harris]

Post by Nathaniel Smith
On Mon, Jan 2, 2017 at 7:12 PM, Charles R Harris
[...]

Post by Charles R Harris

Post by j***@gmail.com
How about dropping python 2 support at the same time, then we can all be

combination of matrices with 1-D arrays. That can be fixed, I think. The

is

Post by Charles R Harris
any chance of getting it backported to 2.7 before support is dropped in
2020? I expect it would be a fight, but I also suspect it would not be
difficult to do if the proposal was accepted. Then at some future date
sparse could simply start returning arrays.

expressed as a denormal.

That's what I figured ;) Hmm, matrices would work fine with the current
combination of '*' (works for scalar muiltiplication) and '@' (works for
matrices). So for Python3 code currently written for matrices can be
reformed to be array compatible. But '@' for Python 2.7 would sure help...

Chuck

[Nathaniel Smith]

On Mon, Jan 2, 2017 at 7:54 PM, Charles R Harris

Post by Charles R Harris

Post by Nathaniel Smith
On Mon, Jan 2, 2017 at 7:12 PM, Charles R Harris
[...]

Post by Charles R Harris

Post by j***@gmail.com
How about dropping python 2 support at the same time, then we can all be

combination of matrices with 1-D arrays. That can be fixed, I think. The
any chance of getting it backported to 2.7 before support is dropped in
2020? I expect it would be a fight, but I also suspect it would not be
difficult to do if the proposal was accepted. Then at some future date
sparse could simply start returning arrays.

expressed as a denormal.

That's what I figured ;) Hmm, matrices would work fine with the current
matrices). So for Python3 code currently written for matrices can be

I mean, it can just use arrays + dot(). It's not as elegant as '@',
but given that almost everyone has already switched it's clearly not
*that* bad...

-n

[Todd]

Post by Charles R Harris
Hi All,
operator is available and things seem to be moving towards Python 3,
especially in the classroom, we should consider the real possibility of
deprecating the matrix type and later removing it. No doubt there are old
scripts that require them, but older versions of numpy are available for
those who need to run old scripts.
Thoughts?
Chuck

What if the matrix class was split out into its own project, perhaps as a
scikit. That way those who really need it can still use it. If there is
sufficient desire for it, those who need it can maintain it. If not, it
will hopefully it will take long enough for it to bitrot that everyone has
transitioned.

[Benjamin Root]

That's not a bad idea. Matplotlib is currently considering something
similar for its mlab module. It has been there since the beginning, but it
is very outdated and very out-of-scope for matplotlib. However, there are
still lots of code out there that depends on it. So, we are looking to
split it off as its own package. The details still need to be worked out
(should we initially depend on the package and simply alias its import with
a DeprecationWarning, or should we go cold turkey and have a good message
explaining the change).

Ben Root

On Mon, Jan 2, 2017 at 8:36 PM, Charles R Harris <

Post by Charles R Harris
Hi All,
operator is available and things seem to be moving towards Python 3,
especially in the classroom, we should consider the real possibility of
deprecating the matrix type and later removing it. No doubt there are old
scripts that require them, but older versions of numpy are available for
those who need to run old scripts.
Thoughts?
Chuck

What if the matrix class was split out into its own project, perhaps as a
scikit. That way those who really need it can still use it. If there is
sufficient desire for it, those who need it can maintain it. If not, it
will hopefully it will take long enough for it to bitrot that everyone has
transitioned.
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Bryan Van de Ven]

There's a good chance that bokeh.charts will be split off into a separately distributed package as well. Hopefully being a much smaller, pure Python project makes it a more accessible target for anyone interested in maintaining it, and if no one is interested in it anymore, well that fact becomes easier to judge. I think it would be a reasonable approach here for the same reasons.

Bryan

That's not a bad idea. Matplotlib is currently considering something similar for its mlab module. It has been there since the beginning, but it is very outdated and very out-of-scope for matplotlib. However, there are still lots of code out there that depends on it. So, we are looking to split it off as its own package. The details still need to be worked out (should we initially depend on the package and simply alias its import with a DeprecationWarning, or should we go cold turkey and have a good message explaining the change).
Ben Root

Post by Charles R Harris
Hi All,
Thoughts?
Chuck

What if the matrix class was split out into its own project, perhaps as a scikit. That way those who really need it can still use it. If there is sufficient desire for it, those who need it can maintain it. If not, it will hopefully it will take long enough for it to bitrot that everyone has transitioned.
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Ralf Gommers]

Post by Bryan Van de Ven
There's a good chance that bokeh.charts will be split off into a
separately distributed package as well. Hopefully being a much smaller,
pure Python project makes it a more accessible target for anyone interested
in maintaining it, and if no one is interested in it anymore, well that
fact becomes easier to judge. I think it would be a reasonable approach
here for the same reasons.
Bryan
That's not a bad idea. Matplotlib is currently considering something
similar for its mlab module. It has been there since the beginning, but it
is very outdated and very out-of-scope for matplotlib. However, there are
still lots of code out there that depends on it. So, we are looking to
split it off as its own package. The details still need to be worked out
(should we initially depend on the package and simply alias its import with
a DeprecationWarning, or should we go cold turkey and have a good message
explaining the change).
Don't go cold turkey please, that still would break a lot of code. Even

with a good message, breaking things isn't great.

Post by Bryan Van de Ven
Ben Root

On Mon, Jan 2, 2017 at 8:36 PM, Charles R Harris <

Post by Charles R Harris
Hi All,
operator is available and things seem to be moving towards Python 3,
especially in the classroom, we should consider the real possibility of
deprecating the matrix type and later removing it. No doubt there are old
scripts that require them, but older versions of numpy are available for
those who need to run old scripts.
Thoughts?
Chuck

What if the matrix class was split out into its own project, perhaps as a
scikit.

Something like "npmatrix" would be a better name, we'd like to keep

scikit- for active well-maintained projects I'd think.

Post by Bryan Van de Ven
That way those who really need it can still use it. If there is

sufficient desire for it, those who need it can maintain it. If not, it
will hopefully it will take long enough for it to bitrot that everyone has
transitioned.

This sounds like a reasonable idea. Timeline could be something like:

1. Now: create new package, deprecate np.matrix in docs.
2. In say 1.5 years: start issuing visible deprecation warnings in numpy
3. After 2020: remove matrix from numpy.

Ralf

[CJ Carey]

Post by Ralf Gommers
1. Now: create new package, deprecate np.matrix in docs.
2. In say 1.5 years: start issuing visible deprecation warnings in numpy
3. After 2020: remove matrix from numpy.
Ralf

I think this sounds reasonable, and reminds me of the deliberate
deprecation process taken for scipy.weave. I guess we'll see how successful
it was when 0.19 is released.

The major problem I have with removing numpy matrices is the effect on
scipy.sparse, which mostly-consistently mimics numpy.matrix semantics and
often produces numpy.matrix results when densifying. The two are coupled
tightly enough that if numpy matrices go away, all of the existing sparse
matrix classes will have to go at the same time.

I don't think that would be the end of the world, but it's definitely
something that should happen while scipy is still pre-1.0, if it's ever
going to happen.

[Ralf Gommers]

Post by CJ Carey

Post by Ralf Gommers
1. Now: create new package, deprecate np.matrix in docs.
2. In say 1.5 years: start issuing visible deprecation warnings in numpy
3. After 2020: remove matrix from numpy.
Ralf

I think this sounds reasonable, and reminds me of the deliberate
deprecation process taken for scipy.weave. I guess we'll see how successful
it was when 0.19 is released.
The major problem I have with removing numpy matrices is the effect on
scipy.sparse, which mostly-consistently mimics numpy.matrix semantics and
often produces numpy.matrix results when densifying. The two are coupled
tightly enough that if numpy matrices go away, all of the existing sparse
matrix classes will have to go at the same time.
I don't think that would be the end of the world,

Not the end of the world literally, but the impact would be pretty major. I
think we're stuck with scipy.sparse, and may at some point will add a new
sparse *array* implementation next to it. For scipy we will have to add a
dependency on the new npmatrix package or vendor it.

Ralf

Post by CJ Carey
but it's definitely something that should happen while scipy is still
pre-1.0, if it's ever going to happen.
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[j***@gmail.com]

Post by Ralf Gommers

Post by CJ Carey

Post by Ralf Gommers
1. Now: create new package, deprecate np.matrix in docs.
2. In say 1.5 years: start issuing visible deprecation warnings in numpy
3. After 2020: remove matrix from numpy.
Ralf

I think this sounds reasonable, and reminds me of the deliberate
deprecation process taken for scipy.weave. I guess we'll see how successful
it was when 0.19 is released.
The major problem I have with removing numpy matrices is the effect on
scipy.sparse, which mostly-consistently mimics numpy.matrix semantics and
often produces numpy.matrix results when densifying. The two are coupled
tightly enough that if numpy matrices go away, all of the existing sparse
matrix classes will have to go at the same time.
I don't think that would be the end of the world,

Not the end of the world literally, but the impact would be pretty major.
I think we're stuck with scipy.sparse, and may at some point will add a new
sparse *array* implementation next to it. For scipy we will have to add a
dependency on the new npmatrix package or vendor it.

That sounds to me like moving maintenance of numpy.matrix from numpy to
scipy, if scipy.sparse is one of the main users and still depends on it.

Josef

Post by Ralf Gommers
Ralf

Post by CJ Carey
but it's definitely something that should happen while scipy is still
pre-1.0, if it's ever going to happen.
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Charles R Harris]

Post by j***@gmail.com

Post by Ralf Gommers

Post by CJ Carey

Post by Ralf Gommers
1. Now: create new package, deprecate np.matrix in docs.
2. In say 1.5 years: start issuing visible deprecation warnings in numpy
3. After 2020: remove matrix from numpy.
Ralf

I think this sounds reasonable, and reminds me of the deliberate
deprecation process taken for scipy.weave. I guess we'll see how successful
it was when 0.19 is released.
The major problem I have with removing numpy matrices is the effect on
scipy.sparse, which mostly-consistently mimics numpy.matrix semantics and
often produces numpy.matrix results when densifying. The two are coupled
tightly enough that if numpy matrices go away, all of the existing sparse
matrix classes will have to go at the same time.
I don't think that would be the end of the world,

Not the end of the world literally, but the impact would be pretty major.
I think we're stuck with scipy.sparse, and may at some point will add a new
sparse *array* implementation next to it. For scipy we will have to add a
dependency on the new npmatrix package or vendor it.

That sounds to me like moving maintenance of numpy.matrix from numpy to
scipy, if scipy.sparse is one of the main users and still depends on it.

What I was thinking was encouraging folks to use `arr.dot(...)` or `@`
instead of `*` for matrix multiplication, keeping `*` for scalar
multiplication. If those operations were defined for matrices, then at some
point sparse could go to arrays and it would not be noticeable except for
the treatment of 1-D arrays -- which admittedly might be a bit tricky.

Chuck

[Ralf Gommers]

Post by Charles R Harris

Post by j***@gmail.com

Post by Ralf Gommers

Post by CJ Carey

Post by Ralf Gommers
1. Now: create new package, deprecate np.matrix in docs.
2. In say 1.5 years: start issuing visible deprecation warnings in numpy
3. After 2020: remove matrix from numpy.
Ralf

I think this sounds reasonable, and reminds me of the deliberate
deprecation process taken for scipy.weave. I guess we'll see how successful
it was when 0.19 is released.
The major problem I have with removing numpy matrices is the effect on
scipy.sparse, which mostly-consistently mimics numpy.matrix semantics and
often produces numpy.matrix results when densifying. The two are coupled
tightly enough that if numpy matrices go away, all of the existing sparse
matrix classes will have to go at the same time.
I don't think that would be the end of the world,

Not the end of the world literally, but the impact would be pretty
major. I think we're stuck with scipy.sparse, and may at some point will
add a new sparse *array* implementation next to it. For scipy we will have
to add a dependency on the new npmatrix package or vendor it.

That sounds to me like moving maintenance of numpy.matrix from numpy to
scipy, if scipy.sparse is one of the main users and still depends on it.

Maintenance costs are pretty low, and are partly still for numpy (it has to
keep subclasses like np.matrix working. I'm not too worried about the
effort. The purpose here is to remove np.matrix from numpy so beginners
will never see it. Educating sparse matrix users is a lot easier, and there
are a lot less such users.

Post by Charles R Harris
instead of `*` for matrix multiplication, keeping `*` for scalar
multiplication.

I don't think that change in behavior of `*` is doable.

Post by Charles R Harris
If those operations were defined for matrices,

Why if? They are defined, and work as expected as far as I can tell.

Post by Charles R Harris
then at some point sparse could go to arrays and it would not be
noticeable except for the treatment of 1-D arrays -- which admittedly might
be a bit tricky.

I'd like that to be feasible, but especially given that any such change
would not break code but rather silently change numerical values, it's
likely not a healthy idea.

Ralf

[Nathaniel Smith]

Post by Ralf Gommers

Post by Charles R Harris

Post by j***@gmail.com

Post by Ralf Gommers

Post by CJ Carey

Post by Ralf Gommers
1. Now: create new package, deprecate np.matrix in docs.
2. In say 1.5 years: start issuing visible deprecation warnings in numpy
3. After 2020: remove matrix from numpy.
Ralf

I think this sounds reasonable, and reminds me of the deliberate
deprecation process taken for scipy.weave. I guess we'll see how successful
it was when 0.19 is released.
The major problem I have with removing numpy matrices is the effect on
scipy.sparse, which mostly-consistently mimics numpy.matrix semantics and
often produces numpy.matrix results when densifying. The two are coupled
tightly enough that if numpy matrices go away, all of the existing sparse
matrix classes will have to go at the same time.
I don't think that would be the end of the world,

Not the end of the world literally, but the impact would be pretty
major. I think we're stuck with scipy.sparse, and may at some point will add
a new sparse *array* implementation next to it. For scipy we will have to
add a dependency on the new npmatrix package or vendor it.

That sounds to me like moving maintenance of numpy.matrix from numpy to
scipy, if scipy.sparse is one of the main users and still depends on it.

Maintenance costs are pretty low, and are partly still for numpy (it has to
keep subclasses like np.matrix working. I'm not too worried about the
effort. The purpose here is to remove np.matrix from numpy so beginners will
never see it. Educating sparse matrix users is a lot easier, and there are a
lot less such users.

Post by Charles R Harris
instead of `*` for matrix multiplication, keeping `*` for scalar
multiplication.

I don't think that change in behavior of `*` is doable.

I guess it would be technically possible to have matrix.__mul__ issue
a deprecation warning before matrix.__init__ does, to try and
encourage people to switch to using .dot and/or @, and thus make it
easier to later port their code to regular arrays? I'm not immediately
seeing how this would help much though, since there would still be
this second porting step required. Especially since there's still lots
of room for things to break at that second step due to matrix's
insistence that everything be 2d always, and my impression is that
users are more annoyed by two-step migrations than one-step
migrations.

-n

[Ralf Gommers]

Post by CJ Carey

On Sat, Jan 7, 2017 at 2:52 PM, Charles R Harris <

Post by Charles R Harris

Post by j***@gmail.com

Post by Ralf Gommers

Post by CJ Carey

Post by Ralf Gommers
This sounds like a reasonable idea. Timeline could be something
1. Now: create new package, deprecate np.matrix in docs.
2. In say 1.5 years: start issuing visible deprecation warnings in numpy
3. After 2020: remove matrix from numpy.
Ralf

I think this sounds reasonable, and reminds me of the deliberate
deprecation process taken for scipy.weave. I guess we'll see how

successful

Post by Charles R Harris

Post by j***@gmail.com

Post by Ralf Gommers

Post by CJ Carey
it was when 0.19 is released.
The major problem I have with removing numpy matrices is the effect

on

Post by Charles R Harris

Post by j***@gmail.com

Post by Ralf Gommers

Post by CJ Carey
scipy.sparse, which mostly-consistently mimics numpy.matrix

semantics and

Post by Charles R Harris

Post by j***@gmail.com

Post by Ralf Gommers

Post by CJ Carey
often produces numpy.matrix results when densifying. The two are

coupled

Post by Charles R Harris

Post by j***@gmail.com

Post by Ralf Gommers

Post by CJ Carey
tightly enough that if numpy matrices go away, all of the existing

sparse

Post by Charles R Harris

Post by j***@gmail.com

Post by Ralf Gommers

Post by CJ Carey
matrix classes will have to go at the same time.
I don't think that would be the end of the world,

Not the end of the world literally, but the impact would be pretty
major. I think we're stuck with scipy.sparse, and may at some point

will add

Post by Charles R Harris

Post by j***@gmail.com

Post by Ralf Gommers
a new sparse *array* implementation next to it. For scipy we will

have to

Post by Charles R Harris

Post by j***@gmail.com

Post by Ralf Gommers
add a dependency on the new npmatrix package or vendor it.

That sounds to me like moving maintenance of numpy.matrix from numpy to
scipy, if scipy.sparse is one of the main users and still depends on

it.

Maintenance costs are pretty low, and are partly still for numpy (it has

to

keep subclasses like np.matrix working. I'm not too worried about the
effort. The purpose here is to remove np.matrix from numpy so beginners

will

never see it. Educating sparse matrix users is a lot easier, and there

are a

lot less such users.

Post by Charles R Harris
instead of `*` for matrix multiplication, keeping `*` for scalar
multiplication.

I don't think that change in behavior of `*` is doable.

I guess it would be technically possible to have matrix.__mul__ issue
a deprecation warning before matrix.__init__ does, to try and
easier to later port their code to regular arrays?

Yes, but that's not very relevant. I'm saying "not doable" since after the
debacle with changing diag return to a view my understanding is we decided
that it's a bad idea to make changes that don't break code but return
different numerical results. There's no good way to work around that here.

With something as widely used as np.matrix, you simply cannot rely on
people porting code. You just need to phase out np.matrix in a way that
breaks code but never changes behavior silently (even across multiple
releases).

Ralf

[Juan Nunez-Iglesias]

Hi all! I've been lurking on this discussion, and don't have too much to add except to encourage a fast deprecation: I can't wait for sparse matrices to have an element-wise multiply operator.

Post by Nathaniel Smith

Post by Ralf Gommers

Post by Charles R Harris

Post by j***@gmail.com

Post by Ralf Gommers

Post by CJ Carey

Post by Ralf Gommers
1. Now: create new package, deprecate np.matrix in docs.
2. In say 1.5 years: start issuing visible deprecation warnings in numpy
3. After 2020: remove matrix from numpy.
Ralf

I think this sounds reasonable, and reminds me of the deliberate
deprecation process taken for scipy.weave. I guess we'll see how successful
it was when 0.19 is released.
The major problem I have with removing numpy matrices is the effect on
scipy.sparse, which mostly-consistently mimics numpy.matrix semantics and
often produces numpy.matrix results when densifying. The two are coupled
tightly enough that if numpy matrices go away, all of the existing sparse
matrix classes will have to go at the same time.
I don't think that would be the end of the world,

Not the end of the world literally, but the impact would be pretty
major. I think we're stuck with scipy.sparse, and may at some point will add
a new sparse *array* implementation next to it. For scipy we will have to
add a dependency on the new npmatrix package or vendor it.

That sounds to me like moving maintenance of numpy.matrix from numpy to
scipy, if scipy.sparse is one of the main users and still depends on it.

Maintenance costs are pretty low, and are partly still for numpy (it has to
keep subclasses like np.matrix working. I'm not too worried about the
effort. The purpose here is to remove np.matrix from numpy so beginners will
never see it. Educating sparse matrix users is a lot easier, and there are a
lot less such users.

Post by Charles R Harris
instead of `*` for matrix multiplication, keeping `*` for scalar
multiplication.

I don't think that change in behavior of `*` is doable.

I guess it would be technically possible to have matrix.__mul__ issue
a deprecation warning before matrix.__init__ does, to try and
easier to later port their code to regular arrays?

Yes, but that's not very relevant. I'm saying "not doable" since after the debacle with changing diag return to a view my understanding is we decided that it's a bad idea to make changes that don't break code but return different numerical results. There's no good way to work around that here.
With something as widely used as np.matrix, you simply cannot rely on people porting code. You just need to phase out np.matrix in a way that breaks code but never changes behavior silently (even across multiple releases).
Ralf
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Marten van Kerkwijk]

Hi All,

It seems there are two steps that can be taken now and are needed no
matter what:

1. Add numpy documentation describing the preferred way to handle
matrices, extolling the virtues of @, and move np.matrix documentation
to a deprecated section

2. Start on a new `sparse` class that is based on regular arrays (and
uses `__array_func__` instead of prepare/wrap?).

All the best,

Marten

[Ralf Gommers]

On Sun, Jan 8, 2017 at 8:33 AM, Marten van Kerkwijk <

Post by Charles R Harris
Hi All,
It seems there are two steps that can be taken now and are needed no
1. Add numpy documentation describing the preferred way to handle
to a deprecated section

That would be good to do asap. Any volunteers?

Post by Charles R Harris
2. Start on a new `sparse` class that is based on regular arrays

There are two efforts that I know of in this direction:
https://github.com/perimosocordiae/sparray
https://github.com/ev-br/sparr

(and

Post by Charles R Harris
uses `__array_func__` instead of prepare/wrap?).

Getting __array_func__ finally into a released version of numpy will be a
major improvement for sparse matrix behavior (like making
np.dot(some_matrix) work) in itself.

Ralf

[Todd]

Post by CJ Carey

Post by Ralf Gommers
1. Now: create new package, deprecate np.matrix in docs.
2. In say 1.5 years: start issuing visible deprecation warnings in numpy
3. After 2020: remove matrix from numpy.
Ralf

I think this sounds reasonable, and reminds me of the deliberate
deprecation process taken for scipy.weave. I guess we'll see how successful
it was when 0.19 is released.
The major problem I have with removing numpy matrices is the effect on
scipy.sparse, which mostly-consistently mimics numpy.matrix semantics and
often produces numpy.matrix results when densifying. The two are coupled
tightly enough that if numpy matrices go away, all of the existing sparse
matrix classes will have to go at the same time.
I don't think that would be the end of the world,

Not the end of the world literally, but the impact would be pretty major. I
think we're stuck with scipy.sparse, and may at some point will add a new
sparse *array* implementation next to it. For scipy we will have to add a
dependency on the new npmatrix package or vendor it.

Ralf

Post by CJ Carey
but it's definitely something that should happen while scipy is still
pre-1.0, if it's ever going to happen.
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
NumPy-***@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion


So what about this:

1. Create a sparse array class
2. (optional) Refactor the sparse matrix class to be based on the sparse
array class (may not be feasible)
3. Copy the spare matrix class into the matrix package
4. Deprecate the scipy sparse matrix class
5. Remove the scipy sparse matrix class when the numpy matrix class

I don't know about the timeline, but this would just need to be done by
2020.

[Ralf Gommers]

Post by Ralf Gommers

Post by CJ Carey

Post by Ralf Gommers
1. Now: create new package, deprecate np.matrix in docs.
2. In say 1.5 years: start issuing visible deprecation warnings in numpy
3. After 2020: remove matrix from numpy.
Ralf

I think this sounds reasonable, and reminds me of the deliberate
deprecation process taken for scipy.weave. I guess we'll see how successful
it was when 0.19 is released.
The major problem I have with removing numpy matrices is the effect on
scipy.sparse, which mostly-consistently mimics numpy.matrix semantics and
often produces numpy.matrix results when densifying. The two are coupled
tightly enough that if numpy matrices go away, all of the existing sparse
matrix classes will have to go at the same time.
I don't think that would be the end of the world,

Not the end of the world literally, but the impact would be pretty major.
I think we're stuck with scipy.sparse, and may at some point will add a new
sparse *array* implementation next to it. For scipy we will have to add a
dependency on the new npmatrix package or vendor it.
Ralf

Post by CJ Carey
but it's definitely something that should happen while scipy is still
pre-1.0, if it's ever going to happen.
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
1. Create a sparse array class
2. (optional) Refactor the sparse matrix class to be based on the sparse
array class (may not be feasible)
3. Copy the spare matrix class into the matrix package
4. Deprecate the scipy sparse matrix class
5. Remove the scipy sparse matrix class when the numpy matrix class

It looks to me like we're getting a bit off track here. The sparse matrices
in scipy are heavily used, and despite rough edges pretty good at what they
do. Deprecating them is not a goal.

The actual goal for the exercise that started this thread (at least as I
see it) is to remove np.matrix from numpy itself so users (that don't know
the difference) will only use ndarrays. And the few users that prefer
np.matrix for teaching can now switch because of @, so their preference
should have disappeared.

To reach that goal, no deprecation or backwards incompatible changes to
scipy.sparse are needed.

Ralf

[Charles R Harris]

Post by Ralf Gommers

Post by Ralf Gommers

Post by CJ Carey

Post by Ralf Gommers
1. Now: create new package, deprecate np.matrix in docs.
2. In say 1.5 years: start issuing visible deprecation warnings in numpy
3. After 2020: remove matrix from numpy.
Ralf

I think this sounds reasonable, and reminds me of the deliberate
deprecation process taken for scipy.weave. I guess we'll see how successful
it was when 0.19 is released.
The major problem I have with removing numpy matrices is the effect on
scipy.sparse, which mostly-consistently mimics numpy.matrix semantics and
often produces numpy.matrix results when densifying. The two are coupled
tightly enough that if numpy matrices go away, all of the existing sparse
matrix classes will have to go at the same time.
I don't think that would be the end of the world,

Not the end of the world literally, but the impact would be pretty major.
I think we're stuck with scipy.sparse, and may at some point will add a new
sparse *array* implementation next to it. For scipy we will have to add a
dependency on the new npmatrix package or vendor it.
Ralf

Post by CJ Carey
but it's definitely something that should happen while scipy is still
pre-1.0, if it's ever going to happen.
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
1. Create a sparse array class
2. (optional) Refactor the sparse matrix class to be based on the sparse
array class (may not be feasible)
3. Copy the spare matrix class into the matrix package
4. Deprecate the scipy sparse matrix class
5. Remove the scipy sparse matrix class when the numpy matrix class

It looks to me like we're getting a bit off track here. The sparse
matrices in scipy are heavily used, and despite rough edges pretty good at
what they do. Deprecating them is not a goal.
The actual goal for the exercise that started this thread (at least as I
see it) is to remove np.matrix from numpy itself so users (that don't know
the difference) will only use ndarrays. And the few users that prefer
should have disappeared.
To reach that goal, no deprecation or backwards incompatible changes to
scipy.sparse are needed.

What is the way forward with sparse? That looks like the biggest blocker on
the road to a matrix free NumPy. I don't see moving the matrix package
elsewhere as a solution for that.

Chuck

[Ralf Gommers]

Post by Charles R Harris

Post by Ralf Gommers
It looks to me like we're getting a bit off track here. The sparse
matrices in scipy are heavily used, and despite rough edges pretty good at
what they do. Deprecating them is not a goal.
The actual goal for the exercise that started this thread (at least as I
see it) is to remove np.matrix from numpy itself so users (that don't know
the difference) will only use ndarrays. And the few users that prefer
should have disappeared.
To reach that goal, no deprecation or backwards incompatible changes to
scipy.sparse are needed.

What is the way forward with sparse? That looks like the biggest blocker
on the road to a matrix free NumPy. I don't see moving the matrix package
elsewhere as a solution for that.

Why not?

Ralf

[Charles R Harris]

On Sun, Jan 8, 2017 at 12:26 PM, Charles R Harris <

Post by Charles R Harris

Post by Ralf Gommers
It looks to me like we're getting a bit off track here. The sparse
matrices in scipy are heavily used, and despite rough edges pretty good at
what they do. Deprecating them is not a goal.
The actual goal for the exercise that started this thread (at least as I
see it) is to remove np.matrix from numpy itself so users (that don't know
the difference) will only use ndarrays. And the few users that prefer
should have disappeared.
To reach that goal, no deprecation or backwards incompatible changes to
scipy.sparse are needed.

What is the way forward with sparse? That looks like the biggest blocker
on the road to a matrix free NumPy. I don't see moving the matrix package
elsewhere as a solution for that.

Why not?

Because it doesn't get rid of matrices in SciPy, not does one gain a scalar
multiplication operator for sparse.

Chuck

[Ralf Gommers]

Post by Charles R Harris

On Sun, Jan 8, 2017 at 12:26 PM, Charles R Harris <

Post by Charles R Harris

Post by Ralf Gommers
It looks to me like we're getting a bit off track here. The sparse
matrices in scipy are heavily used, and despite rough edges pretty good at
what they do. Deprecating them is not a goal.
The actual goal for the exercise that started this thread (at least as
I see it) is to remove np.matrix from numpy itself so users (that don't
know the difference) will only use ndarrays. And the few users that prefer
should have disappeared.
To reach that goal, no deprecation or backwards incompatible changes to
scipy.sparse are needed.

What is the way forward with sparse? That looks like the biggest blocker
on the road to a matrix free NumPy. I don't see moving the matrix package
elsewhere as a solution for that.

Why not?

Because it doesn't get rid of matrices in SciPy, not does one gain a
scalar multiplication operator for sparse.

That's a different goal though. You can reach the "get matrix out of numpy"
goal fairly easily (docs and packaging work), but if you insist on coupling
it to major changes to scipy.sparse (a lot more work + backwards compat
break), then what will likely happen is: nothing.

Ralf

[Charles R Harris]

On Sun, Jan 8, 2017 at 12:42 PM, Charles R Harris <

Post by Charles R Harris

On Sun, Jan 8, 2017 at 12:26 PM, Charles R Harris <

Post by Charles R Harris

Post by Ralf Gommers
It looks to me like we're getting a bit off track here. The sparse
matrices in scipy are heavily used, and despite rough edges pretty good at
what they do. Deprecating them is not a goal.
The actual goal for the exercise that started this thread (at least as
I see it) is to remove np.matrix from numpy itself so users (that don't
know the difference) will only use ndarrays. And the few users that prefer
should have disappeared.
To reach that goal, no deprecation or backwards incompatible changes
to scipy.sparse are needed.

What is the way forward with sparse? That looks like the biggest
blocker on the road to a matrix free NumPy. I don't see moving the matrix
package elsewhere as a solution for that.

Why not?

Because it doesn't get rid of matrices in SciPy, not does one gain a
scalar multiplication operator for sparse.

That's a different goal though. You can reach the "get matrix out of
numpy" goal fairly easily (docs and packaging work), but if you insist on
coupling it to major changes to scipy.sparse (a lot more work + backwards
compat break), then what will likely happen is: nothing.

Could always remove matrix from the top level namespace and make it
private. It still needs to reside someplace as long as sparse uses it.
Fixing sparse is more work, but we have three years and it won't be getting
any easier as time goes on.

Chuck

[CJ Carey]

I agree with Ralf; coupling these changes to sparse is a bad idea.

I think that scipy.sparse will be an important consideration during the
deprecation process, though, perhaps as an indicator of how painful the
transition might be for third party code.

I'm +1 for splitting matrices out into a standalone package.

Post by Charles R Harris

On Sun, Jan 8, 2017 at 12:26 PM, Charles R Harris <

Post by Charles R Harris

Post by Ralf Gommers
It looks to me like we're getting a bit off track here. The sparse
matrices in scipy are heavily used, and despite rough edges pretty good at
what they do. Deprecating them is not a goal.
The actual goal for the exercise that started this thread (at least as
I see it) is to remove np.matrix from numpy itself so users (that don't
know the difference) will only use ndarrays. And the few users that prefer
should have disappeared.
To reach that goal, no deprecation or backwards incompatible changes to
scipy.sparse are needed.

What is the way forward with sparse? That looks like the biggest blocker
on the road to a matrix free NumPy. I don't see moving the matrix package
elsewhere as a solution for that.

Why not?

Because it doesn't get rid of matrices in SciPy, not does one gain a
scalar multiplication operator for sparse.

That's a different goal though. You can reach the "get matrix out of numpy"
goal fairly easily (docs and packaging work), but if you insist on coupling
it to major changes to scipy.sparse (a lot more work + backwards compat
break), then what will likely happen is: nothing.

Ralf

[Charles R Harris]

Post by CJ Carey
I agree with Ralf; coupling these changes to sparse is a bad idea.
I think that scipy.sparse will be an important consideration during the
deprecation process, though, perhaps as an indicator of how painful the
transition might be for third party code.
I'm +1 for splitting matrices out into a standalone package.

Decoupled or not, sparse still needs to be dealt with. What is the plan?

<snip>

Chuck

[Ralf Gommers]

Post by Charles R Harris

Post by CJ Carey
I agree with Ralf; coupling these changes to sparse is a bad idea.
I think that scipy.sparse will be an important consideration during the
deprecation process, though, perhaps as an indicator of how painful the
transition might be for third party code.
I'm +1 for splitting matrices out into a standalone package.

Decoupled or not, sparse still needs to be dealt with. What is the plan?

My view would be:
- keep current sparse matrices as is (with improvements, like
__numpy_func__ and the various performance improvements that regularly get
done)
- once one of the sparse *array* implementations progresses far enough,
merge that and encourage people to switch over
- in the far future, once packages like scikit-learn have switched to the
new sparse arrays, the sparse matrices could potentially also be split off
as a separate package, in the same way as we did for weave and now can do
for npmatrix.

Ralf

[CJ Carey]

Post by Charles R Harris
Decoupled or not, sparse still needs to be dealt with. What is the plan?

My view would be:
- keep current sparse matrices as is (with improvements, like
__numpy_func__ and the various performance improvements that regularly get
done)
- once one of the sparse *array* implementations progresses far enough,
merge that and encourage people to switch over
- in the far future, once packages like scikit-learn have switched to the
new sparse arrays, the sparse matrices could potentially also be split off
as a separate package, in the same way as we did for weave and now can do
for npmatrix.


I think that's the best way forward as well. This can happen independently
of numpy matrix changes, and doesn't leave users with silently broken code.