Discussion:
[Numpy-discussion] Generalize hstack/vstack --> stack; Block matrices like in matlab
Stefan Otte
2014-09-08 13:29:08 UTC
Permalink
Hey,

quite often I work with block matrices. Matlab offers the convenient notation

[ a b; c d ]

to stack matrices. The numpy equivalent is kinda clumsy:

vstack([hstack([a,b]), hstack([c,d])])

I wrote the little function `stack` that does exactly that:

stack([[a, b], [c, d]])

In my case `stack` replaced `hstack` and `vstack` almost completely.

If you're interested in including it in numpy I created a pull request
[1]. I'm looking forward to getting some feedback!


Best,
Stefan



[1] https://github.com/numpy/numpy/pull/5057
Sturla Molden
2014-09-08 14:41:57 UTC
Permalink
Post by Stefan Otte
stack([[a, b], [c, d]])
In my case `stack` replaced `hstack` and `vstack` almost completely.
If you're interested in including it in numpy I created a pull request
[1]. I'm looking forward to getting some feedback!
As far as I can see, it uses hstack and vstack. But that means a and b have
to have the same number of rows, c and d must have the same rumber of rows,
and hstack((a,b)) and hstack((c,d)) must have the same number of columns.

Thus it requires a regularity like this:

AAAABB
AAAABB
CCCDDD
CCCDDD
CCCDDD
CCCDDD

What if we just ignore this constraint, and only require the output to be
rectangular? Now we have a 'tetris game':

AAAABB
AAAABB
CCCCBB
CCCCBB
CCCCDD
CCCCDD

or

AAAABB
AAAABB
CCCCBB
CCCCBB
CCCCBB
CCCCBB

This should be 'stackable', yes? Or perhaps we need another stacking
function for this, say numpy.tetris?

And while we're at it, what about higher dimensions? should there be an
ndstack function too?


Sturla
j***@gmail.com
2014-09-08 16:08:12 UTC
Permalink
Post by Sturla Molden
Post by Stefan Otte
stack([[a, b], [c, d]])
In my case `stack` replaced `hstack` and `vstack` almost completely.
If you're interested in including it in numpy I created a pull request
[1]. I'm looking forward to getting some feedback!
np.asarray(np.bmat(....)) ?

Josef
Post by Sturla Molden
As far as I can see, it uses hstack and vstack. But that means a and b have
to have the same number of rows, c and d must have the same rumber of rows,
and hstack((a,b)) and hstack((c,d)) must have the same number of columns.
AAAABB
AAAABB
CCCDDD
CCCDDD
CCCDDD
CCCDDD
What if we just ignore this constraint, and only require the output to be
AAAABB
AAAABB
CCCCBB
CCCCBB
CCCCDD
CCCCDD
or
AAAABB
AAAABB
CCCCBB
CCCCBB
CCCCBB
CCCCBB
This should be 'stackable', yes? Or perhaps we need another stacking
function for this, say numpy.tetris?
And while we're at it, what about higher dimensions? should there be an
ndstack function too?
Sturla
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Jaime Fernández del Río
2014-09-08 16:10:35 UTC
Permalink
Post by Sturla Molden
Post by Stefan Otte
stack([[a, b], [c, d]])
In my case `stack` replaced `hstack` and `vstack` almost completely.
If you're interested in including it in numpy I created a pull request
[1]. I'm looking forward to getting some feedback!
As far as I can see, it uses hstack and vstack. But that means a and b have
to have the same number of rows, c and d must have the same rumber of rows,
and hstack((a,b)) and hstack((c,d)) must have the same number of columns.
AAAABB
AAAABB
CCCDDD
CCCDDD
CCCDDD
CCCDDD
What if we just ignore this constraint, and only require the output to be
AAAABB
AAAABB
CCCCBB
CCCCBB
CCCCDD
CCCCDD
or
AAAABB
AAAABB
CCCCBB
CCCCBB
CCCCBB
CCCCBB
This should be 'stackable', yes? Or perhaps we need another stacking
function for this, say numpy.tetris?
And while we're at it, what about higher dimensions? should there be an
ndstack function too?
This is starting to look like the second time in a row Stefan tries to
extend numpy with a simple convenience function, and he gets tricked into
implementing some sophisticated algorithm...

For his next PR I expect nothing less than an NP-complete problem. ;-)
Post by Sturla Molden
Jaime
--
(\__/)
( O.o)
( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes
de dominación mundial.
j***@gmail.com
2014-09-08 16:18:25 UTC
Permalink
On Mon, Sep 8, 2014 at 12:10 PM, Jaime Fernández del Río <
Post by Jaime Fernández del Río
Post by Sturla Molden
Post by Stefan Otte
stack([[a, b], [c, d]])
In my case `stack` replaced `hstack` and `vstack` almost completely.
If you're interested in including it in numpy I created a pull request
[1]. I'm looking forward to getting some feedback!
As far as I can see, it uses hstack and vstack. But that means a and b have
to have the same number of rows, c and d must have the same rumber of rows,
and hstack((a,b)) and hstack((c,d)) must have the same number of columns.
AAAABB
AAAABB
CCCDDD
CCCDDD
CCCDDD
CCCDDD
What if we just ignore this constraint, and only require the output to be
AAAABB
AAAABB
CCCCBB
CCCCBB
CCCCDD
CCCCDD
or
AAAABB
AAAABB
CCCCBB
CCCCBB
CCCCBB
CCCCBB
This should be 'stackable', yes? Or perhaps we need another stacking
function for this, say numpy.tetris?
And while we're at it, what about higher dimensions? should there be an
ndstack function too?
This is starting to look like the second time in a row Stefan tries to
extend numpy with a simple convenience function, and he gets tricked into
implementing some sophisticated algorithm...
Maybe the third time is a gem.
Post by Jaime Fernández del Río
For his next PR I expect nothing less than an NP-complete problem. ;-)
How about cholesky or qr updating? I could use one right now.

Josef
Post by Jaime Fernández del Río
Post by Sturla Molden
Jaime
--
(\__/)
( O.o)
( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes
de dominación mundial.
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Eelco Hoogendoorn
2014-09-08 16:41:58 UTC
Permalink
Sturla: im not sure if the intention is always unambiguous, for such more
flexible arrangements.

Also, I doubt such situations arise often in practice; if the arrays arnt a
grid, they are probably a nested grid, and the code would most naturally
concatenate them with nested calls to a stacking function.

However, some form of nd-stack function would be neat in my opinion.

On Mon, Sep 8, 2014 at 6:10 PM, Jaime Fernández del Río <
Post by Jaime Fernández del Río
Post by Sturla Molden
Post by Stefan Otte
stack([[a, b], [c, d]])
In my case `stack` replaced `hstack` and `vstack` almost completely.
If you're interested in including it in numpy I created a pull request
[1]. I'm looking forward to getting some feedback!
As far as I can see, it uses hstack and vstack. But that means a and b have
to have the same number of rows, c and d must have the same rumber of rows,
and hstack((a,b)) and hstack((c,d)) must have the same number of columns.
AAAABB
AAAABB
CCCDDD
CCCDDD
CCCDDD
CCCDDD
What if we just ignore this constraint, and only require the output to be
AAAABB
AAAABB
CCCCBB
CCCCBB
CCCCDD
CCCCDD
or
AAAABB
AAAABB
CCCCBB
CCCCBB
CCCCBB
CCCCBB
This should be 'stackable', yes? Or perhaps we need another stacking
function for this, say numpy.tetris?
And while we're at it, what about higher dimensions? should there be an
ndstack function too?
This is starting to look like the second time in a row Stefan tries to
extend numpy with a simple convenience function, and he gets tricked into
implementing some sophisticated algorithm...
For his next PR I expect nothing less than an NP-complete problem. ;-)
Post by Sturla Molden
Jaime
--
(\__/)
( O.o)
( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes
de dominación mundial.
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Benjamin Root
2014-09-08 16:55:34 UTC
Permalink
A use case would be "image stitching" or even data tiling. I have had to
implement something like this at work (so, I can't share it, unfortunately)
and it even goes so far as to allow the caller to specify how much the
tiles can overlap and such. The specification is ungodly hideous and I
doubt I would be willing to share it even if I could lest I release
code-thulu upon the world...

I think just having this generalize stack feature would be nice start.
Tetris could be built on top of that later. (Although, I do vote for at
least 3 or 4 dimensional stacking, if possible).

Cheers!
Ben Root


On Mon, Sep 8, 2014 at 12:41 PM, Eelco Hoogendoorn <
Post by Eelco Hoogendoorn
Sturla: im not sure if the intention is always unambiguous, for such more
flexible arrangements.
Also, I doubt such situations arise often in practice; if the arrays arnt
a grid, they are probably a nested grid, and the code would most naturally
concatenate them with nested calls to a stacking function.
However, some form of nd-stack function would be neat in my opinion.
On Mon, Sep 8, 2014 at 6:10 PM, Jaime Fernández del Río <
Post by Jaime Fernández del Río
Post by Sturla Molden
Post by Stefan Otte
stack([[a, b], [c, d]])
In my case `stack` replaced `hstack` and `vstack` almost completely.
If you're interested in including it in numpy I created a pull request
[1]. I'm looking forward to getting some feedback!
As far as I can see, it uses hstack and vstack. But that means a and b have
to have the same number of rows, c and d must have the same rumber of rows,
and hstack((a,b)) and hstack((c,d)) must have the same number of columns.
AAAABB
AAAABB
CCCDDD
CCCDDD
CCCDDD
CCCDDD
What if we just ignore this constraint, and only require the output to be
AAAABB
AAAABB
CCCCBB
CCCCBB
CCCCDD
CCCCDD
or
AAAABB
AAAABB
CCCCBB
CCCCBB
CCCCBB
CCCCBB
This should be 'stackable', yes? Or perhaps we need another stacking
function for this, say numpy.tetris?
And while we're at it, what about higher dimensions? should there be an
ndstack function too?
This is starting to look like the second time in a row Stefan tries to
extend numpy with a simple convenience function, and he gets tricked into
implementing some sophisticated algorithm...
For his next PR I expect nothing less than an NP-complete problem. ;-)
Post by Sturla Molden
Jaime
--
(\__/)
( O.o)
( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes
de dominación mundial.
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Benjamin Root
2014-09-08 17:00:11 UTC
Permalink
Btw, on a somewhat related note, whoever can implement ndarray to be able
to use views from other ndarrays stitched together would get a fruit basket
from me come the holidays and possibly naming rights for the next kid...

Cheers!
Ben Root
Post by Benjamin Root
A use case would be "image stitching" or even data tiling. I have had to
implement something like this at work (so, I can't share it, unfortunately)
and it even goes so far as to allow the caller to specify how much the
tiles can overlap and such. The specification is ungodly hideous and I
doubt I would be willing to share it even if I could lest I release
code-thulu upon the world...
I think just having this generalize stack feature would be nice start.
Tetris could be built on top of that later. (Although, I do vote for at
least 3 or 4 dimensional stacking, if possible).
Cheers!
Ben Root
On Mon, Sep 8, 2014 at 12:41 PM, Eelco Hoogendoorn <
Post by Eelco Hoogendoorn
Sturla: im not sure if the intention is always unambiguous, for such more
flexible arrangements.
Also, I doubt such situations arise often in practice; if the arrays arnt
a grid, they are probably a nested grid, and the code would most naturally
concatenate them with nested calls to a stacking function.
However, some form of nd-stack function would be neat in my opinion.
On Mon, Sep 8, 2014 at 6:10 PM, Jaime Fernández del Río <
Post by Jaime Fernández del Río
Post by Sturla Molden
Post by Stefan Otte
stack([[a, b], [c, d]])
In my case `stack` replaced `hstack` and `vstack` almost completely.
If you're interested in including it in numpy I created a pull request
[1]. I'm looking forward to getting some feedback!
As far as I can see, it uses hstack and vstack. But that means a and b have
to have the same number of rows, c and d must have the same rumber of rows,
and hstack((a,b)) and hstack((c,d)) must have the same number of columns.
AAAABB
AAAABB
CCCDDD
CCCDDD
CCCDDD
CCCDDD
What if we just ignore this constraint, and only require the output to be
AAAABB
AAAABB
CCCCBB
CCCCBB
CCCCDD
CCCCDD
or
AAAABB
AAAABB
CCCCBB
CCCCBB
CCCCBB
CCCCBB
This should be 'stackable', yes? Or perhaps we need another stacking
function for this, say numpy.tetris?
And while we're at it, what about higher dimensions? should there be an
ndstack function too?
This is starting to look like the second time in a row Stefan tries to
extend numpy with a simple convenience function, and he gets tricked into
implementing some sophisticated algorithm...
For his next PR I expect nothing less than an NP-complete problem. ;-)
Post by Sturla Molden
Jaime
--
(\__/)
( O.o)
( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus
planes de dominación mundial.
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Stephan Hoyer
2014-09-08 17:18:45 UTC
Permalink
Post by Benjamin Root
Btw, on a somewhat related note, whoever can implement ndarray to be able
to use views from other ndarrays stitched together would get a fruit basket
from me come the holidays and possibly naming rights for the next kid...
Ben, you should check out Biggus, which does (at least some) of what you're
describing:
https://github.com/SciTools/biggus

Two things I would like to see before this makes it into numpy:
(1) It should handle arbitrary dimensional arrays, not just 2D.
(2) It needs to be more efficient. Composing vstack and hstack directly
makes a whole level of unnecessary intermediate copies.
Eelco Hoogendoorn
2014-09-08 17:37:31 UTC
Permalink
Blaze aims to do something like that; to make the notion of an array and how it stores it's data far more flexible. But if it isn't a single strided ND array, it isn't numpy. This concept lies at its very heart; and for good reasons I would add.

-----Original Message-----
From: "Benjamin Root" <***@ou.edu>
Sent: ‎8-‎9-‎2014 19:00
To: "Discussion of Numerical Python" <numpy-***@scipy.org>
Subject: Re: [Numpy-discussion] Generalize hstack/vstack --> stack; Blockmatrices like in matlab

Btw, on a somewhat related note, whoever can implement ndarray to be able to use views from other ndarrays stitched together would get a fruit basket from me come the holidays and possibly naming rights for the next kid...


Cheers!
Ben Root



On Mon, Sep 8, 2014 at 12:55 PM, Benjamin Root <***@ou.edu> wrote:

A use case would be "image stitching" or even data tiling. I have had to implement something like this at work (so, I can't share it, unfortunately) and it even goes so far as to allow the caller to specify how much the tiles can overlap and such. The specification is ungodly hideous and I doubt I would be willing to share it even if I could lest I release code-thulu upon the world...


I think just having this generalize stack feature would be nice start. Tetris could be built on top of that later. (Although, I do vote for at least 3 or 4 dimensional stacking, if possible).


Cheers!
Ben Root




On Mon, Sep 8, 2014 at 12:41 PM, Eelco Hoogendoorn <***@gmail.com> wrote:

Sturla: im not sure if the intention is always unambiguous, for such more flexible arrangements.

Also, I doubt such situations arise often in practice; if the arrays arnt a grid, they are probably a nested grid, and the code would most naturally concatenate them with nested calls to a stacking function.

However, some form of nd-stack function would be neat in my opinion.
Post by Stefan Otte
stack([[a, b], [c, d]])
In my case `stack` replaced `hstack` and `vstack` almost completely.
If you're interested in including it in numpy I created a pull request
[1]. I'm looking forward to getting some feedback!
As far as I can see, it uses hstack and vstack. But that means a and b have
to have the same number of rows, c and d must have the same rumber of rows,
and hstack((a,b)) and hstack((c,d)) must have the same number of columns.

Thus it requires a regularity like this:

AAAABB
AAAABB
CCCDDD
CCCDDD
CCCDDD
CCCDDD

What if we just ignore this constraint, and only require the output to be
rectangular? Now we have a 'tetris game':

AAAABB
AAAABB
CCCCBB
CCCCBB
CCCCDD
CCCCDD

or

AAAABB
AAAABB
CCCCBB
CCCCBB
CCCCBB
CCCCBB

This should be 'stackable', yes? Or perhaps we need another stacking
function for this, say numpy.tetris?

And while we're at it, what about higher dimensions? should there be an
ndstack function too?



This is starting to look like the second time in a row Stefan tries to extend numpy with a simple convenience function, and he gets tricked into implementing some sophisticated algorithm...


For his next PR I expect nothing less than an NP-complete problem. ;-)

Jaime
--
(\__/)
( O.o)
( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.
Paul Hobson
2014-09-19 14:47:46 UTC
Permalink
Hey Ben,

Side note: I've had to do the same thing for stitching curvilinear model
grid coordinates together. Usings pandas DataFrames indexed by `i` and `j`
is really good for this. You can offset the indices directly, unstack the
DF, and the pandas will align for you.

Happy to send an example along if you're curious.
-p
Post by Benjamin Root
A use case would be "image stitching" or even data tiling. I have had to
implement something like this at work (so, I can't share it, unfortunately)
and it even goes so far as to allow the caller to specify how much the
tiles can overlap and such. The specification is ungodly hideous and I
doubt I would be willing to share it even if I could lest I release
code-thulu upon the world...
I think just having this generalize stack feature would be nice start.
Tetris could be built on top of that later. (Although, I do vote for at
least 3 or 4 dimensional stacking, if possible).
Cheers!
Ben Root
On Mon, Sep 8, 2014 at 12:41 PM, Eelco Hoogendoorn <
Post by Eelco Hoogendoorn
Sturla: im not sure if the intention is always unambiguous, for such more
flexible arrangements.
Also, I doubt such situations arise often in practice; if the arrays arnt
a grid, they are probably a nested grid, and the code would most naturally
concatenate them with nested calls to a stacking function.
However, some form of nd-stack function would be neat in my opinion.
On Mon, Sep 8, 2014 at 6:10 PM, Jaime Fernández del Río <
Post by Jaime Fernández del Río
Post by Sturla Molden
Post by Stefan Otte
stack([[a, b], [c, d]])
In my case `stack` replaced `hstack` and `vstack` almost completely.
If you're interested in including it in numpy I created a pull request
[1]. I'm looking forward to getting some feedback!
As far as I can see, it uses hstack and vstack. But that means a and b have
to have the same number of rows, c and d must have the same rumber of rows,
and hstack((a,b)) and hstack((c,d)) must have the same number of columns.
AAAABB
AAAABB
CCCDDD
CCCDDD
CCCDDD
CCCDDD
What if we just ignore this constraint, and only require the output to be
AAAABB
AAAABB
CCCCBB
CCCCBB
CCCCDD
CCCCDD
or
AAAABB
AAAABB
CCCCBB
CCCCBB
CCCCBB
CCCCBB
This should be 'stackable', yes? Or perhaps we need another stacking
function for this, say numpy.tetris?
And while we're at it, what about higher dimensions? should there be an
ndstack function too?
This is starting to look like the second time in a row Stefan tries to
extend numpy with a simple convenience function, and he gets tricked into
implementing some sophisticated algorithm...
For his next PR I expect nothing less than an NP-complete problem. ;-)
Post by Sturla Molden
Jaime
--
(\__/)
( O.o)
( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus
planes de dominación mundial.
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Stefan Otte
2014-10-28 18:34:14 UTC
Permalink
Hey,

In the last weeks I tested `np.asarray(np.bmat(....))` as `stack`
function and it works quite well. So the question persits: If `bmat`
already offers something like `stack` should we even bother
implementing `stack`? More code leads to more
bugs and maintenance work. (However, the current implementation is
only 5 lines and by using `bmat` which would reduce that even more.)


Best,
Stefan
Post by Paul Hobson
Hey Ben,
Side note: I've had to do the same thing for stitching curvilinear model
grid coordinates together. Usings pandas DataFrames indexed by `i` and `j`
is really good for this. You can offset the indices directly, unstack the
DF, and the pandas will align for you.
Happy to send an example along if you're curious.
-p
Post by Benjamin Root
A use case would be "image stitching" or even data tiling. I have had to
implement something like this at work (so, I can't share it, unfortunately)
and it even goes so far as to allow the caller to specify how much the tiles
can overlap and such. The specification is ungodly hideous and I doubt I
would be willing to share it even if I could lest I release code-thulu upon
the world...
I think just having this generalize stack feature would be nice start.
Tetris could be built on top of that later. (Although, I do vote for at
least 3 or 4 dimensional stacking, if possible).
Cheers!
Ben Root
On Mon, Sep 8, 2014 at 12:41 PM, Eelco Hoogendoorn
Post by Eelco Hoogendoorn
Sturla: im not sure if the intention is always unambiguous, for such more
flexible arrangements.
Also, I doubt such situations arise often in practice; if the arrays arnt
a grid, they are probably a nested grid, and the code would most naturally
concatenate them with nested calls to a stacking function.
However, some form of nd-stack function would be neat in my opinion.
On Mon, Sep 8, 2014 at 6:10 PM, Jaime Fernández del Río
Post by Jaime Fernández del Río
Post by Sturla Molden
Post by Stefan Otte
stack([[a, b], [c, d]])
In my case `stack` replaced `hstack` and `vstack` almost completely.
If you're interested in including it in numpy I created a pull request
[1]. I'm looking forward to getting some feedback!
As far as I can see, it uses hstack and vstack. But that means a and b have
to have the same number of rows, c and d must have the same rumber of rows,
and hstack((a,b)) and hstack((c,d)) must have the same number of columns.
AAAABB
AAAABB
CCCDDD
CCCDDD
CCCDDD
CCCDDD
What if we just ignore this constraint, and only require the output to be
AAAABB
AAAABB
CCCCBB
CCCCBB
CCCCDD
CCCCDD
or
AAAABB
AAAABB
CCCCBB
CCCCBB
CCCCBB
CCCCBB
This should be 'stackable', yes? Or perhaps we need another stacking
function for this, say numpy.tetris?
And while we're at it, what about higher dimensions? should there be an
ndstack function too?
This is starting to look like the second time in a row Stefan tries to
extend numpy with a simple convenience function, and he gets tricked into
implementing some sophisticated algorithm...
For his next PR I expect nothing less than an NP-complete problem. ;-)
Jaime
--
(\__/)
( O.o)
( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus
planes de dominación mundial.
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Nathaniel Smith
2014-10-28 18:46:24 UTC
Permalink
Post by Stefan Otte
Hey,
In the last weeks I tested `np.asarray(np.bmat(....))` as `stack`
function and it works quite well. So the question persits: If `bmat`
already offers something like `stack` should we even bother
implementing `stack`? More code leads to more
bugs and maintenance work. (However, the current implementation is
only 5 lines and by using `bmat` which would reduce that even more.)
In the long run we're trying to reduce usage of np.matrix and ideally
deprecate it entirely. So yes, providing ndarray equivalents of matrix
functionality (like bmat) is valuable.

-n
Stefan Otte
2014-10-29 14:59:30 UTC
Permalink
Hey,

there are several ways how to proceed.

- My proposed solution covers the 80% case quite well (at least I use
it all the time). I'd convert the doctests into unittests and we're
done.

- We could slightly change the interface to leave out the surrounding
square brackets, i.e. turning `stack([[a, b], [c, d]])` into
`stack([a, b], [c, d])`

- We could extend it even further allowing a "filler value" for non
set values and a "shape" argument. This could be done later as well.

- `bmat` is not really matrix specific. We could refactor `bmat` a bit
to use the same logic in `stack`. Except the `matrix` calls `bmat` and
`_from_string` are pretty agnostic to the input.

I'm in favor of the first or last approach. The first: because it
already works and is quite simple. The last: because the logic and
tests of both `bmat` and `stack` would be the same and the feature to
specify a string representation of the block matrix is nice.


Best,
Stefan
Post by Nathaniel Smith
Post by Stefan Otte
Hey,
In the last weeks I tested `np.asarray(np.bmat(....))` as `stack`
function and it works quite well. So the question persits: If `bmat`
already offers something like `stack` should we even bother
implementing `stack`? More code leads to more
bugs and maintenance work. (However, the current implementation is
only 5 lines and by using `bmat` which would reduce that even more.)
In the long run we're trying to reduce usage of np.matrix and ideally
deprecate it entirely. So yes, providing ndarray equivalents of matrix
functionality (like bmat) is valuable.
-n
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Stefan Otte
2014-10-31 13:13:04 UTC
Permalink
To make the last point more concrete the implementation could look
something like this (note that I didn't test it and that it still
takes some work):


def bmat(obj, ldict=None, gdict=None):
return matrix(stack(obj, ldict, gdict))


def stack(obj, ldict=None, gdict=None):
# the old bmat code minus the matrix calls
if isinstance(obj, str):
if gdict is None:
# get previous frame
frame = sys._getframe().f_back
glob_dict = frame.f_globals
loc_dict = frame.f_locals
else:
glob_dict = gdict
loc_dict = ldict
return _from_string(obj, glob_dict, loc_dict)

if isinstance(obj, (tuple, list)):
# [[A,B],[C,D]]
arr_rows = []
for row in obj:
if isinstance(row, N.ndarray): # not 2-d
return concatenate(obj, axis=-1)
else:
arr_rows.append(concatenate(row, axis=-1))
return concatenate(arr_rows, axis=0)

if isinstance(obj, N.ndarray):
return obj


I basically turned the old `bmat` into `stack` and removed the matrix calls.


Best,
Stefan
Post by Stefan Otte
Hey,
there are several ways how to proceed.
- My proposed solution covers the 80% case quite well (at least I use
it all the time). I'd convert the doctests into unittests and we're
done.
- We could slightly change the interface to leave out the surrounding
square brackets, i.e. turning `stack([[a, b], [c, d]])` into
`stack([a, b], [c, d])`
- We could extend it even further allowing a "filler value" for non
set values and a "shape" argument. This could be done later as well.
- `bmat` is not really matrix specific. We could refactor `bmat` a bit
to use the same logic in `stack`. Except the `matrix` calls `bmat` and
`_from_string` are pretty agnostic to the input.
I'm in favor of the first or last approach. The first: because it
already works and is quite simple. The last: because the logic and
tests of both `bmat` and `stack` would be the same and the feature to
specify a string representation of the block matrix is nice.
Best,
Stefan
Post by Nathaniel Smith
Post by Stefan Otte
Hey,
In the last weeks I tested `np.asarray(np.bmat(....))` as `stack`
function and it works quite well. So the question persits: If `bmat`
already offers something like `stack` should we even bother
implementing `stack`? More code leads to more
bugs and maintenance work. (However, the current implementation is
only 5 lines and by using `bmat` which would reduce that even more.)
In the long run we're trying to reduce usage of np.matrix and ideally
deprecate it entirely. So yes, providing ndarray equivalents of matrix
functionality (like bmat) is valuable.
-n
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Stefan Otte
2015-05-10 10:33:30 UTC
Permalink
Hey,

Just a quick update. I updated the pull request and renamed `stack` into
`block`. Have a look: https://github.com/numpy/numpy/pull/5057

I'm sticking with simple initial implementation because it's simple and
does what you think it does.


Cheers,
Stefan
Post by Stefan Otte
To make the last point more concrete the implementation could look
something like this (note that I didn't test it and that it still
return matrix(stack(obj, ldict, gdict))
# the old bmat code minus the matrix calls
# get previous frame
frame = sys._getframe().f_back
glob_dict = frame.f_globals
loc_dict = frame.f_locals
glob_dict = gdict
loc_dict = ldict
return _from_string(obj, glob_dict, loc_dict)
# [[A,B],[C,D]]
arr_rows = []
if isinstance(row, N.ndarray): # not 2-d
return concatenate(obj, axis=-1)
arr_rows.append(concatenate(row, axis=-1))
return concatenate(arr_rows, axis=0)
return obj
I basically turned the old `bmat` into `stack` and removed the matrix calls.
Best,
Stefan
Post by Stefan Otte
Hey,
there are several ways how to proceed.
- My proposed solution covers the 80% case quite well (at least I use
it all the time). I'd convert the doctests into unittests and we're
done.
- We could slightly change the interface to leave out the surrounding
square brackets, i.e. turning `stack([[a, b], [c, d]])` into
`stack([a, b], [c, d])`
- We could extend it even further allowing a "filler value" for non
set values and a "shape" argument. This could be done later as well.
- `bmat` is not really matrix specific. We could refactor `bmat` a bit
to use the same logic in `stack`. Except the `matrix` calls `bmat` and
`_from_string` are pretty agnostic to the input.
I'm in favor of the first or last approach. The first: because it
already works and is quite simple. The last: because the logic and
tests of both `bmat` and `stack` would be the same and the feature to
specify a string representation of the block matrix is nice.
Best,
Stefan
Post by Nathaniel Smith
Post by Stefan Otte
Hey,
In the last weeks I tested `np.asarray(np.bmat(....))` as `stack`
function and it works quite well. So the question persits: If `bmat`
already offers something like `stack` should we even bother
implementing `stack`? More code leads to more
bugs and maintenance work. (However, the current implementation is
only 5 lines and by using `bmat` which would reduce that even more.)
In the long run we're trying to reduce usage of np.matrix and ideally
deprecate it entirely. So yes, providing ndarray equivalents of matrix
functionality (like bmat) is valuable.
-n
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Stefan Otte
2016-01-09 11:58:58 UTC
Permalink
Hey,

one of my new year's resolutions is to get my pull requests accepted (or
closed). So here we go...

Here is the update pull request: https://github.com/numpy/numpy/pull/5057
Here is the docstring:
https://github.com/sotte/numpy/commit/3d4c5d19a8f15b35df50d945b9c8853b683f7ab6#diff-2270128d50ff15badd1aba4021c50a8cR358

The new `block` function is very similar to matlab's `[A, B; C, D]`.

Pros:
- it's very useful (in my experiment)
- less friction for people coming from matlab
- it's conceptually simple
- the implementation is simple
- it's documented
- it's tested

Cons:
- the implementation is not super efficient. Temporary copies are created.
However, bmat also does that.

Feedback is very welcome!


Best,
Stefan
Post by Stefan Otte
Hey,
Just a quick update. I updated the pull request and renamed `stack` into
`block`. Have a look: https://github.com/numpy/numpy/pull/5057
I'm sticking with simple initial implementation because it's simple and
does what you think it does.
Cheers,
Stefan
Post by Stefan Otte
To make the last point more concrete the implementation could look
something like this (note that I didn't test it and that it still
return matrix(stack(obj, ldict, gdict))
# the old bmat code minus the matrix calls
# get previous frame
frame = sys._getframe().f_back
glob_dict = frame.f_globals
loc_dict = frame.f_locals
glob_dict = gdict
loc_dict = ldict
return _from_string(obj, glob_dict, loc_dict)
# [[A,B],[C,D]]
arr_rows = []
if isinstance(row, N.ndarray): # not 2-d
return concatenate(obj, axis=-1)
arr_rows.append(concatenate(row, axis=-1))
return concatenate(arr_rows, axis=0)
return obj
I basically turned the old `bmat` into `stack` and removed the matrix calls.
Best,
Stefan
Post by Stefan Otte
Hey,
there are several ways how to proceed.
- My proposed solution covers the 80% case quite well (at least I use
it all the time). I'd convert the doctests into unittests and we're
done.
- We could slightly change the interface to leave out the surrounding
square brackets, i.e. turning `stack([[a, b], [c, d]])` into
`stack([a, b], [c, d])`
- We could extend it even further allowing a "filler value" for non
set values and a "shape" argument. This could be done later as well.
- `bmat` is not really matrix specific. We could refactor `bmat` a bit
to use the same logic in `stack`. Except the `matrix` calls `bmat` and
`_from_string` are pretty agnostic to the input.
I'm in favor of the first or last approach. The first: because it
already works and is quite simple. The last: because the logic and
tests of both `bmat` and `stack` would be the same and the feature to
specify a string representation of the block matrix is nice.
Best,
Stefan
Post by Nathaniel Smith
Post by Stefan Otte
Hey,
In the last weeks I tested `np.asarray(np.bmat(....))` as `stack`
function and it works quite well. So the question persits: If `bmat`
already offers something like `stack` should we even bother
implementing `stack`? More code leads to more
bugs and maintenance work. (However, the current implementation is
only 5 lines and by using `bmat` which would reduce that even more.)
In the long run we're trying to reduce usage of np.matrix and ideally
deprecate it entirely. So yes, providing ndarray equivalents of matrix
functionality (like bmat) is valuable.
-n
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Gyro Funch
2016-01-09 17:11:24 UTC
Permalink
Post by Stefan Otte
Hey,
one of my new year's resolutions is to get my pull requests
accepted (or closed). So here we go...
https://github.com/sotte/numpy/commit/3d4c5d19a8f15b35df50d945b9c8853b683f7ab6#diff-2270128d50ff15badd1aba4021c50a8cR358
The new `block` function is very similar to matlab's `[A, B; C, D]`.
Pros: - it's very useful (in my experiment) - less friction for
people coming from matlab - it's conceptually simple - the
implementation is simple - it's documented - it's tested
Cons: - the implementation is not super efficient. Temporary
copies are created. However, bmat also does that.
Feedback is very welcome!
Best, Stefan
Without commenting on the implementation, I would find this function
*very* useful and convenient in my own work.

-gyro

Nathaniel Smith
2014-09-08 16:13:46 UTC
Permalink
Post by Sturla Molden
Post by Stefan Otte
stack([[a, b], [c, d]])
In my case `stack` replaced `hstack` and `vstack` almost completely.
If you're interested in including it in numpy I created a pull request
[1]. I'm looking forward to getting some feedback!
As far as I can see, it uses hstack and vstack. But that means a and b have
to have the same number of rows, c and d must have the same rumber of rows,
and hstack((a,b)) and hstack((c,d)) must have the same number of columns.
AAAABB
AAAABB
CCCDDD
CCCDDD
CCCDDD
CCCDDD
What if we just ignore this constraint, and only require the output to be
AAAABB
AAAABB
CCCCBB
CCCCBB
CCCCDD
CCCCDD
or
AAAABB
AAAABB
CCCCBB
CCCCBB
CCCCBB
CCCCBB
This should be 'stackable', yes? Or perhaps we need another stacking
function for this, say numpy.tetris?
It's not at all obvious to me how to describe such "tetris" configurations,
or interpret then unambiguously. Do you have a more detailed specification
in mind?
Post by Sturla Molden
And while we're at it, what about higher dimensions? should there be an
ndstack function too?
Same comment here.

-n
Joseph Martinot-Lagarde
2014-09-08 20:39:49 UTC
Permalink
Post by Sturla Molden
Post by Stefan Otte
stack([[a, b], [c, d]])
In my case `stack` replaced `hstack` and `vstack` almost completely.
If you're interested in including it in numpy I created a pull request
[1]. I'm looking forward to getting some feedback!
As far as I can see, it uses hstack and vstack. But that means a and b have
to have the same number of rows, c and d must have the same rumber of rows,
and hstack((a,b)) and hstack((c,d)) must have the same number of columns.
AAAABB
AAAABB
CCCDDD
CCCDDD
CCCDDD
CCCDDD
What if we just ignore this constraint, and only require the output to be
AAAABB
AAAABB
CCCCBB
CCCCBB
CCCCDD
CCCCDD
or
AAAABB
AAAABB
CCCCBB
CCCCBB
CCCCBB
CCCCBB
stack([stack([[a], [c]]), b])
Post by Sturla Molden
This should be 'stackable', yes? Or perhaps we need another stacking
function for this, say numpy.tetris?
The function should be implemented for its name only ! I like it !
Post by Sturla Molden
And while we're at it, what about higher dimensions? should there be an
ndstack function too?
Sturla
Joseph Martinot-Lagarde
2014-09-08 20:40:39 UTC
Permalink
Post by Stefan Otte
Hey,
quite often I work with block matrices. Matlab offers the convenient notation
[ a b; c d ]
vstack([hstack([a,b]), hstack([c,d])])
stack([[a, b], [c, d]])
In my case `stack` replaced `hstack` and `vstack` almost completely.
If you're interested in including it in numpy I created a pull request
[1]. I'm looking forward to getting some feedback!
Best,
Stefan
[1] https://github.com/numpy/numpy/pull/5057
The outside brackets are redundant, stack([[a, b], [c, d]]) should be
stack([a, b], [c, d])
cjw
2014-09-08 22:14:24 UTC
Permalink
Post by Joseph Martinot-Lagarde
Post by Stefan Otte
Hey,
quite often I work with block matrices. Matlab offers the convenient notation
[ a b; c d ]
This would appear to be a desirable way to go.

Numpy has something similar for strings. The above is neater.

Colin W.
Post by Joseph Martinot-Lagarde
Post by Stefan Otte
vstack([hstack([a,b]), hstack([c,d])])
stack([[a, b], [c, d]])
In my case `stack` replaced `hstack` and `vstack` almost completely.
If you're interested in including it in numpy I created a pull request
[1]. I'm looking forward to getting some feedback!
Best,
Stefan
[1] https://github.com/numpy/numpy/pull/5057
The outside brackets are redundant, stack([[a, b], [c, d]]) should be
stack([a, b], [c, d])
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Stefan Otte
2014-09-09 09:42:38 UTC
Permalink
Hey,

@Josef, I wasn't aware of `bmat` and `np.asarray(np.bmat(....))` does
basically what I want and what I'm already using.

Regarding the Tetris problem: that never happened to me, but stack, as
Josef pointed out, can handle that already :)

I like the idea of removing the redundant square brackets:
stack([[a, b], [c, d]]) --> stack([a, b], [c, d])
However, if the brackets are there there is no difference between
creating a `np.array` and stacking arrays with `np.stack`.

If we want to get fancy and turn this PR into something bigger
(working our way up to a NP-complete problem ;)) then how about this.
I sometimes have arrays that look like:
AB0
0 C
Where 0 is a scalar but is supposed to fill the rest of the array.
Having something like 0 in there might lead to ambiguities though. What does
ABC
0D0
mean? One could limit the "filler" to appear only on the left or the right:
AB0
0CD
But even then the shape is not completely determined. So we could
require to have one row that only consists of arrays and determines
the shape. Alternatively we could have a keyword parameter `shape`:
stack([A, B, 0], [0, C, D], shape=(8, 8))

Colin, with `bmat` you can do what you're asking for. Directly taken
Post by cjw
Post by Joseph Martinot-Lagarde
np.bmat('A,B; C,D')
matrix([[1, 1, 2, 2],
[1, 1, 2, 2],
[3, 4, 7, 8],
[5, 6, 9, 0]])


General question: If `bmat` already offers something like `stack`
should we even bother implementing `stack`? More code leads to more
bugs and maintenance work.


Best,
Stefan
Post by cjw
Post by Joseph Martinot-Lagarde
Hey,
quite often I work with block matrices. Matlab offers the convenient notation
[ a b; c d ]
This would appear to be a desirable way to go.
Numpy has something similar for strings. The above is neater.
Colin W.
Post by Joseph Martinot-Lagarde
vstack([hstack([a,b]), hstack([c,d])])
stack([[a, b], [c, d]])
In my case `stack` replaced `hstack` and `vstack` almost completely.
If you're interested in including it in numpy I created a pull request
[1]. I'm looking forward to getting some feedback!
Best,
Stefan
[1] https://github.com/numpy/numpy/pull/5057
The outside brackets are redundant, stack([[a, b], [c, d]]) should be
stack([a, b], [c, d])
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion
j***@gmail.com
2014-09-09 12:30:13 UTC
Permalink
Post by Stefan Otte
Hey,
@Josef, I wasn't aware of `bmat` and `np.asarray(np.bmat(....))` does
basically what I want and what I'm already using.
I never needed any tetris or anything similar except for the matched block
version.

Just to point out two more related functions

scipy.sparse also has `bmat` for sparse block matrices
scipy.linalg and scipy.sparse have `block_diag` to complement bmat.


What I sometimes wish for is a sparse pseudo kronecker product as
convenience to bmat, where the first (or the second) matrix contains (0,1)
flags where the 1's specify where to put the blocks.
(I'm not sure what I really mean, similar to block_diag but with a
different filling pattern.)

Josef
Post by Stefan Otte
Regarding the Tetris problem: that never happened to me, but stack, as
Josef pointed out, can handle that already :)
stack([[a, b], [c, d]]) --> stack([a, b], [c, d])
However, if the brackets are there there is no difference between
creating a `np.array` and stacking arrays with `np.stack`.
If we want to get fancy and turn this PR into something bigger
(working our way up to a NP-complete problem ;)) then how about this.
AB0
0 C
Where 0 is a scalar but is supposed to fill the rest of the array.
Having something like 0 in there might lead to ambiguities though. What does
ABC
0D0
AB0
0CD
But even then the shape is not completely determined. So we could
require to have one row that only consists of arrays and determines
stack([A, B, 0], [0, C, D], shape=(8, 8))
Colin, with `bmat` you can do what you're asking for. Directly taken
Post by cjw
Post by Joseph Martinot-Lagarde
np.bmat('A,B; C,D')
matrix([[1, 1, 2, 2],
[1, 1, 2, 2],
[3, 4, 7, 8],
[5, 6, 9, 0]])
General question: If `bmat` already offers something like `stack`
should we even bother implementing `stack`? More code leads to more
bugs and maintenance work.
Best,
Stefan
Post by cjw
Post by Joseph Martinot-Lagarde
Hey,
quite often I work with block matrices. Matlab offers the convenient
notation
Post by cjw
Post by Joseph Martinot-Lagarde
[ a b; c d ]
This would appear to be a desirable way to go.
Numpy has something similar for strings. The above is neater.
Colin W.
Post by Joseph Martinot-Lagarde
vstack([hstack([a,b]), hstack([c,d])])
stack([[a, b], [c, d]])
In my case `stack` replaced `hstack` and `vstack` almost completely.
If you're interested in including it in numpy I created a pull request
[1]. I'm looking forward to getting some feedback!
Best,
Stefan
[1] https://github.com/numpy/numpy/pull/5057
The outside brackets are redundant, stack([[a, b], [c, d]]) should be
stack([a, b], [c, d])
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion
j***@gmail.com
2014-09-09 12:42:27 UTC
Permalink
Post by j***@gmail.com
Post by Stefan Otte
Hey,
@Josef, I wasn't aware of `bmat` and `np.asarray(np.bmat(....))` does
basically what I want and what I'm already using.
I never needed any tetris or anything similar except for the matched block
version.
Just to point out two more related functions
scipy.sparse also has `bmat` for sparse block matrices
scipy.linalg and scipy.sparse have `block_diag` to complement bmat.
What I sometimes wish for is a sparse pseudo kronecker product as
convenience to bmat, where the first (or the second) matrix contains (0,1)
flags where the 1's specify where to put the blocks.
(I'm not sure what I really mean, similar to block_diag but with a
different filling pattern.)
(or in analogy to kronecker, np.nonzero provides the filling pattern, but
the submatrix is multiplied by the value.)

(application: unbalanced repeated measures or panel data)

Josef
(....)
Post by j***@gmail.com
Josef
Post by Stefan Otte
Regarding the Tetris problem: that never happened to me, but stack, as
Josef pointed out, can handle that already :)
stack([[a, b], [c, d]]) --> stack([a, b], [c, d])
However, if the brackets are there there is no difference between
creating a `np.array` and stacking arrays with `np.stack`.
If we want to get fancy and turn this PR into something bigger
(working our way up to a NP-complete problem ;)) then how about this.
AB0
0 C
Where 0 is a scalar but is supposed to fill the rest of the array.
Having something like 0 in there might lead to ambiguities though. What does
ABC
0D0
AB0
0CD
But even then the shape is not completely determined. So we could
require to have one row that only consists of arrays and determines
stack([A, B, 0], [0, C, D], shape=(8, 8))
Colin, with `bmat` you can do what you're asking for. Directly taken
Post by cjw
Post by Joseph Martinot-Lagarde
np.bmat('A,B; C,D')
matrix([[1, 1, 2, 2],
[1, 1, 2, 2],
[3, 4, 7, 8],
[5, 6, 9, 0]])
General question: If `bmat` already offers something like `stack`
should we even bother implementing `stack`? More code leads to more
bugs and maintenance work.
Best,
Stefan
Post by cjw
Post by Joseph Martinot-Lagarde
Hey,
quite often I work with block matrices. Matlab offers the convenient
notation
Post by cjw
Post by Joseph Martinot-Lagarde
[ a b; c d ]
This would appear to be a desirable way to go.
Numpy has something similar for strings. The above is neater.
Colin W.
Post by Joseph Martinot-Lagarde
vstack([hstack([a,b]), hstack([c,d])])
stack([[a, b], [c, d]])
In my case `stack` replaced `hstack` and `vstack` almost completely.
If you're interested in including it in numpy I created a pull request
[1]. I'm looking forward to getting some feedback!
Best,
Stefan
[1] https://github.com/numpy/numpy/pull/5057
The outside brackets are redundant, stack([[a, b], [c, d]]) should be
stack([a, b], [c, d])
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Loading...