[Numpy-discussion] The process I intend to follow for any proposed changes to NumPy

[Travis Oliphant]

Hey all,

I just wanted to clarify, that I am very excited about a few ideas I have
--- but I don't have time myself to engage in the community process to get
these changes into NumPy. However, those are real processes --- I've
been coaching a few people in those processes for the past several years
already.

So, rather than do nothing, what I'm looking to do is to work with a few
people who I can share my ideas with, get excited about the ideas, and then
who will work with the community to get them implemented. That's what I
was announcing and talking about yesterday --- looking for interested
people who want to work on NumPy *with* the NumPy community.

In my enthusiasm, I realize that some may have mis-understood my
intention. There is no 'imminent' fork, nor am I planning on doing some
crazy amount of work that I then try to force on other developers of NumPy.


What I'm planning to do is find people to train on NumPy code base (people
to increase the diversity of the developers would be ideal -- but hard to
accomplish). I plan to train them on NumPy based on my experience, and on
what I think should be done --- and then have *them* work through the
community process and engage with others to get consensus (hopefully not
losing too much in translation in the process --- but instead getting even
better).

During that process I will engage as a member of the community and help
write NEPs and other documents and help clarify where it makes sense as I
can. I will be filtering for people that actually want to see NumPy get
better. Until I identify the people and work with them, it will be hard
to tell how this will best work. So, stay tuned.

If all goes well, what you should see in a few weeks time are specific
proposals, a branch or two, and the beginnings of some pull requests. If
you don't see that, then I will not have found the right people to help me,
and we will all continue to go back to searching.

While I'm expecting the best, in the worst case, we get additional people
who know the NumPy code base and can help squash bugs as well as implement
changes that are desired. Three things are needed if you want to
participate in this: 1) A willingness to work with the open source
community, 2) a deep knowledge of C and in-particular CPython's brand of C,
and 3) a willingness to engage with me, do a mind-meld and dump around the
NumPy code base, and then improve on what is in my head with the rest of
the community.

Thanks,

-Travis

[Ondřej Čertík]

Hi Travis,

Post by Travis Oliphant
Hey all,
I just wanted to clarify, that I am very excited about a few ideas I have
--- but I don't have time myself to engage in the community process to get
these changes into NumPy. However, those are real processes --- I've
been coaching a few people in those processes for the past several years
already.
So, rather than do nothing, what I'm looking to do is to work with a few
people who I can share my ideas with, get excited about the ideas, and then
who will work with the community to get them implemented. That's what I
was announcing and talking about yesterday --- looking for interested people
who want to work on NumPy *with* the NumPy community.
In my enthusiasm, I realize that some may have mis-understood my intention.
There is no 'imminent' fork, nor am I planning on doing some crazy amount of
work that I then try to force on other developers of NumPy.
What I'm planning to do is find people to train on NumPy code base (people
to increase the diversity of the developers would be ideal -- but hard to
accomplish). I plan to train them on NumPy based on my experience, and on
what I think should be done --- and then have *them* work through the
community process and engage with others to get consensus (hopefully not
losing too much in translation in the process --- but instead getting even
better).
During that process I will engage as a member of the community and help
write NEPs and other documents and help clarify where it makes sense as I
can. I will be filtering for people that actually want to see NumPy get
better. Until I identify the people and work with them, it will be hard
to tell how this will best work. So, stay tuned.
If all goes well, what you should see in a few weeks time are specific
proposals, a branch or two, and the beginnings of some pull requests. If
you don't see that, then I will not have found the right people to help me,
and we will all continue to go back to searching.
While I'm expecting the best, in the worst case, we get additional people
who know the NumPy code base and can help squash bugs as well as implement
changes that are desired. Three things are needed if you want to
participate in this: 1) A willingness to work with the open source
community, 2) a deep knowledge of C and in-particular CPython's brand of C,
and 3) a willingness to engage with me, do a mind-meld and dump around the
NumPy code base, and then improve on what is in my head with the rest of the
community.

I don't have time to do the work myself, but I'll be happy to help in
terms of the community. Travis has paid me in the past to do the NumPy
1.7 release, and implement/fix some things that Travis had in mind. I
had no prior involvement with the NumPy codebase / community, and
within few months I had to get myself familiar with the code, fix
release critical bugs, get my PRs accepted by the community,
eventually get push access (from the core developers, not from Travis)
so that I can merge other people's patches, and most importantly, make
the community allow me to actually do the release, even though I came
from outside. As far as I know, Travis didn't have to pull any strings
for me, and as far as I know, there was no friction in the community
either --- there was a job to be done, we got it done, and that was
it.

So I'll be happy to share my experience and help the person (if coming
from outside) how to do this. In this case a push access is not
necessary, so it's just about sending high quality PRs and good
communication.

Ondrej

[Chris Barker]

Travis,

I'm sure you appreciate that this might all look a bit scary, given the
recent discussion about numpy governance.

But it's an open-source project, and I, at least, fully understand that
going through a big process is NOT the way to get a new idea tried out and
implemented. So I think think this is a great development -- I know I want
to see something like this dtype work done.

So, as someone who has been around this community for a long time, and
dependent on Numeric, numarray, and numpy over the years, this looks like a
great development.

And, in fact, with the new governance effort -- I think less scary --
people can go off and work on a branch or fork, do good stuff, and we, as a
community, can be assured that API (or even ABI) changes won't be thrust
upon us unawares :-)

As for the technical details -- I get a bit lost, not fully understanding
the current dtype system either, but do your ideas take us in the direction
of having dtypes independent of the container and ufunc machinery -- and
thus easier to create new dtypes (even in Python?) 'cause that would be
great.

I hope you find the partner you're looking for -- that's a challenge!

-Chris
--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE [phone] fax
Seattle, WA 98115 [phone] main reception

***@noaa.gov

[Travis Oliphant]

Hey Chris (limiting to NumPy only),

I've had some great conversations with Nathaniel in the past few days and
I'm glad he posted his thoughts so that there is no confusion about
governance or what I was implying.

With respect to governance, I'm very supportive of what everyone is doing
in organizing a governance document and approach and appreciate the effort
of Nathaniel and others to move this forward. Nothing I said was meant to
imply differently. I'm sorry if it made anyone nervous.

I'm a very enthusiastic person when I get an idea of what to do. I like
to see things implemented. In this case, it also turns out that in terms
of overall architecture, my ideas are actually very similar to Nathaniel's
ideas. That's a good sign. We have different tactical approaches as to
how to move forward, but I think it's a good thing to note that we see a
very similar path forward. Nothing will be done in NumPy itself except
via pull-request and review.

My approach for the ideas I'm pursuing will be to organize people around
two new prototype packages I'm calling memtype and gufunc. The purpose of
these is to allow playing with the design and ideas quickly before looking
at how to put them into NumPy itself --- there will also be some training
involved in getting people up to speed. There was a long discussion
today at this BIDS data-structures for data-science summit part of which
talked about how to improve NumPy's dtype system. I would love to these
independent objects evolve into independent packages that could even go
into Python standard library. Not everyone agrees that is the best
idea, but regardless of whether this happens or not, the intent is to do
work that could go into NumPy now.

I look forward to the activity.

-Travis

Post by Ondřej Čertík
Travis,
I'm sure you appreciate that this might all look a bit scary, given the
recent discussion about numpy governance.
But it's an open-source project, and I, at least, fully understand that
going through a big process is NOT the way to get a new idea tried out and
implemented. So I think think this is a great development -- I know I want
to see something like this dtype work done.
So, as someone who has been around this community for a long time, and
dependent on Numeric, numarray, and numpy over the years, this looks like a
great development.
And, in fact, with the new governance effort -- I think less scary --
people can go off and work on a branch or fork, do good stuff, and we, as a
community, can be assured that API (or even ABI) changes won't be thrust
upon us unawares :-)
As for the technical details -- I get a bit lost, not fully understanding
the current dtype system either, but do your ideas take us in the direction
of having dtypes independent of the container and ufunc machinery -- and
thus easier to create new dtypes (even in Python?) 'cause that would be
great.
I hope you find the partner you're looking for -- that's a challenge!
-Chris
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R [phone] voice
7600 Sand Point Way NE [phone] fax
Seattle, WA 98115 [phone] main reception
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Nathaniel Smith]

Hi all,

Thanks, Travis, for the followup. I know some people were confused or
concerned by some points in Travis’s recent emails (as was I,
initially), but after checking in with Travis and the NumPy steering
council, it sounds like the main points of possible confusion are
actually things where we at least are actually on the same page. So to
avoid any uncertainty or miscommunication, I just want to reiterate
some hopefully-uncontroversial points here:

- There is currently no plan whatsoever for NumPy to drop Python 2
support; something like ~80% of our users are still using Python 2,
and we anticipate that both Python 2 and Python 3 will continue to
receive full support for the foreseeable future.

- While it is possible that there will eventually be a
compatibility-breaking “NumPy 2.0” release, it won’t happen without a
community consensus that this is a good idea, and that consensus has
not yet been reached.

- As a clarification on how NumPy governance works: While we all
continue to be grateful to Travis for his past contributions, in terms
of formal authority he stepped down from his leadership role in the
project several years ago, and the project has since switched to a
community-driven governance model [1]. Under this model, all
contributors participate as equal peers, and (outside of specific
exceptional circumstances) all contributors should be assumed to be
speaking only on behalf of themselves, not the project as a whole.

- Discussion about the best way to improve NumPy’s dtype system is
ongoing, and several approaches are under consideration. No proposal
will be accepted without review and consensus by the broader NumPy
community. We welcome anyone who’s interested in these issues to join
us on the mailing list -- the more input we have, the better the
result will be :-).

Apologies for the wide distribution of this message; I suggest any
followups be directed to numpy-***@scipy.org only.

Thanks,
- Nathaniel

[1] http://thread.gmane.org/gmane.comp.python.numeric.general/61106
(we’ll move it into the repository soon, we promise!)

Post by Travis Oliphant
Hey all,
I just wanted to clarify, that I am very excited about a few ideas I have
--- but I don't have time myself to engage in the community process to get
these changes into NumPy. However, those are real processes --- I've
been coaching a few people in those processes for the past several years
already.
So, rather than do nothing, what I'm looking to do is to work with a few
people who I can share my ideas with, get excited about the ideas, and then
who will work with the community to get them implemented. That's what I
was announcing and talking about yesterday --- looking for interested people
who want to work on NumPy *with* the NumPy community.
In my enthusiasm, I realize that some may have mis-understood my intention.
There is no 'imminent' fork, nor am I planning on doing some crazy amount of
work that I then try to force on other developers of NumPy.
What I'm planning to do is find people to train on NumPy code base (people
to increase the diversity of the developers would be ideal -- but hard to
accomplish). I plan to train them on NumPy based on my experience, and on
what I think should be done --- and then have *them* work through the
community process and engage with others to get consensus (hopefully not
losing too much in translation in the process --- but instead getting even
better).
During that process I will engage as a member of the community and help
write NEPs and other documents and help clarify where it makes sense as I
can. I will be filtering for people that actually want to see NumPy get
better. Until I identify the people and work with them, it will be hard
to tell how this will best work. So, stay tuned.
If all goes well, what you should see in a few weeks time are specific
proposals, a branch or two, and the beginnings of some pull requests. If
you don't see that, then I will not have found the right people to help me,
and we will all continue to go back to searching.
While I'm expecting the best, in the worst case, we get additional people
who know the NumPy code base and can help squash bugs as well as implement
changes that are desired. Three things are needed if you want to
participate in this: 1) A willingness to work with the open source
community, 2) a deep knowledge of C and in-particular CPython's brand of C,
and 3) a willingness to engage with me, do a mind-meld and dump around the
NumPy code base, and then improve on what is in my head with the rest of the
community.
Thanks,
-Travis
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion