[Numpy-discussion] Video meeting this week

[Nathaniel Smith]

Hi all,

In a week and a half, this is happening:

https://github.com/numpy/numpy/wiki/SciPy-2015-developer-meeting

It's somewhat short notice (my bad :-/), but I think it would be good
to have a short video meeting sometime this week as a kind of
"pre-meeting" -- to at least briefly go over the main issues we see
facing the project to prime the pump, get a better idea about what we
want to accomplish at the meeting itself, and gather some early
feedback from anyone who won't be able to make it to SciPy (we'll miss
you).

The obligatory doodle:
http://doodle.com/6b4s6thqt9xt4vnh

Depending on the interest level, I'm thinking we'll either use Google
Hangouts or Bluejeans (https://bluejeans.com/ -- same as what Ralf
used for the similar SciPy meeting a few months ago; needs a plugin
installed but is available for Windows / OS X / 64-bit Linux / Android
/ iOS, or regular telephone, or h323 softphone).

-n

[Nathaniel Smith]

Post by Nathaniel Smith
Hi all,
https://github.com/numpy/numpy/wiki/SciPy-2015-developer-meeting
It's somewhat short notice (my bad :-/), but I think it would be good
to have a short video meeting sometime this week as a kind of
"pre-meeting" -- to at least briefly go over the main issues we see
facing the project to prime the pump, get a better idea about what we
want to accomplish at the meeting itself, and gather some early
feedback from anyone who won't be able to make it to SciPy (we'll miss
you).
http://doodle.com/6b4s6thqt9xt4vnh

It's looking like sometime Thursday between 12:00 and 15:00 California time
(1900-2200 UTC) is most likely. Please fill this out now if you have an
interest, and I'll declare a winner this afternoon...

-n

[Nathaniel Smith]

Post by Nathaniel Smith

Post by Nathaniel Smith
Hi all,
https://github.com/numpy/numpy/wiki/SciPy-2015-developer-meeting
It's somewhat short notice (my bad :-/), but I think it would be good
to have a short video meeting sometime this week as a kind of
"pre-meeting" -- to at least briefly go over the main issues we see
facing the project to prime the pump, get a better idea about what we
want to accomplish at the meeting itself, and gather some early
feedback from anyone who won't be able to make it to SciPy (we'll miss
you).
http://doodle.com/6b4s6thqt9xt4vnh

It's looking like sometime Thursday between 12:00 and 15:00 California time
(1900-2200 UTC) is most likely. Please fill this out now if you have an
interest, and I'll declare a winner this afternoon...

Actually Wednesday 11:00-15:00 California time = 18:00-22:00 UTC is
also viable -- Chuck/Sebastian/Julian/anyone else, any preference?

-n

[Charles R Harris]

Any of those times would work for me.

Post by Nathaniel Smith

Post by Nathaniel Smith

Post by Nathaniel Smith
Hi all,
https://github.com/numpy/numpy/wiki/SciPy-2015-developer-meeting
It's somewhat short notice (my bad :-/), but I think it would be good
to have a short video meeting sometime this week as a kind of
"pre-meeting" -- to at least briefly go over the main issues we see
facing the project to prime the pump, get a better idea about what we
want to accomplish at the meeting itself, and gather some early
feedback from anyone who won't be able to make it to SciPy (we'll miss
you).
http://doodle.com/6b4s6thqt9xt4vnh

It's looking like sometime Thursday between 12:00 and 15:00 California

time

Post by Nathaniel Smith
(1900-2200 UTC) is most likely. Please fill this out now if you have an
interest, and I'll declare a winner this afternoon...

Actually Wednesday 11:00-15:00 California time = 18:00-22:00 UTC is
also viable -- Chuck/Sebastian/Julian/anyone else, any preference?
-n
--
Nathaniel J. Smith -- http://vorpus.org
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Nathaniel Smith]

Post by Nathaniel Smith
Hi all,
https://github.com/numpy/numpy/wiki/SciPy-2015-developer-meeting
It's somewhat short notice (my bad :-/), but I think it would be good
to have a short video meeting sometime this week as a kind of
"pre-meeting" -- to at least briefly go over the main issues we see
facing the project to prime the pump, get a better idea about what we
want to accomplish at the meeting itself, and gather some early
feedback from anyone who won't be able to make it to SciPy (we'll miss
you).
http://doodle.com/6b4s6thqt9xt4vnh

Okay, let's aim for:

Thursday July 2 at 20:00 UTC.

I believe that's 1pm California / 4 pm New York / 9pm London / 10pm
western Europe

And so far it looks like we'll be under the 10 person Google Hangouts
limit, which I'm assuming is simpler for everybody, so let's assume
we're doing that unless otherwise specified. (This does mean that I'd
appreciate a quick email if you're planning on dialling in but haven't
otherwise responded to the poll, though!)

-n

[Jeff Reback]

you guys have an agenda?

I can be reached on my cell 917-971-6387

Post by Nathaniel Smith

Post by Nathaniel Smith
Hi all,
https://github.com/numpy/numpy/wiki/SciPy-2015-developer-meeting
It's somewhat short notice (my bad :-/), but I think it would be good
to have a short video meeting sometime this week as a kind of
"pre-meeting" -- to at least briefly go over the main issues we see
facing the project to prime the pump, get a better idea about what we
want to accomplish at the meeting itself, and gather some early
feedback from anyone who won't be able to make it to SciPy (we'll miss
you).
http://doodle.com/6b4s6thqt9xt4vnh

Thursday July 2 at 20:00 UTC.
I believe that's 1pm California / 4 pm New York / 9pm London / 10pm
western Europe
And so far it looks like we'll be under the 10 person Google Hangouts
limit, which I'm assuming is simpler for everybody, so let's assume
we're doing that unless otherwise specified. (This does mean that I'd
appreciate a quick email if you're planning on dialling in but haven't
otherwise responded to the poll, though!)
-n
--
Nathaniel J. Smith -- http://vorpus.org
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Ralf Gommers]

Post by Jeff Reback
you guys have an agenda?

I'm guessing a subset of what's listed on
https://github.com/numpy/numpy/wiki/SciPy-2015-developer-meeting

Would indeed be good to make a proper agenda, so we can prepare properly
for the meeting at SciPy. There are some topics (like commit rights) that
can easily fill the whole call and are better done in person instead.
@Nathaniel: as the organizer, do you want to make a proposal?

Ralf

Post by Jeff Reback
I can be reached on my cell 917-971-6387

Post by Nathaniel Smith

Post by Nathaniel Smith
Hi all,
https://github.com/numpy/numpy/wiki/SciPy-2015-developer-meeting
It's somewhat short notice (my bad :-/), but I think it would be good
to have a short video meeting sometime this week as a kind of
"pre-meeting" -- to at least briefly go over the main issues we see
facing the project to prime the pump, get a better idea about what we
want to accomplish at the meeting itself, and gather some early
feedback from anyone who won't be able to make it to SciPy (we'll miss
you).
http://doodle.com/6b4s6thqt9xt4vnh

Thursday July 2 at 20:00 UTC.
I believe that's 1pm California / 4 pm New York / 9pm London / 10pm
western Europe
And so far it looks like we'll be under the 10 person Google Hangouts
limit, which I'm assuming is simpler for everybody, so let's assume
we're doing that unless otherwise specified. (This does mean that I'd
appreciate a quick email if you're planning on dialling in but haven't
otherwise responded to the poll, though!)
-n
--
Nathaniel J. Smith -- http://vorpus.org
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Nathaniel Smith]

Post by Ralf Gommers

Post by Jeff Reback
you guys have an agenda?

I'm guessing a subset of what's listed on
https://github.com/numpy/numpy/wiki/SciPy-2015-developer-meeting
Would indeed be good to make a proper agenda, so we can prepare properly for
the meeting at SciPy. There are some topics (like commit rights) that can
easily fill the whole call and are better done in person instead.
@Nathaniel: as the organizer, do you want to make a proposal?

Yep. Just reorganized the wiki page a bit to hopefully highlight the
big picture stuff (which I'm thinking is the stuff that's the highest
priority for our limited face-to-face time?), and made a
world-writeable google doc for us to use for the call here:
https://docs.google.com/document/d/11KC2p3cCsbDVjLcQSCehUiWGyWDNCyOunKfrO7Q7m3E/edit?usp=sharing

Copy/pasting the proposed agenda for the call from that google doc:

* Logistics for next week:

** What time do we want to start?

** What are our main goals for the meeting?

** How do we want to organize our time? (Maybe just sitting in a
meeting room grinding through an agenda for 10 hours straight is not
the best approach.)

** Are there any topics that e.g. we should plan to discuss at a
particular time so Ralf can plan to join remotely?

** Stuff like that.

* Do a *brief* pass through the list of topics for the meeting listed
on the wiki page (linked above). The goal here is not to actually have
a detailed discussion, but just:

** Get a general sense of where we stand so people can come prepared next week

** Get feedback on these topics from those who won’t be present for
the meeting next week

-n

[Honi Sanders]

I’m pretty new to Numpy so I won’t be able to contribute much, but I would appreciate being able to sit in on this if possible.
Honi

Post by Nathaniel Smith

Post by Nathaniel Smith
Hi all,
https://github.com/numpy/numpy/wiki/SciPy-2015-developer-meeting
It's somewhat short notice (my bad :-/), but I think it would be good
to have a short video meeting sometime this week as a kind of
"pre-meeting" -- to at least briefly go over the main issues we see
facing the project to prime the pump, get a better idea about what we
want to accomplish at the meeting itself, and gather some early
feedback from anyone who won't be able to make it to SciPy (we'll miss
you).
http://doodle.com/6b4s6thqt9xt4vnh

Thursday July 2 at 20:00 UTC.
I believe that's 1pm California / 4 pm New York / 9pm London / 10pm
western Europe
And so far it looks like we'll be under the 10 person Google Hangouts
limit, which I'm assuming is simpler for everybody, so let's assume
we're doing that unless otherwise specified. (This does mean that I'd
appreciate a quick email if you're planning on dialling in but haven't
otherwise responded to the poll, though!)
-n
--
Nathaniel J. Smith -- http://vorpus.org
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Nathaniel Smith]

Post by Honi Sanders
I’m pretty new to Numpy so I won’t be able to contribute much, but I would appreciate being able to sit in on this if possible.

You certainly would be welcome.

-n

[Chris Barker - NOAA Federal]

I _may_ be able to join -- but don't go setting up an alternative
conferencing system just for me.

But I'm planning on ring in Austin Tues in any case.

-Chris


Sent from my iPhone

Post by Nathaniel Smith

Post by Nathaniel Smith
Hi all,
https://github.com/numpy/numpy/wiki/SciPy-2015-developer-meeting
It's somewhat short notice (my bad :-/), but I think it would be good
to have a short video meeting sometime this week as a kind of
"pre-meeting" -- to at least briefly go over the main issues we see
facing the project to prime the pump, get a better idea about what we
want to accomplish at the meeting itself, and gather some early
feedback from anyone who won't be able to make it to SciPy (we'll miss
you).
http://doodle.com/6b4s6thqt9xt4vnh

Thursday July 2 at 20:00 UTC.
I believe that's 1pm California / 4 pm New York / 9pm London / 10pm
western Europe
And so far it looks like we'll be under the 10 person Google Hangouts
limit, which I'm assuming is simpler for everybody, so let's assume
we're doing that unless otherwise specified. (This does mean that I'd
appreciate a quick email if you're planning on dialling in but haven't
otherwise responded to the poll, though!)
-n
--
Nathaniel J. Smith -- http://vorpus.org
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Nathaniel Smith]

Hi all,

Meeting is starting in a few minutes!

Hangouts link:
https://plus.google.com/hangouts/_/gxtuplvm6g7s55abhjexqerll4a

Google doc for agenda and notes:
https://docs.google.com/document/d/11KC2p3cCsbDVjLcQSCehUiWGyWDNCyOunKfrO7Q7m3E/edit?usp=sharing

Wiki page:
https://github.com/numpy/numpy/wiki/SciPy-2015-developer-meeting

-n

Post by Nathaniel Smith
Hi all,
https://github.com/numpy/numpy/wiki/SciPy-2015-developer-meeting
It's somewhat short notice (my bad :-/), but I think it would be good
to have a short video meeting sometime this week as a kind of
"pre-meeting" -- to at least briefly go over the main issues we see
facing the project to prime the pump, get a better idea about what we
want to accomplish at the meeting itself, and gather some early
feedback from anyone who won't be able to make it to SciPy (we'll miss
you).
http://doodle.com/6b4s6thqt9xt4vnh
Depending on the interest level, I'm thinking we'll either use Google
Hangouts or Bluejeans (https://bluejeans.com/ -- same as what Ralf
used for the similar SciPy meeting a few months ago; needs a plugin
installed but is available for Windows / OS X / 64-bit Linux / Android
/ iOS, or regular telephone, or h323 softphone).
-n
--
Nathaniel J. Smith -- http://vorpus.org

[Honi Sanders]

I’m interested in listening in just to see what it’s like, but I have to leave after ~15 minutes because I have a meeting at 4:30. Is that too disruptive?

Post by Nathaniel Smith
Hi all,
Meeting is starting in a few minutes!
https://plus.google.com/hangouts/_/gxtuplvm6g7s55abhjexqerll4a
https://docs.google.com/document/d/11KC2p3cCsbDVjLcQSCehUiWGyWDNCyOunKfrO7Q7m3E/edit?usp=sharing
https://github.com/numpy/numpy/wiki/SciPy-2015-developer-meeting
-n

Post by Nathaniel Smith
Hi all,
https://github.com/numpy/numpy/wiki/SciPy-2015-developer-meeting
It's somewhat short notice (my bad :-/), but I think it would be good
to have a short video meeting sometime this week as a kind of
"pre-meeting" -- to at least briefly go over the main issues we see
facing the project to prime the pump, get a better idea about what we
want to accomplish at the meeting itself, and gather some early
feedback from anyone who won't be able to make it to SciPy (we'll miss
you).
http://doodle.com/6b4s6thqt9xt4vnh
Depending on the interest level, I'm thinking we'll either use Google
Hangouts or Bluejeans (https://bluejeans.com/ -- same as what Ralf
used for the similar SciPy meeting a few months ago; needs a plugin
installed but is available for Windows / OS X / 64-bit Linux / Android
/ iOS, or regular telephone, or h323 softphone).
-n
--
Nathaniel J. Smith -- http://vorpus.org

--
Nathaniel J. Smith -- http://vorpus.org
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Ralf Gommers]

I’m interested in listening in just to see what it’s like, but I have to
leave after ~15 minutes because I have a meeting at 4:30. Is that too
disruptive?

Don't worry about it, just join as long as you can.

Ralf

Post by Nathaniel Smith
Hi all,
Meeting is starting in a few minutes!
https://plus.google.com/hangouts/_/gxtuplvm6g7s55abhjexqerll4a

https://docs.google.com/document/d/11KC2p3cCsbDVjLcQSCehUiWGyWDNCyOunKfrO7Q7m3E/edit?usp=sharing

Post by Nathaniel Smith
https://github.com/numpy/numpy/wiki/SciPy-2015-developer-meeting
-n

Post by Nathaniel Smith
Hi all,
https://github.com/numpy/numpy/wiki/SciPy-2015-developer-meeting
It's somewhat short notice (my bad :-/), but I think it would be good
to have a short video meeting sometime this week as a kind of
"pre-meeting" -- to at least briefly go over the main issues we see
facing the project to prime the pump, get a better idea about what we
want to accomplish at the meeting itself, and gather some early
feedback from anyone who won't be able to make it to SciPy (we'll miss
you).
http://doodle.com/6b4s6thqt9xt4vnh
Depending on the interest level, I'm thinking we'll either use Google
Hangouts or Bluejeans (https://bluejeans.com/ -- same as what Ralf
used for the similar SciPy meeting a few months ago; needs a plugin
installed but is available for Windows / OS X / 64-bit Linux / Android
/ iOS, or regular telephone, or h323 softphone).
-n
--
Nathaniel J. Smith -- http://vorpus.org

--
Nathaniel J. Smith -- http://vorpus.org
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Carl Kleffner]

unfortunately I can't start the hangout. Both Firefox and chrome hangs.
Tried it again and again.

For this reason a short status:

I can now build the mingwpy toolchain as pip installable wheel.

%USERPROFILE%\pydistutils.cfg should be configured to use the mingw
compiler.

With the preliminary mingwpy wheels deployed at binstar: (2.7, 3.3, 3.4 /
amd64, win32):

pip install -i https://pypi.binstar.org/carlkl/simple mingwpy

the toolchain can be installed.

Test i.e. it with bottleneck:

pip install bottleneck

compiles and installs the bottleneck package on the fly.

I would like to add some documenation to the package and wait for some
feedback from others before deployment on pypi. Supplementary libraries
(like OpenBLAS) should be provided as individual packages.

I will need some time at the weekend to put all this stuff on github as
well as many of my usual manual steps had to be scripted first.

There is a lot of my TODO list:

- further work numpy, scipy compiled with mingwpy and OpenBLAS
- setup documenation and mingwpy.org
- deploy on PYPI
- win32 bugs in the math code of mingwpw
- precision failures with scipy.test
- alternative implemention of some transz. numeric functions in minw-w64

Wait for your feedback

Cheers

Carl

Post by Ralf Gommers

I’m interested in listening in just to see what it’s like, but I have to
leave after ~15 minutes because I have a meeting at 4:30. Is that too
disruptive?

Don't worry about it, just join as long as you can.
Ralf

Post by Nathaniel Smith
Hi all,
Meeting is starting in a few minutes!
https://plus.google.com/hangouts/_/gxtuplvm6g7s55abhjexqerll4a

https://docs.google.com/document/d/11KC2p3cCsbDVjLcQSCehUiWGyWDNCyOunKfrO7Q7m3E/edit?usp=sharing

Post by Nathaniel Smith
https://github.com/numpy/numpy/wiki/SciPy-2015-developer-meeting
-n

Post by Nathaniel Smith
Hi all,
https://github.com/numpy/numpy/wiki/SciPy-2015-developer-meeting
It's somewhat short notice (my bad :-/), but I think it would be good
to have a short video meeting sometime this week as a kind of
"pre-meeting" -- to at least briefly go over the main issues we see
facing the project to prime the pump, get a better idea about what we
want to accomplish at the meeting itself, and gather some early
feedback from anyone who won't be able to make it to SciPy (we'll miss
you).
http://doodle.com/6b4s6thqt9xt4vnh
Depending on the interest level, I'm thinking we'll either use Google
Hangouts or Bluejeans (https://bluejeans.com/ -- same as what Ralf
used for the similar SciPy meeting a few months ago; needs a plugin
installed but is available for Windows / OS X / 64-bit Linux / Android
/ iOS, or regular telephone, or h323 softphone).
-n
--
Nathaniel J. Smith -- http://vorpus.org

--
Nathaniel J. Smith -- http://vorpus.org
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Olivier Grisel]

Hi Carl,

Sorry for the slow reply.

I ran some tests with your binstar packages:

I installed numpy, scipy and mingwpy for Python 2.7 32 bit and Python
3.4 64 bit (downloaded from python.org) on a freshly provisionned
windows VM on rackspace.

I then used the mingwpy C & C++ compilers to build the master branch
of scikit-learn which worked without any issue.

Then I ran the tests for numpy, scipy and scikit-learn

Here are the results:

https://gist.github.com/ogrisel/7542a56d5740100e66b9

To summarize:

- there are a few numpy failures on both 32 bit and 64 bit (see gist
for details)
- there are scipy failures on 32 bit and a segfault on 64 bit, I
installed gdb to get a not very informative backtrace
- scikit-learn tests all pass on 32 bit
- scikit-learn tests segfault on 64 bit (see backtrace as well).

Let met know how I can help debug those. I installed gdb using msys2
for 64 bit windows but I did not have msys2 in the path when building
scikit-learn.

I can also grant you rackspace cloud credentials if you want to debug
this by yourself.

[Olivier Grisel]

Good news,

The segfaults on scikit-lern and scipy test suites are caused by a bug
in openblas core type detection: setting the OPENBLAS_CORETYPE
environment variable to "Nehalem" can make the test suite complete
without any failure for scikit-learn.

I will update my gist with the new test results for scipy.

[Olivier Grisel]

I have updated my gist with more test reports when
OPENBLAS_CORETYPE="Nehalem" is fixed as an environment variable.

Note that on this machine, OpenBLAS detects the "Barcelona" core type.
I used the following ctypes based script to introspect the OpenBLAS
runtime:

https://gist.github.com/ogrisel/ad4e547a32d0eb18b4ff

[Carl Kleffner]

Hi Olivier,

yes, this is all explained in
https://github.com/xianyi/OpenBLAS/wiki/Faq#choose_target_dynamic as well.
This seems to be necessary for CI systems, right?

BTW: i just now renewed the numpy-1.9.2 and scipy-0.15.0 wheels for
python-2.6, 2.7, 3.3, 3.4 on anaconda.org. I also added scipy-0.16.0rc1 for
python-2.6, 2.7, 3.3, 3.4. This means I phase out python-3.2 builds for the
future. python-3.5 support is up in the air right now.

Cheers

Carl

Post by Olivier Grisel
I have updated my gist with more test reports when
OPENBLAS_CORETYPE="Nehalem" is fixed as an environment variable.
Note that on this machine, OpenBLAS detects the "Barcelona" core type.
I used the following ctypes based script to introspect the OpenBLAS
https://gist.github.com/ogrisel/ad4e547a32d0eb18b4ff
--
Olivier
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Olivier Grisel]

I narrowed down the segfault from the scipy tests on my machine to:

OPENBLAS_CORETYPE='Barcelona' /c/Python34_x64/python -c"import numpy
as np; print(np.linalg.svd(np.ones((129, 129), dtype=np.float64))"

Barcelona is the architecture detected by OpenBLAS. If I force Nehalem
or if I reduce the matrix to shape (128, 128), it no longer segfaults.
To debug it further I think I have to write a test program in C that
uses the sgesdd function of the libopenblaspy.dll or against the
official libopenblas.dll. Setting the dev environment under windows is
painful though.

I don't know if it's a bug in OpenBLAS' coretype detection or in the
SGESDD kernel for the Barcelona architecture.

[Nathaniel Smith]

Post by Olivier Grisel
OPENBLAS_CORETYPE='Barcelona' /c/Python34_x64/python -c"import numpy
as np; print(np.linalg.svd(np.ones((129, 129), dtype=np.float64))"
Barcelona is the architecture detected by OpenBLAS. If I force Nehalem
or if I reduce the matrix to shape (128, 128), it no longer segfaults.
To debug it further I think I have to write a test program in C that
uses the sgesdd function of the libopenblaspy.dll or against the
official libopenblas.dll. Setting the dev environment under windows is
painful though.

I assume you've already checked that this is a Windows specific issue?

-n

[Carl Kleffner]

I could provide you with a debug build of libopenblaspy.dll. The segfault -
if ithrown from openblas - could be detected with gdb or with the help of
backtrace.dll.

Carl

Post by Nathaniel Smith

Post by Olivier Grisel
OPENBLAS_CORETYPE='Barcelona' /c/Python34_x64/python -c"import numpy
as np; print(np.linalg.svd(np.ones((129, 129), dtype=np.float64))"
Barcelona is the architecture detected by OpenBLAS. If I force Nehalem
or if I reduce the matrix to shape (128, 128), it no longer segfaults.
To debug it further I think I have to write a test program in C that
uses the sgesdd function of the libopenblaspy.dll or against the
official libopenblas.dll. Setting the dev environment under windows is
painful though.

I assume you've already checked that this is a Windows specific issue?
-n
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Olivier Grisel]

Post by Carl Kleffner
I could provide you with a debug build of libopenblaspy.dll. The segfault -
if ithrown from openblas - could be detected with gdb or with the help of
backtrace.dll.

That would be great thanks. Also can you give the build options /
instructions you used to build openblas with mingwpy? Also which
version of openblas did you use? The last stable or some snapshot from
the develop branch?

[Olivier Grisel]

Post by Olivier Grisel

Post by Carl Kleffner
I could provide you with a debug build of libopenblaspy.dll. The segfault -
if ithrown from openblas - could be detected with gdb or with the help of
backtrace.dll.

That would be great thanks. Also can you give the build options /
instructions you used to build openblas with mingwpy? Also which
version of openblas did you use? The last stable or some snapshot from
the develop branch?

I just saw you replied to the question on the version number in your last email.

[Olivier Grisel]

Post by Nathaniel Smith

Post by Olivier Grisel
OPENBLAS_CORETYPE='Barcelona' /c/Python34_x64/python -c"import numpy
as np; print(np.linalg.svd(np.ones((129, 129), dtype=np.float64))"
Barcelona is the architecture detected by OpenBLAS. If I force Nehalem
or if I reduce the matrix to shape (128, 128), it no longer segfaults.
To debug it further I think I have to write a test program in C that
uses the sgesdd function of the libopenblaspy.dll or against the
official libopenblas.dll. Setting the dev environment under windows is
painful though.

I assume you've already checked that this is a Windows specific issue?

I am starting a rackspace VM with linux to check. Hopefully it will
also be detected as Barcelona by openblas.

On my laptop I have a Sandybridge coretype (I just checked).

[Olivier Grisel]

Post by Olivier Grisel

Post by Nathaniel Smith
I assume you've already checked that this is a Windows specific issue?

I am starting a rackspace VM with linux to check. Hopefully it will
also be detected as Barcelona by openblas.

I just built OpenBLAS 0.2.14 and numpy 1.9.2 under Linux on a
rackspace VM (2GB Standard Instance) that gets detected as "Barcelona"
by OpenBLAS and I cannot reproduce the issue.

So this is either:

- windows specific
- mingpy specific (as it was used to build numpy and potentially the
embedded libopenblaspy.dll as well)
- specific to an old version of OpenBLAS and fixed in 0.2.14 if Carl
has used an old version

Note bigger rackspace VMs get detected as Nehalem. So use 2GB Standard
Instance if you want a Barcelona machine.

[Olivier Grisel]

Post by Carl Kleffner
Hi Olivier,
yes, this is all explained in
https://github.com/xianyi/OpenBLAS/wiki/Faq#choose_target_dynamic as well.
This seems to be necessary for CI systems, right?

The auto detection should work. If not it's a bug and we should find a
minimalistic reproduction case to report it to the openblas
maintainers.

Or we could choose to build openblas for an old architecture that
should work everywhere like nehalem for instance.

Post by Carl Kleffner
BTW: i just now renewed the numpy-1.9.2 and scipy-0.15.0 wheels for
python-2.6, 2.7, 3.3, 3.4 on anaconda.org.

You mean binstar.org right? What did you change in the new
numpy-1.9.2? Have you upgraded OpenBLAS? Which version is embedded by
the way?

Post by Carl Kleffner
I also added scipy-0.16.0rc1 for
python-2.6, 2.7, 3.3, 3.4. This means I phase out python-3.2 builds for the
future.

I don't think Windows users care about python 3.2 support.

Post by Carl Kleffner
python-3.5 support is up in the air right now.

What do you mean?

[Carl Kleffner]

Post by Olivier Grisel

Post by Carl Kleffner
Hi Olivier,
yes, this is all explained in
https://github.com/xianyi/OpenBLAS/wiki/Faq#choose_target_dynamic as

well.

Post by Carl Kleffner
This seems to be necessary for CI systems, right?

The auto detection should work. If not it's a bug and we should find a
minimalistic reproduction case to report it to the openblas
maintainers.
I have a debug version of libopenblaspy.dll at hand for 32bit

architecture, 64bit I have to build. I propose to build small testcases
with python, as @wernsaar, the main developer of the newer OpenBLAS kernels
is able to handle this.

Debug builds of libopenblaspy.dll can be used as drop in replacement and
can be used to emit a stack trace as long the segfault is thrown from numpy
or OpenBLAS code. DrMingw may be able to catch segfaults from either, I
havn't tried this yet.

Post by Olivier Grisel
Or we could choose to build openblas for an old architecture that
should work everywhere like nehalem for instance.

This is a safe solution. Or one could leave out all kernels above
sandybridge in a dynamic build as long OpenBLAS has errors with newer
kernels.

Post by Olivier Grisel

Post by Carl Kleffner
BTW: i just now renewed the numpy-1.9.2 and scipy-0.15.0 wheels for
python-2.6, 2.7, 3.3, 3.4 on anaconda.org.

The best technical solution IMHO would be to make an extra mingwpy.openblas
python package to load libopenblaspy.dll into the process space, as this
package could be upgraded independant from numpy/scipy. On the other side
this would mean an additionally package dependancy for numpy at least on
the windows platform.

Post by Olivier Grisel
You mean binstar.org right? What did you change in the new
numpy-1.9.2? Have you upgraded OpenBLAS? Which version is embedded by
the way?

anaconda.org is the new replacement for binstar.org as announced in the
anaconda ml and should be used right now.

Openblas versions contained:
32bit: 3e33afe (june)
64bit: fb02cb0 (april used due to a segfault in the more recent rev)

Post by Olivier Grisel

Post by Carl Kleffner
I also added scipy-0.16.0rc1 for
python-2.6, 2.7, 3.3, 3.4. This means I phase out python-3.2 builds for

the

Post by Carl Kleffner
future.

I don't think Windows users care about python 3.2 support.

Post by Carl Kleffner
python-3.5 support is up in the air right now.

What do you mean?
Python-3.5 uses VC 2105 on windows. This compiler uses a complete new

variant of CRT libraries. Google for 'universal CRT' to find more
information. The mingw-w64 community does not support this right now. I
didn't find the time to explore this.

Carl

--

Post by Olivier Grisel
Olivier
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Olivier Grisel]

Post by Olivier Grisel

Post by Carl Kleffner
Hi Olivier,
yes, this is all explained in
https://github.com/xianyi/OpenBLAS/wiki/Faq#choose_target_dynamic as well.
This seems to be necessary for CI systems, right?

The auto detection should work. If not it's a bug and we should find a
minimalistic reproduction case to report it to the openblas
maintainers.

I have a debug version of libopenblaspy.dll at hand for 32bit architecture,
64bit I have to build. I propose to build small testcases with python, as
@wernsaar, the main developer of the newer OpenBLAS kernels is able to
handle this.

Done here, feel free to join the discussion:

https://github.com/xianyi/OpenBLAS/issues/603

Debug builds of libopenblaspy.dll can be used as drop in replacement and can
be used to emit a stack trace as long the segfault is thrown from numpy or
OpenBLAS code. DrMingw may be able to catch segfaults from either, I havn't
tried this yet.

Post by Olivier Grisel
Or we could choose to build openblas for an old architecture that
should work everywhere like nehalem for instance.

This is a safe solution.

Is Nehalem the most common denominator?

Or one could leave out all kernels above
sandybridge in a dynamic build as long OpenBLAS has errors with newer
kernels.

Yes that's an option. How do you do that? I cannot find such a build
option in the documentation.

The best technical solution IMHO would be to make an extra mingwpy.openblas
python package to load libopenblaspy.dll into the process space, as this
package could be upgraded independant from numpy/scipy. On the other side
this would mean an additionally package dependancy for numpy at least on the
windows platform.

Having a windows only dependency on such a wheel sounds fine with me.
However that would mean to ensure that the dll is properly loaded in
the windows dll search path prior to importing extensions such as
numpy.core._dotblas right? Could this be done via adding something to
`numpy.__init__.py` such as:

```
if sys.platform == 'win32':
try:
# Ensure that libopenblaspy.dll can be found prior to
# loading numpy.core._dotblas
from mingwpy import openblas
except:
warnings.warn('Failed to load mingwpy's openblas')

# import numpy.core stuff here
```

anaconda.org is the new replacement for binstar.org as announced in the
anaconda ml and should be used right now.

Ok I did not know.

[Carl Kleffner]

Post by Carl Kleffner

Post by Olivier Grisel

Post by Olivier Grisel

Post by Carl Kleffner
Hi Olivier,
yes, this is all explained in
https://github.com/xianyi/OpenBLAS/wiki/Faq#choose_target_dynamic as well.
This seems to be necessary for CI systems, right?

The auto detection should work. If not it's a bug and we should find a
minimalistic reproduction case to report it to the openblas
maintainers.

I have a debug version of libopenblaspy.dll at hand for 32bit

architecture,

Post by Olivier Grisel
64bit I have to build. I propose to build small testcases with python, as
@wernsaar, the main developer of the newer OpenBLAS kernels is able to
handle this.

https://github.com/xianyi/OpenBLAS/issues/603

Post by Olivier Grisel
Debug builds of libopenblaspy.dll can be used as drop in replacement and

can

Post by Olivier Grisel
be used to emit a stack trace as long the segfault is thrown from numpy

or

Post by Olivier Grisel
OpenBLAS code. DrMingw may be able to catch segfaults from either, I

havn't

Post by Olivier Grisel
tried this yet.

Post by Olivier Grisel
Or we could choose to build openblas for an old architecture that
should work everywhere like nehalem for instance.

This is a safe solution.

Is Nehalem the most common denominator?
I think it is PRESCOTT. BTW: For now you could use now OPTERON_SSE3 or

OPTERON instead of NEHALEM, as these are AMD kernels without the barcelona
assembler part.

Post by Carl Kleffner

Post by Olivier Grisel
Or one could leave out all kernels above
sandybridge in a dynamic build as long OpenBLAS has errors with newer
kernels.

Yes that's an option. How do you do that? I cannot find such a build
option in the documentation.
Just patches to be include as an option later on.

Post by Olivier Grisel
The best technical solution IMHO would be to make an extra

mingwpy.openblas

Post by Olivier Grisel
python package to load libopenblaspy.dll into the process space, as this
package could be upgraded independant from numpy/scipy. On the other side
this would mean an additionally package dependancy for numpy at least on

the

Post by Olivier Grisel
windows platform.

Having a windows only dependency on such a wheel sounds fine with me.
However that would mean to ensure that the dll is properly loaded in
the windows dll search path prior to importing extensions such as
numpy.core._dotblas right? Could this be done via adding something to
```
# Ensure that libopenblaspy.dll can be found prior to
# loading numpy.core._dotblas
from mingwpy import openblas
warnings.warn('Failed to load mingwpy's openblas')
# import numpy.core stuff here
```
Or equip numpy and scipy with the DLL as a fallback, if no 'package' for

OpenBLAS is installed. With an additional OpenBLAS package you then can
augument the numpy OpenBLAS with a newer one.

Post by Carl Kleffner

Post by Olivier Grisel
anaconda.org is the new replacement for binstar.org as announced in the
anaconda ml and should be used right now.

Ok I did not know.
--
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Nathaniel Smith]

Post by Olivier Grisel

Post by Carl Kleffner
The best technical solution IMHO would be to make an extra

mingwpy.openblas

Post by Olivier Grisel

Post by Carl Kleffner
python package to load libopenblaspy.dll into the process space, as this
package could be upgraded independant from numpy/scipy. On the other side
this would mean an additionally package dependancy for numpy at least on the
windows platform.

Having a windows only dependency on such a wheel sounds fine with me.
However that would mean to ensure that the dll is properly loaded in
the windows dll search path prior to importing extensions such as
numpy.core._dotblas right? Could this be done via adding something to
```
# Ensure that libopenblaspy.dll can be found prior to
# loading numpy.core._dotblas
from mingwpy import openblas
warnings.warn('Failed to load mingwpy's openblas')
# import numpy.core stuff here
```

We might even just want to add to numpy/__init__.py something like

try:
import .distributor_hook
except ImportError:
pass

so wheels (and conda builds, etc.) could be patched by adding a single file
without modifying any upstream files.

-n

[Eric Moore]

It looks like all of the numpy failures there are due to a poor
implementation of hypot. One solution would be to force the use of the
hypot code in npymath for this tool chain. Currently this is done in
numpy/core/src/private/npy_config.h for both MSVC and mingw32.

-Eric

Post by Olivier Grisel
Good news,
The segfaults on scikit-lern and scipy test suites are caused by a bug
in openblas core type detection: setting the OPENBLAS_CORETYPE
environment variable to "Nehalem" can make the test suite complete
without any failure for scikit-learn.
I will update my gist with the new test results for scipy.
--
Olivier
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion