[Numpy-discussion] PyPy wheels (was: NumPy 1.12.0b1 released)

[Ralf Gommers]

Another thing to think about is that 1.12 on pypy won't pass its test
suite (though it's close), and we're not yet testing new PRs on pypy, so no
guarantees about 1.13 yet. I think on balance these probably aren't reasons
*not* to upload wheels, but it's a funny place where we're talking about
providing "official" builds even though it's not an "officially supported
platform". So we will at least want to be clear about that. And someone
will have to handle the bug reports about the test suite failing :-).

Those are good points. We could run PyPy on TravisCI; the PyPy install and
numpy build aren't difficult anymore.

Handling bug reports is mostly checking if it's PyPy specific, and if so
refer to the PyPy tracker I'd think. It is some work, but given that PyPy
has finally chosen a way to support Numpy that's not a dead end and has
come quite a long way quite quickly, taking on that bit of extra work as
Numpy maintainers is a good time investment imho.

Many bug reports will go straight to PyPy though I expect, because often
that is the obvious place to go. This is what I just got from downloading
the OS X PyPy binary and pip installing numpy master:

Python version 2.7.12 (aff251e54385, Nov 09 2016, 17:25:49)[PyPy 5.6.0 with
GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)]
nose version 1.3.7
.................................S.......................................................E........................................................................................S.............................................................RPython
traceback:
File "pypy_interpreter.c", line 43348, in
BuiltinCodePassThroughArguments1_funcrun_obj
File "pypy_module_cpyext_4.c", line 16627, in
generic_cpy_call__StdObjSpaceConst_funcPtr_SomeI_17
Fatal RPython error: AssertionError
Abort trap: 6

So guess I'll go find out where that issue tracker is:)

Ralf

Hi,

Is there a guide to building standard wheels for NumPy?

I don't think so - there is a repository that we use to build the
wheels, that has the Windows, OSX and manyllinux recipes for the
https://github.com/MacPython/numpy-wheelso
If you can work out a way to automate the PyPy builds and tests -
especially using the same repo - that would be very useful.

Assuming I can build standardized PyPy 2.7 wheels for Ubuntu, Win32

and

OSX64, how can I get them blessed and uploaded to PyPI?

If you can automate the build and tests, I'm guessing there will be no
objections - but it's not my call...

I'm in favor, assuming that the wheel tags and PyPy backwards

compatibility situation is OK. Can't really find any examples. What I mean
is that for CPython wheels contain tags like "cp27" or "cp35". PyPy wheels
should have tags "pp<something>". Are the PyPy cpyext layer and the
<something> defined such that a new PyPy release won't break older wheels?

[Nathaniel Smith]

Post by Ralf Gommers

Another thing to think about is that 1.12 on pypy won't pass its test

suite (though it's close), and we're not yet testing new PRs on pypy, so no
guarantees about 1.13 yet. I think on balance these probably aren't reasons
*not* to upload wheels, but it's a funny place where we're talking about
providing "official" builds even though it's not an "officially supported
platform". So we will at least want to be clear about that. And someone
will have to handle the bug reports about the test suite failing :-).

Post by Ralf Gommers
Those are good points. We could run PyPy on TravisCI; the PyPy install

and numpy build aren't difficult anymore.

I'm not sure how useful this until the test suite is passing, though, just
because of how travis-ci works.

The main outstanding issue needs some work on the numpy side: basically
UPDATEIFCOPY, as currently conceived, just can't work reliably on pypy,
because it assumes that Python objects get deterministically deallocated as
soon as their reference count drops to zero, and pypy doesn't have
reference counts.

I think fixing this should be straightforward enough, in case anyone wants
to work on it. Every user of UPDATEIFCOPY already has to be aware of the
reference counts and know when the pseudo-"view" array is supposed to be
deallocated. So I think we should define a new UPDATEIFCOPY2 flag, which
acts like the current UPDATEIFCOPY, except that instead of using __del__ to
do the writeback, there's an explicit API call you have to make, like
PyArray_UpdateIfCopy2_Writeback, that checks for the flag and does the
writeback if set. Then we should transition to using this internally, and
probably deprecate UPDATEIFCOPY (though we may never be able to get rid of
it entirely).

-n

[Ralf Gommers]

Post by Ralf Gommers

Another thing to think about is that 1.12 on pypy won't pass its test

suite (though it's close), and we're not yet testing new PRs on pypy, so no
guarantees about 1.13 yet. I think on balance these probably aren't reasons
*not* to upload wheels, but it's a funny place where we're talking about
providing "official" builds even though it's not an "officially supported
platform". So we will at least want to be clear about that. And someone
will have to handle the bug reports about the test suite failing :-).

Post by Ralf Gommers
Those are good points. We could run PyPy on TravisCI; the PyPy install

and numpy build aren't difficult anymore.
I'm not sure how useful this until the test suite is passing, though, just
because of how travis-ci works.

It's not hard to skip the currently failing tests only on a single run in
the build matrix. That would keep TravisCI green and ensure there's no
regressions.

Ralf

The main outstanding issue needs some work on the numpy side: basically
UPDATEIFCOPY, as currently conceived, just can't work reliably on pypy,
because it assumes that Python objects get deterministically deallocated as
soon as their reference count drops to zero, and pypy doesn't have
reference counts.
I think fixing this should be straightforward enough, in case anyone wants
to work on it. Every user of UPDATEIFCOPY already has to be aware of the
reference counts and know when the pseudo-"view" array is supposed to be
deallocated. So I think we should define a new UPDATEIFCOPY2 flag, which
acts like the current UPDATEIFCOPY, except that instead of using __del__ to
do the writeback, there's an explicit API call you have to make, like
PyArray_UpdateIfCopy2_Writeback, that checks for the flag and does the
writeback if set. Then we should transition to using this internally, and
probably deprecate UPDATEIFCOPY (though we may never be able to get rid of
it entirely).
-n
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion