Discussion:
[Numpy-discussion] mtrand.c update 1.11 breaks my crappy code
Neal Becker
2016-04-06 13:18:24 UTC
Permalink
I have C++ code that tries to share the mtrand state. It unfortunately
depends on the layout of RandomState which used to be:

struct __pyx_obj_6mtrand_RandomState {
PyObject_HEAD
rk_state *internal_state;
PyObject *lock;
};

But with 1.11 it's:
struct __pyx_obj_6mtrand_RandomState {
PyObject_HEAD
struct __pyx_vtabstruct_6mtrand_RandomState *__pyx_vtab;
rk_state *internal_state;
PyObject *lock;
PyObject *state_address;
};

So
1. Why the change?
2. How can I write portable code?
Robert Kern
2016-04-06 13:29:57 UTC
Permalink
Post by Neal Becker
I have C++ code that tries to share the mtrand state. It unfortunately
struct __pyx_obj_6mtrand_RandomState {
PyObject_HEAD
rk_state *internal_state;
PyObject *lock;
};
struct __pyx_obj_6mtrand_RandomState {
PyObject_HEAD
struct __pyx_vtabstruct_6mtrand_RandomState *__pyx_vtab;
rk_state *internal_state;
PyObject *lock;
PyObject *state_address;
};
So
1. Why the change?
2. How can I write portable code?
There is no C API to RandomState at this time, stable, portable or
otherwise. It's all private implementation detail. If you would like a
stable and portable C API for RandomState, you will need to contribute one
using PyCapsules to expose the underlying rk_state* pointer.

https://docs.python.org/2.7/c-api/capsule.html

--
Robert Kern
Neal Becker
2016-04-06 13:49:36 UTC
Permalink
Post by Robert Kern
Post by Neal Becker
I have C++ code that tries to share the mtrand state. It unfortunately
struct __pyx_obj_6mtrand_RandomState {
PyObject_HEAD
rk_state *internal_state;
PyObject *lock;
};
struct __pyx_obj_6mtrand_RandomState {
PyObject_HEAD
struct __pyx_vtabstruct_6mtrand_RandomState *__pyx_vtab;
rk_state *internal_state;
PyObject *lock;
PyObject *state_address;
};
So
1. Why the change?
2. How can I write portable code?
There is no C API to RandomState at this time, stable, portable or
otherwise. It's all private implementation detail. If you would like a
stable and portable C API for RandomState, you will need to contribute one
using PyCapsules to expose the underlying rk_state* pointer.
https://docs.python.org/2.7/c-api/capsule.html
--
Robert Kern
I don't see how pycapsule helps here. What I need is, my C++ code receives
a RandomState object. I need to call e.g., rk_random, passing the pointer
to rk_state - code looks like this;

RandomState* r = (RandomState*)(rs.ptr());
// result_type buffer;
// rk_fill ((void*)&buffer, sizeof(buffer), r->internal_state);
if (sizeof(result_type) == sizeof (uint64_t))
return rk_ulong (r->internal_state);
else if (sizeof(result_type) == sizeof (uint32_t))
return rk_random (r->internal_state);
Neal Becker
2016-04-06 14:02:37 UTC
Permalink
Post by Neal Becker
Post by Robert Kern
Post by Neal Becker
I have C++ code that tries to share the mtrand state. It unfortunately
struct __pyx_obj_6mtrand_RandomState {
PyObject_HEAD
rk_state *internal_state;
PyObject *lock;
};
struct __pyx_obj_6mtrand_RandomState {
PyObject_HEAD
struct __pyx_vtabstruct_6mtrand_RandomState *__pyx_vtab;
rk_state *internal_state;
PyObject *lock;
PyObject *state_address;
};
So
1. Why the change?
2. How can I write portable code?
There is no C API to RandomState at this time, stable, portable or
otherwise. It's all private implementation detail. If you would like a
stable and portable C API for RandomState, you will need to contribute
one using PyCapsules to expose the underlying rk_state* pointer.
https://docs.python.org/2.7/c-api/capsule.html
--
Robert Kern
I don't see how pycapsule helps here. What I need is, my C++ code receives
a RandomState object. I need to call e.g., rk_random, passing the pointer
to rk_state - code looks like this;
RandomState* r = (RandomState*)(rs.ptr());
// result_type buffer;
// rk_fill ((void*)&buffer, sizeof(buffer), r->internal_state);
if (sizeof(result_type) == sizeof (uint64_t))
return rk_ulong (r->internal_state);
else if (sizeof(result_type) == sizeof (uint32_t))
return rk_random (r->internal_state);
Nevermind, I see it's described here:
https://docs.python.org/2.7/extending/extending.html#using-capsules
Nathaniel Smith
2016-04-06 14:31:32 UTC
Permalink
Post by Robert Kern
Post by Neal Becker
I have C++ code that tries to share the mtrand state. It unfortunately
struct __pyx_obj_6mtrand_RandomState {
PyObject_HEAD
rk_state *internal_state;
PyObject *lock;
};
struct __pyx_obj_6mtrand_RandomState {
PyObject_HEAD
struct __pyx_vtabstruct_6mtrand_RandomState *__pyx_vtab;
rk_state *internal_state;
PyObject *lock;
PyObject *state_address;
};
So
1. Why the change?
2. How can I write portable code?
There is no C API to RandomState at this time, stable, portable or
otherwise. It's all private implementation detail. If you would like a
stable and portable C API for RandomState, you will need to contribute one
using PyCapsules to expose the underlying rk_state* pointer.
Post by Robert Kern
https://docs.python.org/2.7/c-api/capsule.html
I'm very wary about the idea of exposing the rk_state pointer at all. We
could have a C API to random but my strong preference would be for
something that only exposes opaque function calls that take a RandomState
and return some random numbers, and getting even this right in a clean and
maintainable way isn't trivial.

Obviously another option is to call one of the python methods to get an
ndarray and read out its memory contents. If you can do this in a batch
(fetching a bunch of numbers for each call) to amortize the additional
overhead of going through python, then it might work fine. (Python overhead
is not actually that much -- mostly just having to do a handful of extra
allocations.)

Or, possibly the best option, one could use one of the many fine C random
libraries inside your code, and if you need your code to be deterministic
given a RandomState you could derive your state initialization from a
single call to some RandomState method.

-n
Neal Becker
2016-04-06 15:17:20 UTC
Permalink
Post by Robert Kern
Post by Robert Kern
Post by Neal Becker
I have C++ code that tries to share the mtrand state. It unfortunately
struct __pyx_obj_6mtrand_RandomState {
PyObject_HEAD
rk_state *internal_state;
PyObject *lock;
};
struct __pyx_obj_6mtrand_RandomState {
PyObject_HEAD
struct __pyx_vtabstruct_6mtrand_RandomState *__pyx_vtab;
rk_state *internal_state;
PyObject *lock;
PyObject *state_address;
};
So
1. Why the change?
2. How can I write portable code?
There is no C API to RandomState at this time, stable, portable or
otherwise. It's all private implementation detail. If you would like a
stable and portable C API for RandomState, you will need to contribute one
using PyCapsules to expose the underlying rk_state* pointer.
Post by Robert Kern
https://docs.python.org/2.7/c-api/capsule.html
I'm very wary about the idea of exposing the rk_state pointer at all. We
could have a C API to random but my strong preference would be for
something that only exposes opaque function calls that take a RandomState
and return some random numbers, and getting even this right in a clean and
maintainable way isn't trivial.
Obviously another option is to call one of the python methods to get an
ndarray and read out its memory contents. If you can do this in a batch
(fetching a bunch of numbers for each call) to amortize the additional
overhead of going through python, then it might work fine. (Python
overhead is not actually that much -- mostly just having to do a handful
of extra allocations.)
Or, possibly the best option, one could use one of the many fine C random
libraries inside your code, and if you need your code to be deterministic
given a RandomState you could derive your state initialization from a
single call to some RandomState method.
-n
I prefer to use a single instance of a RandomState so that there are
guarantees about the independence of streams generated from python random
functions, and from my c++ code. True, there are simpler approaches - but
I'm a purist.

Yes, if there were an api use mkl random functions from a RandomState object
that would solve my problem. Or even if there was an API to get a
internal_state pointer from a RandomState object.
Robert Kern
2016-04-06 15:27:49 UTC
Permalink
Post by Neal Becker
I prefer to use a single instance of a RandomState so that there are
guarantees about the independence of streams generated from python random
functions, and from my c++ code. True, there are simpler approaches - but
I'm a purist.
Consider using PRNGs that actually expose truly independent streams instead
of a single shared stream:

https://github.com/bashtage/ng-numpy-randomstate

--
Robert Kern

Loading...