[Numpy-discussion] Vendorize tempita

Discussion:

Charles R Harris

2016-09-30 14:30:10 UTC

Hi All,

There is a PR <https://github.com/numpy/numpy/pull/8096> to vendorize
tempita. This removes tempita as a dependency and simplifies some things.
Feedback on this step is welcome. One question is whether the package
should be renamed to something like `npy_tempita`, as otherwise installed
tempita, if any has priority.

Thoughts?

Chuck

Stephan Hoyer

2016-09-30 15:13:15 UTC

Permalink

One way to do this is to move to vendorized dependencies into an submodule
of numpy itself (e.g., sklearn.externals.joblib, though maybe even a little
more indirection than that would be valuable to make it clear that it isn't
part of NumPy public API). This would avoid further enlarging the set of
namespaces we use.

In any case, I'm perfectly OK with using something like npy_tempita
internally, too, as long as we can be sure that we're using NumPy's
vendorized version, not whatever version is installed locally. We're not
planning to actually install "npy_tempita" when installing numpy (even for
dev installs), right?

Post by Charles R Harris
Hi All,
There is a PR <https://github.com/numpy/numpy/pull/8096> to vendorize
tempita. This removes tempita as a dependency and simplifies some things.
Feedback on this step is welcome. One question is whether the package
should be renamed to something like `npy_tempita`, as otherwise installed
tempita, if any has priority.
Thoughts?
Chuck
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Benjamin Root

2016-09-30 15:21:56 UTC

Permalink

This is the first I am hearing of tempita (looks to be a templating
language). How is it a dependency of numpy? Do I now need tempita in order
to use numpy, or is it a build-time-only dependency?

Ben

Post by Stephan Hoyer
One way to do this is to move to vendorized dependencies into an submodule
of numpy itself (e.g., sklearn.externals.joblib, though maybe even a little
more indirection than that would be valuable to make it clear that it isn't
part of NumPy public API). This would avoid further enlarging the set of
namespaces we use.
In any case, I'm perfectly OK with using something like npy_tempita
internally, too, as long as we can be sure that we're using NumPy's
vendorized version, not whatever version is installed locally. We're not
planning to actually install "npy_tempita" when installing numpy (even for
dev installs), right?
On Fri, Sep 30, 2016 at 7:30 AM, Charles R Harris <

_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Charles R Harris

2016-09-30 15:29:36 UTC

Permalink

Post by Benjamin Root
This is the first I am hearing of tempita (looks to be a templating
language). How is it a dependency of numpy? Do I now need tempita in order
to use numpy, or is it a build-time-only dependency?

Build time only. The virtue of tempita is that it can be used to generate
cython sources. We could adapt one of our current templating scripts to do
that also, but that would seem to be more work. Note that tempita is
currently included in cython, but the cython folks consider that an
implemention detail that should not be depended upon.

<snip>

Chuck

Evgeni Burovski

2016-09-30 15:48:53 UTC

Permalink

On Fri, Sep 30, 2016 at 6:29 PM, Charles R Harris

Post by Charles R Harris

Ideally, it's packaged in such a way that it's usable for scipy too --
at the moment it's used in scipy.sparse via Cython.Tempita + a
fallback to system installed tempita if Cython.Tempita is not
available (however I'm not sure that fallback is ever exercised).
Since scipy needs to support numpy down to 1.8.2, a vendorized copy
will not be usable for scipy for quite a while.

So, it'd be great to handle it like numpydoc: to have npy_tempita as a
small self-contained package with the repo under the numpy
organization and include it via a git submodule. Chuck, do you think
tempita would need much in terms of maintenance?

To put some money where my mouth is, I can offer to do some legwork
for packaging it up.

Cheers,

Evgeni

Charles R Harris

2016-09-30 16:10:38 UTC

Permalink

Post by Evgeni Burovski
On Fri, Sep 30, 2016 at 6:29 PM, Charles R Harris

Post by Charles R Harris

Post by Benjamin Root
This is the first I am hearing of tempita (looks to be a templating
language). How is it a dependency of numpy? Do I now need tempita in

order

Post by Charles R Harris

Post by Benjamin Root
to use numpy, or is it a build-time-only dependency?

Build time only. The virtue of tempita is that it can be used to generate
cython sources. We could adapt one of our current templating scripts to

Post by Charles R Harris
that also, but that would seem to be more work. Note that tempita is
currently included in cython, but the cython folks consider that an
implemention detail that should not be depended upon.
<snip>
Chuck
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Ideally, it's packaged in such a way that it's usable for scipy too --
at the moment it's used in scipy.sparse via Cython.Tempita + a
fallback to system installed tempita if Cython.Tempita is not
available (however I'm not sure that fallback is ever exercised).
Since scipy needs to support numpy down to 1.8.2, a vendorized copy
will not be usable for scipy for quite a while.
So, it'd be great to handle it like numpydoc: to have npy_tempita as a
small self-contained package with the repo under the numpy
organization and include it via a git submodule. Chuck, do you think
tempita would need much in terms of maintenance?
To put some money where my mouth is, I can offer to do some legwork
for packaging it up.

It might be better to keep tempita and cythonize together so that the
search path works out right. It is also possible that other scripts might
be wanted as cythonize is currently restricted to cython files (*.pyx.in, *.
pxi.in). There are two other templating scripts in numpy/distutils, and I
think f2py has a dependency on one of those.

If there is a set of tools that would be common to both scipy and numpy,
having them included as a submodule would be a good idea.

Chuck

Charles R Harris

2016-09-30 16:36:47 UTC

Permalink

On Fri, Sep 30, 2016 at 10:10 AM, Charles R Harris <

On Fri, Sep 30, 2016 at 9:48 AM, Evgeni Burovski <

Post by Evgeni Burovski
On Fri, Sep 30, 2016 at 6:29 PM, Charles R Harris

Post by Charles R Harris

Post by Benjamin Root
This is the first I am hearing of tempita (looks to be a templating
language). How is it a dependency of numpy? Do I now need tempita in

order

Post by Charles R Harris

Post by Benjamin Root
to use numpy, or is it a build-time-only dependency?

Build time only. The virtue of tempita is that it can be used to

generate

Post by Charles R Harris
cython sources. We could adapt one of our current templating scripts to

Ideally, it's packaged in such a way that it's usable for scipy too --
at the moment it's used in scipy.sparse via Cython.Tempita + a
fallback to system installed tempita if Cython.Tempita is not
available (however I'm not sure that fallback is ever exercised).
Since scipy needs to support numpy down to 1.8.2, a vendorized copy
will not be usable for scipy for quite a while.
So, it'd be great to handle it like numpydoc: to have npy_tempita as a
small self-contained package with the repo under the numpy
organization and include it via a git submodule. Chuck, do you think
tempita would need much in terms of maintenance?
To put some money where my mouth is, I can offer to do some legwork
for packaging it up.

It might be better to keep tempita and cythonize together so that the
search path works out right. It is also possible that other scripts might
be wanted as cythonize is currently restricted to cython files (*.pyx.in,
*.pxi.in). There are two other templating scripts in numpy/distutils, and
I think f2py has a dependency on one of those.
If there is a set of tools that would be common to both scipy and numpy,
having them included as a submodule would be a good idea.

Hmm, I suppose it just depends on where submodule is, so a npy_tempita
alone would work fine. There isn't much maintenance needed if you resist
the urge to refactor the code. I removed a six dependency, but that is now
upstream as well.

Chuck

Charles R Harris

2016-10-01 00:42:28 UTC

Permalink

On Fri, Sep 30, 2016 at 10:36 AM, Charles R Harris <

Post by Charles R Harris
On Fri, Sep 30, 2016 at 10:10 AM, Charles R Harris <

On Fri, Sep 30, 2016 at 9:48 AM, Evgeni Burovski <

Post by Evgeni Burovski
On Fri, Sep 30, 2016 at 6:29 PM, Charles R Harris

Post by Charles R Harris

Post by Benjamin Root
This is the first I am hearing of tempita (looks to be a templating
language). How is it a dependency of numpy? Do I now need tempita in

order

Post by Charles R Harris

Post by Benjamin Root
to use numpy, or is it a build-time-only dependency?

Build time only. The virtue of tempita is that it can be used to

generate

Post by Charles R Harris
cython sources. We could adapt one of our current templating scripts

to do

Ideally, it's packaged in such a way that it's usable for scipy too --
at the moment it's used in scipy.sparse via Cython.Tempita + a
fallback to system installed tempita if Cython.Tempita is not
available (however I'm not sure that fallback is ever exercised).
Since scipy needs to support numpy down to 1.8.2, a vendorized copy
will not be usable for scipy for quite a while.
So, it'd be great to handle it like numpydoc: to have npy_tempita as a
small self-contained package with the repo under the numpy
organization and include it via a git submodule. Chuck, do you think
tempita would need much in terms of maintenance?
To put some money where my mouth is, I can offer to do some legwork
for packaging it up.

It might be better to keep tempita and cythonize together so that the
search path works out right. It is also possible that other scripts might
be wanted as cythonize is currently restricted to cython files (*.pyx.in,
*.pxi.in). There are two other templating scripts in numpy/distutils,
and I think f2py has a dependency on one of those.
If there is a set of tools that would be common to both scipy and numpy,
having them included as a submodule would be a good idea.

There don't seem to be any objections, so I will put the current
vendorization in. Evgeni, if you think it a good idea to make a repo for
this and use submodules, go ahead with that. I have left out the testing
infrastructure at https://github.com/gjhiggins/tempita which runs a sparse
set of doctests.

Chuck

Ralf Gommers

2016-10-01 03:08:01 UTC

Permalink

Post by Charles R Harris
On Fri, Sep 30, 2016 at 10:36 AM, Charles R Harris <

Post by Charles R Harris
On Fri, Sep 30, 2016 at 10:10 AM, Charles R Harris <

On Fri, Sep 30, 2016 at 9:48 AM, Evgeni Burovski <

Post by Evgeni Burovski
On Fri, Sep 30, 2016 at 6:29 PM, Charles R Harris

Post by Charles R Harris

Post by Benjamin Root
This is the first I am hearing of tempita (looks to be a templating
language). How is it a dependency of numpy? Do I now need tempita in

order

Post by Charles R Harris

Post by Benjamin Root
to use numpy, or is it a build-time-only dependency?

Build time only. The virtue of tempita is that it can be used to

generate

Post by Charles R Harris
cython sources. We could adapt one of our current templating scripts

to do

Ideally, it's packaged in such a way that it's usable for scipy too --
at the moment it's used in scipy.sparse via Cython.Tempita + a
fallback to system installed tempita if Cython.Tempita is not
available (however I'm not sure that fallback is ever exercised).
Since scipy needs to support numpy down to 1.8.2, a vendorized copy
will not be usable for scipy for quite a while.
So, it'd be great to handle it like numpydoc: to have npy_tempita as a
small self-contained package with the repo under the numpy
organization and include it via a git submodule. Chuck, do you think
tempita would need much in terms of maintenance?
To put some money where my mouth is, I can offer to do some legwork
for packaging it up.

It might be better to keep tempita and cythonize together so that the
search path works out right. It is also possible that other scripts might
be wanted as cythonize is currently restricted to cython files (*.pyx.in,
*.pxi.in). There are two other templating scripts in numpy/distutils,
and I think f2py has a dependency on one of those.
If there is a set of tools that would be common to both scipy and numpy,
having them included as a submodule would be a good idea.

There don't seem to be any objections, so I will put the current
vendorization in.

LGTM as is. tools/ seems to be the right place, its outside the numpy
package so no one can import it as numpy.something, which is better than a
numpy.externals or numpy.vendor submodule.

Post by Charles R Harris
Evgeni, if you think it a good idea to make a repo for this and use
submodules, go ahead with that. I have left out the testing infrastructure
at https://github.com/gjhiggins/tempita which runs a sparse set of
doctests.

There's a problem with that: it will then not be possible to do:
git clone ...
python setup.py install # or equivalent
We shouldn't force everyone to mess with "git submodule" for this. I
suspect a submodule would also break a pip install directly from github.
There'll be very few (if any) changes from upstream Tempita, so making a
reusable npy_tempita seems premature.

Cheers,
Ralf

Evgeni Burovski

2016-10-01 19:54:01 UTC

Permalink

Post by Charles R Harris
On Fri, Sep 30, 2016 at 10:36 AM, Charles R Harris <

Post by Charles R Harris
On Fri, Sep 30, 2016 at 10:10 AM, Charles R Harris <

On Fri, Sep 30, 2016 at 9:48 AM, Evgeni Burovski <

Post by Evgeni Burovski
On Fri, Sep 30, 2016 at 6:29 PM, Charles R Harris

Post by Charles R Harris

Ideally, it's packaged in such a way that it's usable for scipy too --
at the moment it's used in scipy.sparse via Cython.Tempita + a
fallback to system installed tempita if Cython.Tempita is not
available (however I'm not sure that fallback is ever exercised).
Since scipy needs to support numpy down to 1.8.2, a vendorized copy
will not be usable for scipy for quite a while.
So, it'd be great to handle it like numpydoc: to have npy_tempita as a
small self-contained package with the repo under the numpy
organization and include it via a git submodule. Chuck, do you think
tempita would need much in terms of maintenance?
To put some money where my mouth is, I can offer to do some legwork
for packaging it up.

It might be better to keep tempita and cythonize together so that the

search path works out right. It is also possible that other scripts might
be wanted as cythonize is currently restricted to cython files (*.pyx.in, *.
pxi.in). There are two other templating scripts in numpy/distutils, and I
think f2py has a dependency on one of those.

Post by Charles R Harris

If there is a set of tools that would be common to both scipy and

numpy, having them included as a submodule would be a good idea.

Post by Charles R Harris

Post by Charles R Harris
Hmm, I suppose it just depends on where submodule is, so a npy_tempita

alone would work fine. There isn't much maintenance needed if you resist
the urge to refactor the code. I removed a six dependency, but that is now
upstream as well.

Post by Charles R Harris
There don't seem to be any objections, so I will put the current

vendorization in. Evgeni, if you think it a good idea to make a repo for
this and use submodules, go ahead with that. I have left out the testing
infrastructure at https://github.com/gjhiggins/tempita which runs a sparse
set of doctests.

As long as it's being vendored into numpy/tools, I don't think there's much
point in having one more copy. If any of
cython.tempita, gjhiggins/tempita, and numpy/tools/npy_tempita disappears,
we can reconsider adding a submodule.

Thanks for working on this!

Cheers,

Evgeni

Charles R Harris

2016-09-30 15:25:15 UTC

Permalink

The only thing in the tools directory included in a source distribution is
the swig directory. Tempita is only currently used by the cythonize script
also in the tools directory. The search path for the cythonize script is 1)
installed modules, 2) modules in same directory, which is why it might be
good to rename the module npy_tempita` so that is always the one used.

<snip>

Chuck