Discussion:
[Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead
Nathaniel Smith
2015-10-27 04:31:12 UTC
Permalink
Hi all,

Apparently it is not well known that if you have a Python project
source tree (e.g., a numpy checkout), then the correct way to install
it is NOT to type

python setup.py install # bad and broken!

but rather to type

pip install .

(I.e., pip install isn't just for packages on pypi -- you can also
pass it the path to an arbitrary source directory or the URL of a
source tarball and it will do its thing. In this case "install ."
means "install the project in the current directory".)

These don't quite have identical results -- the main difference is
that the latter makes sure that proper metadata gets installed so that
later on it will be possible to upgrade or uninstall correctly. If you
call setup.py directly, and then later you try to upgrade your
package, then it's entirely possible to end up with a mixture of old
and new versions of the package installed in your PYTHONPATH. (One
common effect is in numpy's case is that we get people sending us
mysterious bug reports about failing tests in files don't even exist
(!) -- because nose is finding tests in files from one version of
numpy and running them against a different version of numpy.)

But this isn't the only issue -- using pip also avoids a bunch of
weird corner cases in distutils/setuptools. E.g., if setup.py uses
plain distutils, then it turns out this will mangle numpy version
numbers in ways that cause weird horribleness -- see [1] for a bug
report of the form "matplotlib doesn't build anymore" which turned out
to be because of using 'setup.py install' to install numpy. OTOH if
setup.py uses setuptools then you get different weirdnesses, like you
can easily end up with multiple versions of the same library installed
simultaneously.

And finally, an advantage of getting used to using 'pip install .' now
is that you'll be prepared for the glorious future when we kill
distutils and get rid of setup.py entirely in favor of something less
terrible [2].

So a proposal that came out of the discussion in [1] is that we modify
numpy's setup.py now so that if you try running

python setup.py install

you get

Error: Calling 'setup.py install' directly is NOT SUPPORTED!
Instead, do:

pip install .

Alternatively, if you want to proceed at your own risk, you
can try 'setup.py install --force-raw-setup.py'
For more information see http://...

(Other setup.py commands would continue to work as normal.)

I believe that this would also break both 'easy_install numpy', and
attempts to install numpy via the setup_requires= argument to
setuptools.setup (because setup_requires= implicitly calls
easy_install). install_requires= would *not* be affected, and
setup_requires= would still be fine in cases where numpy was already
installed.

This would hopefully cut down on the amount of time everyone spends
trying to track down these stupid weird bugs, but it will also require
some adjustment in people's workflows, so... objections? concerns?

-n

[1] https://github.com/numpy/numpy/issues/6551
[2] https://mail.python.org/pipermail/distutils-sig/2015-October/027360.html
--
Nathaniel J. Smith -- http://vorpus.org
Nathaniel Smith
2015-10-27 05:44:15 UTC
Permalink
On Mon, Oct 26, 2015 at 9:31 PM, Nathaniel Smith <***@pobox.com> wrote:
[...]
Post by Nathaniel Smith
I believe that this would also break both 'easy_install numpy', and
attempts to install numpy via the setup_requires= argument to
setuptools.setup (because setup_requires= implicitly calls
easy_install). install_requires= would *not* be affected, and
setup_requires= would still be fine in cases where numpy was already
installed.
On further investigation, it looks like the simplest approach to doing
this would actually treat easy_install and setup_requires= the same
way as they treat pip, i.e., they would all be allowed. (I was
misreading some particularly confusing code in setuptools.)

It also looks like easy_installed packages can at least be safely
upgraded, so I guess allowing this is okay :-).

-n
--
Nathaniel J. Smith -- http://vorpus.org
Ralf Gommers
2015-10-27 07:19:04 UTC
Permalink
Post by Nathaniel Smith
[...]
Post by Nathaniel Smith
I believe that this would also break both 'easy_install numpy', and
attempts to install numpy via the setup_requires= argument to
setuptools.setup (because setup_requires= implicitly calls
easy_install). install_requires= would *not* be affected, and
setup_requires= would still be fine in cases where numpy was already
installed.
On further investigation, it looks like the simplest approach to doing
this would actually treat easy_install and setup_requires= the same
way as they treat pip, i.e., they would all be allowed. (I was
misreading some particularly confusing code in setuptools.)
It also looks like easy_installed packages can at least be safely
upgraded, so I guess allowing this is okay :-).
I just discovered https://bitbucket.org/dholth/setup-requires, which
ensures that setup_requires uses pip instead of easy_install. So we can not
only keep setup-requires working, but make it work significantly better.

So if/when we accept the proposal in this thread, I'm thinking we should
make a bunch of changes at once:
- always use setuptools (this is a new dependency)
- error on ``python setup.py install``
- add the setup-requires trick
- error on ``python setup.py clean`` (saying "use `git clean -xdf` (or -Xdf
...) instead")
- change ``python setup.py --help`` to first show numpy-specific stuff
before setuptools help info
- update all our install docs

And when "pip upgrade" is released (should be soon, see
https://github.com/pypa/pip/pull/3194), officially change our mind and
recommend the use of install_requires/setup_requires to packages depending
on numpy.

Ralf
Nathaniel Smith
2015-10-27 07:28:05 UTC
Permalink
Post by Nathaniel Smith
[...]
Post by Nathaniel Smith
I believe that this would also break both 'easy_install numpy', and
attempts to install numpy via the setup_requires= argument to
setuptools.setup (because setup_requires= implicitly calls
easy_install). install_requires= would *not* be affected, and
setup_requires= would still be fine in cases where numpy was already
installed.
On further investigation, it looks like the simplest approach to doing
this would actually treat easy_install and setup_requires= the same
way as they treat pip, i.e., they would all be allowed. (I was
misreading some particularly confusing code in setuptools.)
It also looks like easy_installed packages can at least be safely
upgraded, so I guess allowing this is okay :-).
I just discovered https://bitbucket.org/dholth/setup-requires, which ensures
that setup_requires uses pip instead of easy_install. So we can not only
keep setup-requires working, but make it work significantly better.
IIUC this is not something that we (= numpy) could use ourselves, but
instead something that everyone who does setup_requires=["numpy"]
would have to set up in their individual projects?

-n
--
Nathaniel J. Smith -- http://vorpus.org
Ralf Gommers
2015-10-27 07:32:00 UTC
Permalink
Post by Ralf Gommers
Post by Ralf Gommers
Post by Nathaniel Smith
[...]
Post by Nathaniel Smith
I believe that this would also break both 'easy_install numpy', and
attempts to install numpy via the setup_requires= argument to
setuptools.setup (because setup_requires= implicitly calls
easy_install). install_requires= would *not* be affected, and
setup_requires= would still be fine in cases where numpy was already
installed.
On further investigation, it looks like the simplest approach to doing
this would actually treat easy_install and setup_requires= the same
way as they treat pip, i.e., they would all be allowed. (I was
misreading some particularly confusing code in setuptools.)
It also looks like easy_installed packages can at least be safely
upgraded, so I guess allowing this is okay :-).
I just discovered https://bitbucket.org/dholth/setup-requires, which
ensures
Post by Ralf Gommers
that setup_requires uses pip instead of easy_install. So we can not only
keep setup-requires working, but make it work significantly better.
IIUC this is not something that we (= numpy) could use ourselves, but
instead something that everyone who does setup_requires=["numpy"]
would have to set up in their individual projects?
Right. I was thinking about using it in scipy. Ah well, I'm sure we can
manage to not break ``setup_requires=numpy`` in some way.

Ralf
j***@gmail.com
2015-10-27 13:07:56 UTC
Permalink
Post by Ralf Gommers
Post by Ralf Gommers
Post by Ralf Gommers
Post by Nathaniel Smith
[...]
Post by Nathaniel Smith
I believe that this would also break both 'easy_install numpy', and
attempts to install numpy via the setup_requires= argument to
setuptools.setup (because setup_requires= implicitly calls
easy_install). install_requires= would *not* be affected, and
setup_requires= would still be fine in cases where numpy was already
installed.
On further investigation, it looks like the simplest approach to doing
this would actually treat easy_install and setup_requires= the same
way as they treat pip, i.e., they would all be allowed. (I was
misreading some particularly confusing code in setuptools.)
It also looks like easy_installed packages can at least be safely
upgraded, so I guess allowing this is okay :-).
I just discovered https://bitbucket.org/dholth/setup-requires, which
ensures
Post by Ralf Gommers
that setup_requires uses pip instead of easy_install. So we can not only
keep setup-requires working, but make it work significantly better.
IIUC this is not something that we (= numpy) could use ourselves, but
instead something that everyone who does setup_requires=["numpy"]
would have to set up in their individual projects?
Right. I was thinking about using it in scipy. Ah well, I'm sure we can
manage to not break ``setup_requires=numpy`` in some way.
What's the equivalent of
python setup.py build_ext --inplace


brief google search (I didn't follow up on those)

https://github.com/pypa/pip/issues/1887
https://github.com/pypa/pip/issues/18


Given that I rely completely on binary distributions for numpy and scipy, I
won't be affected.

(I'm still allergic to pip and will switch only several years after
everybody else.)

Josef
Post by Ralf Gommers
Ralf
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Nathaniel Smith
2015-10-27 14:59:48 UTC
Permalink
On Oct 27, 2015 6:08 AM, <***@gmail.com> wrote:
[...]
Post by j***@gmail.com
What's the equivalent of
python setup.py build_ext --inplace
It's
python setup.py build_ext --inplace

;-)

There's also no replacement for setup.py sdist, or setup.py upload (which
is broken and should never be used), or setup.py clean (which is also
broken and should never be used in numpy's case). pip is a better package
installer than raw distutils or setuptools, for non-installation-related
tasks it has nothing to offer. (With the partial exception of 'pip wheel'.)

-n
j***@gmail.com
2015-10-27 15:28:52 UTC
Permalink
Post by j***@gmail.com
[...]
Post by j***@gmail.com
What's the equivalent of
python setup.py build_ext --inplace
It's
python setup.py build_ext --inplace
;-)
Ok, Sorry, I read now the small print and the issue.

Sounds reasonable, given we can `force` our way out.

(If the reason to run to pip is a misspelled dev version number, then it
looks like a hammer to me.)

Josef
Post by j***@gmail.com
There's also no replacement for setup.py sdist, or setup.py upload (which
is broken and should never be used), or setup.py clean (which is also
broken and should never be used in numpy's case). pip is a better package
installer than raw distutils or setuptools, for non-installation-related
tasks it has nothing to offer. (With the partial exception of 'pip wheel'.)
-n
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Nathan Goldbaum
2015-10-27 15:33:37 UTC
Permalink
Would this happen at the level of numpy's setup.py script or would it be
implemented in numpy.distutils? I'm asking as the developer of a package
that uses numpy.distutils to manage C extensions.
Post by j***@gmail.com
Post by j***@gmail.com
[...]
Post by j***@gmail.com
What's the equivalent of
python setup.py build_ext --inplace
It's
python setup.py build_ext --inplace
;-)
Ok, Sorry, I read now the small print and the issue.
Sounds reasonable, given we can `force` our way out.
(If the reason to run to pip is a misspelled dev version number, then it
looks like a hammer to me.)
Josef
Post by j***@gmail.com
There's also no replacement for setup.py sdist, or setup.py upload (which
is broken and should never be used), or setup.py clean (which is also
broken and should never be used in numpy's case). pip is a better package
installer than raw distutils or setuptools, for non-installation-related
tasks it has nothing to offer. (With the partial exception of 'pip wheel'.)
-n
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Nathaniel Smith
2015-10-27 18:40:42 UTC
Permalink
Post by Nathan Goldbaum
Would this happen at the level of numpy's setup.py script or would it be
implemented in numpy.distutils? I'm asking as the developer of a package
that uses numpy.distutils to manage C extensions.

NumPy's setup.py, no effect on numpy.distutils users. Unless you also get
fed up and implement the same thing, of course ;-).

-n
Ralf Gommers
2015-10-27 21:54:10 UTC
Permalink
Post by j***@gmail.com
Post by j***@gmail.com
[...]
Post by j***@gmail.com
What's the equivalent of
python setup.py build_ext --inplace
It's
python setup.py build_ext --inplace
;-)
Ok, Sorry, I read now the small print and the issue.
Sounds reasonable, given we can `force` our way out.
(If the reason to run to pip is a misspelled dev version number, then it
looks like a hammer to me.)
That's not the reason. A main reason is that we want reliable uninstall of
numpy, as explained in the initial post. We also want to avoid as many
other broken parts of setuptools/easy_install as possible.

Ralf
Ralf Gommers
2015-10-27 22:16:56 UTC
Permalink
On Tue, Oct 27, 2015 at 8:19 AM, Ralf Gommers <***@gmail.com>
wrote:

Updating this list for comments made after I sent it and now that I've
Post by Ralf Gommers
So if/when we accept the proposal in this thread, I'm thinking we should
- always use setuptools (this is a new dependency)
- error on ``python setup.py install``
(removed the item about setup_requires, relevant for scipy but not numpy)
Post by Ralf Gommers
- error on ``python setup.py clean`` (saying "use `git clean -xdf` (or
-Xdf ...) instead")
- change ``python setup.py --help`` to first show numpy-specific stuff
before setuptools help info
- update all our install docs
- error on ``python setup.py upload`` (saying "use `twine upload -s`
instead")
- error on ``python setup.py upload_docs``
- error on ``python setup.py easy_install`` (I'm not joking, that exists)
- error on ``python setup.py test`` (saying "use `python runtests.py`
instead")
- remove setupegg.py

Ralf

And when "pip upgrade" is released (should be soon, see
Post by Ralf Gommers
https://github.com/pypa/pip/pull/3194), officially change our mind and
recommend the use of install_requires/setup_requires to packages depending
on numpy.
Juan Nunez-Iglesias
2015-10-27 22:35:50 UTC
Permalink
Can someone here who understands more about distribution maybe write a blog post detailing:




- why these setup.py commands are bad

- which alternative corresponds to each command and why it's better

- where to find information about this




For example, I had never heard of "twine", and parenthetical statements such as "setup.py upload (which is broken and should never be used)" are useless to those who don't know this and useless to those who do.





I understand that this is an "internal" discussion, but it's nice if those following to learn can get quick pointers. Since there is a *ton* of material online telling us *to use* python setup.py install, all the time, it would be extremely helpful for the community if discussions such as this one helped to bubble up the Right Way of doing Python packaging and distribution.




Thanks,




Juan.
Post by Ralf Gommers
Updating this list for comments made after I sent it and now that I've
Post by Ralf Gommers
So if/when we accept the proposal in this thread, I'm thinking we should
- always use setuptools (this is a new dependency)
- error on ``python setup.py install``
(removed the item about setup_requires, relevant for scipy but not numpy)
Post by Ralf Gommers
- error on ``python setup.py clean`` (saying "use `git clean -xdf` (or
-Xdf ...) instead")
- change ``python setup.py --help`` to first show numpy-specific stuff
before setuptools help info
- update all our install docs
- error on ``python setup.py upload`` (saying "use `twine upload -s`
instead")
- error on ``python setup.py upload_docs``
- error on ``python setup.py easy_install`` (I'm not joking, that exists)
- error on ``python setup.py test`` (saying "use `python runtests.py`
instead")
- remove setupegg.py
Ralf
And when "pip upgrade" is released (should be soon, see
Post by Ralf Gommers
https://github.com/pypa/pip/pull/3194), officially change our mind and
recommend the use of install_requires/setup_requires to packages depending
on numpy.
Ralf Gommers
2015-10-27 23:02:26 UTC
Permalink
Post by Juan Nunez-Iglesias
- why these setup.py commands are bad
- which alternative corresponds to each command and why it's better
- where to find information about this
Good question. Not that I have a blog, but I can try to write something a
bit longer the coming weekend.
Post by Juan Nunez-Iglesias
For example, I had never heard of "twine", and parenthetical statements
such as "setup.py upload (which is broken and should never be used)" are
useless to those who don't know this and useless to those who do.
IIRC `setup.py upload` sends passwords over plain http. I've also seen it
do weird things like change one's own PyPi rights from maintainer to owner.

The most comprehensive overview of all this stuff is
https://packaging.python.org/en/latest/, which starts with tool
recommendations. Twine is one of the first things mentioned.

Ralf
Post by Juan Nunez-Iglesias
I understand that this is an "internal" discussion, but it's nice if those
following to learn can get quick pointers. Since there is a *ton* of
material online telling us *to use* python setup.py install, all the time,
it would be extremely helpful for the community if discussions such as this
one helped to bubble up the Right Way of doing Python packaging and
distribution.
Thanks,
Juan.
Post by Ralf Gommers
Updating this list for comments made after I sent it and now that I've
Post by Ralf Gommers
So if/when we accept the proposal in this thread, I'm thinking we should
- always use setuptools (this is a new dependency)
- error on ``python setup.py install``
(removed the item about setup_requires, relevant for scipy but not numpy)
Post by Ralf Gommers
- error on ``python setup.py clean`` (saying "use `git clean -xdf` (or
-Xdf ...) instead")
- change ``python setup.py --help`` to first show numpy-specific stuff
before setuptools help info
- update all our install docs
- error on ``python setup.py upload`` (saying "use `twine upload -s`
instead")
- error on ``python setup.py upload_docs``
- error on ``python setup.py easy_install`` (I'm not joking, that exists)
- error on ``python setup.py test`` (saying "use `python runtests.py`
instead")
- remove setupegg.py
Ralf
And when "pip upgrade" is released (should be soon, see
Post by Ralf Gommers
https://github.com/pypa/pip/pull/3194), officially change our mind and
recommend the use of install_requires/setup_requires to packages depending
on numpy.
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Juan Nunez-Iglesias
2015-10-28 00:04:40 UTC
Permalink
Thanks Ralf! The pointer to Python Packaging User Guide is already gold! (But a wider discussion e.g. in the NumPy repo, mirroring the docstring conventions, would also be good!)
Post by Ralf Gommers
Post by Juan Nunez-Iglesias
Can someone here who understands more about distribution maybe write a
- why these setup.py commands are bad
- which alternative corresponds to each command and why it's better
- where to find information about this
Good question. Not that I have a blog, but I can try to write something a
bit longer the coming weekend.
Post by Juan Nunez-Iglesias
For example, I had never heard of "twine", and parenthetical statements
such as "setup.py upload (which is broken and should never be used)" are
useless to those who don't know this and useless to those who do.
IIRC `setup.py upload` sends passwords over plain http. I've also seen it
do weird things like change one's own PyPi rights from maintainer to owner.
The most comprehensive overview of all this stuff is
https://packaging.python.org/en/latest/, which starts with tool
recommendations. Twine is one of the first things mentioned.
Ralf
Post by Juan Nunez-Iglesias
I understand that this is an "internal" discussion, but it's nice if those
following to learn can get quick pointers. Since there is a *ton* of
material online telling us *to use* python setup.py install, all the time,
it would be extremely helpful for the community if discussions such as this
one helped to bubble up the Right Way of doing Python packaging and
distribution.
Thanks,
Juan.
Post by Ralf Gommers
Updating this list for comments made after I sent it and now that I've
Post by Ralf Gommers
So if/when we accept the proposal in this thread, I'm thinking we should
- always use setuptools (this is a new dependency)
- error on ``python setup.py install``
(removed the item about setup_requires, relevant for scipy but not numpy)
Post by Ralf Gommers
- error on ``python setup.py clean`` (saying "use `git clean -xdf` (or
-Xdf ...) instead")
- change ``python setup.py --help`` to first show numpy-specific stuff
before setuptools help info
- update all our install docs
- error on ``python setup.py upload`` (saying "use `twine upload -s`
instead")
- error on ``python setup.py upload_docs``
- error on ``python setup.py easy_install`` (I'm not joking, that exists)
- error on ``python setup.py test`` (saying "use `python runtests.py`
instead")
- remove setupegg.py
Ralf
And when "pip upgrade" is released (should be soon, see
Post by Ralf Gommers
https://github.com/pypa/pip/pull/3194), officially change our mind and
recommend the use of install_requires/setup_requires to packages depending
on numpy.
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Jerome Kieffer
2015-10-28 07:36:23 UTC
Permalink
On Tue, 27 Oct 2015 15:35:50 -0700 (PDT)
Hi,

Olivier Grisel from sklearn gave a very good talk on this topic at PyCon, earlier
this year:
http://www.pyvideo.org/video/3473/build-and-test-wheel-packages-on-linux-osx-win

Very instructive.
--
Jérôme Kieffer
tel +33 476 882 445
Juan Nunez-Iglesias
2015-10-28 09:19:29 UTC
Permalink
Thanks, Jerome! I’ve added it to my to-watch list. It sounds really useful!




Juan.
Post by Jerome Kieffer
On Tue, 27 Oct 2015 15:35:50 -0700 (PDT)
Hi,
Olivier Grisel from sklearn gave a very good talk on this topic at PyCon, earlier
http://www.pyvideo.org/video/3473/build-and-test-wheel-packages-on-linux-osx-win
Very instructive.
--
JérÎme Kieffer
tel +33 476 882 445
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Charles R Harris
2015-10-27 06:03:22 UTC
Permalink
Post by Nathaniel Smith
Hi all,
Apparently it is not well known that if you have a Python project
source tree (e.g., a numpy checkout), then the correct way to install
it is NOT to type
python setup.py install # bad and broken!
but rather to type
pip install .
(I.e., pip install isn't just for packages on pypi -- you can also
pass it the path to an arbitrary source directory or the URL of a
source tarball and it will do its thing. In this case "install ."
means "install the project in the current directory".)
These don't quite have identical results -- the main difference is
that the latter makes sure that proper metadata gets installed so that
later on it will be possible to upgrade or uninstall correctly. If you
call setup.py directly, and then later you try to upgrade your
package, then it's entirely possible to end up with a mixture of old
and new versions of the package installed in your PYTHONPATH. (One
common effect is in numpy's case is that we get people sending us
mysterious bug reports about failing tests in files don't even exist
(!) -- because nose is finding tests in files from one version of
numpy and running them against a different version of numpy.)
But this isn't the only issue -- using pip also avoids a bunch of
weird corner cases in distutils/setuptools. E.g., if setup.py uses
plain distutils, then it turns out this will mangle numpy version
numbers in ways that cause weird horribleness -- see [1] for a bug
report of the form "matplotlib doesn't build anymore" which turned out
to be because of using 'setup.py install' to install numpy. OTOH if
setup.py uses setuptools then you get different weirdnesses, like you
can easily end up with multiple versions of the same library installed
simultaneously.
And finally, an advantage of getting used to using 'pip install .' now
is that you'll be prepared for the glorious future when we kill
distutils and get rid of setup.py entirely in favor of something less
terrible [2].
So a proposal that came out of the discussion in [1] is that we modify
numpy's setup.py now so that if you try running
python setup.py install
you get
Error: Calling 'setup.py install' directly is NOT SUPPORTED!
pip install .
Alternatively, if you want to proceed at your own risk, you
can try 'setup.py install --force-raw-setup.py'
For more information see http://...
(Other setup.py commands would continue to work as normal.)
I believe that this would also break both 'easy_install numpy', and
attempts to install numpy via the setup_requires= argument to
setuptools.setup (because setup_requires= implicitly calls
easy_install). install_requires= would *not* be affected, and
setup_requires= would still be fine in cases where numpy was already
installed.
This would hopefully cut down on the amount of time everyone spends
trying to track down these stupid weird bugs, but it will also require
some adjustment in people's workflows, so... objections? concerns?
I gave it a shot the other day. Pip keeps a record of the path to the repo
and in order to cleanup I needed to search out the file and delete the repo
path. There is probably a better way to do that, but it didn't strike me as
less troublesome than ` python setup.py install --local`.

Chuck
Nathaniel Smith
2015-10-27 06:08:07 UTC
Permalink
On Mon, Oct 26, 2015 at 11:03 PM, Charles R Harris
<***@gmail.com> wrote:
[...]
Post by Charles R Harris
I gave it a shot the other day. Pip keeps a record of the path to the repo
and in order to cleanup I needed to search out the file and delete the repo
path. There is probably a better way to do that, but it didn't strike me as
less troublesome than ` python setup.py install --local`.
Sorry, what did you "give a shot", and what problem did it create?
What does `setup.py install --local` do? (it doesn't seem to be
mentioned in `setup.py install --help`.)

-n
--
Nathaniel J. Smith -- http://vorpus.org
Charles R Harris
2015-10-27 06:33:01 UTC
Permalink
Post by Nathaniel Smith
On Mon, Oct 26, 2015 at 11:03 PM, Charles R Harris
[...]
Post by Charles R Harris
I gave it a shot the other day. Pip keeps a record of the path to the
repo
Post by Charles R Harris
and in order to cleanup I needed to search out the file and delete the
repo
Post by Charles R Harris
path. There is probably a better way to do that, but it didn't strike me
as
Post by Charles R Harris
less troublesome than ` python setup.py install --local`.
Sorry, what did you "give a shot", and what problem did it create?
What does `setup.py install --local` do? (it doesn't seem to be
mentioned in `setup.py install --help`.)
`pip install --user -e . `. However, `pip install --user .` seems to work
fine. The pip documentation isn't the best.

Yeah, `--user` not `--local`. It's getting late...

Chuck
Juan Nunez-Iglesias
2015-10-27 06:53:09 UTC
Permalink
Is there a pip equivalent of "python setup.py develop"?
Post by Charles R Harris
Post by Nathaniel Smith
On Mon, Oct 26, 2015 at 11:03 PM, Charles R Harris
[...]
Post by Charles R Harris
I gave it a shot the other day. Pip keeps a record of the path to the
repo
Post by Charles R Harris
and in order to cleanup I needed to search out the file and delete the
repo
Post by Charles R Harris
path. There is probably a better way to do that, but it didn't strike
me as
Post by Charles R Harris
less troublesome than ` python setup.py install --local`.
Sorry, what did you "give a shot", and what problem did it create?
What does `setup.py install --local` do? (it doesn't seem to be
mentioned in `setup.py install --help`.)
`pip install --user -e . `. However, `pip install --user .` seems to work
fine. The pip documentation isn't the best.
Yeah, `--user` not `--local`. It's getting late...
Chuck
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Nathaniel Smith
2015-10-27 07:31:09 UTC
Permalink
On Mon, Oct 26, 2015 at 11:53 PM, Juan Nunez-Iglesias
Post by Juan Nunez-Iglesias
Is there a pip equivalent of "python setup.py develop"?
Kinda answered this already when replying to Chuck, but: yes, it's
`pip install -e <path or URL or package name>` (the -e is short for
--editable), not that you would need it necessarily because `setup.py
develop` would still be legal.

-n
--
Nathaniel J. Smith -- http://vorpus.org
Nathaniel Smith
2015-10-27 07:18:21 UTC
Permalink
On Mon, Oct 26, 2015 at 11:33 PM, Charles R Harris
Post by Charles R Harris
Post by Nathaniel Smith
On Mon, Oct 26, 2015 at 11:03 PM, Charles R Harris
[...]
Post by Charles R Harris
I gave it a shot the other day. Pip keeps a record of the path to the repo
and in order to cleanup I needed to search out the file and delete the repo
path. There is probably a better way to do that, but it didn't strike me as
less troublesome than ` python setup.py install --local`.
Sorry, what did you "give a shot", and what problem did it create?
What does `setup.py install --local` do? (it doesn't seem to be
mentioned in `setup.py install --help`.)
`pip install --user -e . `. However, `pip install --user .` seems to work
fine. The pip documentation isn't the best.
Yeah, `--user` not `--local`. It's getting late...
Ah. I think if you want to undo a `pip install --user -e .` you can
just do `pip uninstall numpy`? But in any case the equivalent of 'pip
install -e' is 'setup.py develop', and the proposal is only to disable
`setup.py install`. So if you prefer `setup.py develop` then you could
still use it.

(IIUC `pip install -e` is a *very* thin wrapper around `setup.py
develop` -- normally pip takes over the job of actually installing
files, so pip gets to interpret options like --user and figure out
what it means for where it puts the files, but in the case of `install
-e` it looks like it just calls `setup.py develop` directly, so if
--user doesn't work it's probably because setup.py develop doesn't
know about --user?)

-n
--
Nathaniel J. Smith -- http://vorpus.org
James E.H. Turner
2015-10-27 13:48:21 UTC
Permalink
Post by Nathaniel Smith
Apparently it is not well known that if you have a Python project
source tree (e.g., a numpy checkout), then the correct way to install
it is NOT to type
python setup.py install # bad and broken!
but rather to type
pip install .
Though I haven't studied it exhaustively, it always seems to me that
pip is bad & broken, whereas python setup.py install does what I
expect (even if it's a mess internally). In particular, when
maintaining a distribution of Python packages, you try to have some
well-defined, reproducible build from source tarballs and then you
find that pip is going off and downloading stuff under the radar
without being asked (etc.). Stopping that can be a pain & I always
groan whenever some package insists on using pip. Maybe I don't
understand it well enough but in this role its dependency handling
is an unnecessary complication with no purpose. Just a comment that
not every installation is someone trying to get numpy on their
laptop...

Cheers,

James.
Edison Gustavo Muenz
2015-10-27 14:31:56 UTC
Permalink
I'm sorry if this is out-of-topic, but I'm curious on why nobody mentioned
Conda yet.

Is there any particular reason for not using it?
Post by Nathaniel Smith
Apparently it is not well known that if you have a Python project
Post by Nathaniel Smith
source tree (e.g., a numpy checkout), then the correct way to install
it is NOT to type
python setup.py install # bad and broken!
but rather to type
pip install .
Though I haven't studied it exhaustively, it always seems to me that
pip is bad & broken, whereas python setup.py install does what I
expect (even if it's a mess internally). In particular, when
maintaining a distribution of Python packages, you try to have some
well-defined, reproducible build from source tarballs and then you
find that pip is going off and downloading stuff under the radar
without being asked (etc.). Stopping that can be a pain & I always
groan whenever some package insists on using pip. Maybe I don't
understand it well enough but in this role its dependency handling
is an unnecessary complication with no purpose. Just a comment that
not every installation is someone trying to get numpy on their
laptop...
Cheers,
James.
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
David Cournapeau
2015-10-27 14:46:28 UTC
Permalink
On Tue, Oct 27, 2015 at 2:31 PM, Edison Gustavo Muenz <
Post by Edison Gustavo Muenz
I'm sorry if this is out-of-topic, but I'm curious on why nobody mentioned
Conda yet.
Conda is a binary distribution system, whereas we are talking about
installing from sources. You will need a way to install things when
building a conda package in any case

David
Post by Edison Gustavo Muenz
Is there any particular reason for not using it?
Post by Nathaniel Smith
Apparently it is not well known that if you have a Python project
Post by Nathaniel Smith
source tree (e.g., a numpy checkout), then the correct way to install
it is NOT to type
python setup.py install # bad and broken!
but rather to type
pip install .
Though I haven't studied it exhaustively, it always seems to me that
pip is bad & broken, whereas python setup.py install does what I
expect (even if it's a mess internally). In particular, when
maintaining a distribution of Python packages, you try to have some
well-defined, reproducible build from source tarballs and then you
find that pip is going off and downloading stuff under the radar
without being asked (etc.). Stopping that can be a pain & I always
groan whenever some package insists on using pip. Maybe I don't
understand it well enough but in this role its dependency handling
is an unnecessary complication with no purpose. Just a comment that
not every installation is someone trying to get numpy on their
laptop...
Cheers,
James.
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Benjamin Root
2015-10-27 14:47:01 UTC
Permalink
Conda is for binary installs and largely targeted for end-users. This topic
pertains to source installs, and is mostly relevant to developers, testers,
and those who like to live on the bleeding edge of a particular project.

On Tue, Oct 27, 2015 at 10:31 AM, Edison Gustavo Muenz <
Post by Edison Gustavo Muenz
I'm sorry if this is out-of-topic, but I'm curious on why nobody mentioned
Conda yet.
Is there any particular reason for not using it?
Post by Nathaniel Smith
Apparently it is not well known that if you have a Python project
Post by Nathaniel Smith
source tree (e.g., a numpy checkout), then the correct way to install
it is NOT to type
python setup.py install # bad and broken!
but rather to type
pip install .
Though I haven't studied it exhaustively, it always seems to me that
pip is bad & broken, whereas python setup.py install does what I
expect (even if it's a mess internally). In particular, when
maintaining a distribution of Python packages, you try to have some
well-defined, reproducible build from source tarballs and then you
find that pip is going off and downloading stuff under the radar
without being asked (etc.). Stopping that can be a pain & I always
groan whenever some package insists on using pip. Maybe I don't
understand it well enough but in this role its dependency handling
is an unnecessary complication with no purpose. Just a comment that
not every installation is someone trying to get numpy on their
laptop...
Cheers,
James.
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Nathaniel Smith
2015-10-27 15:18:29 UTC
Permalink
Post by James E.H. Turner
Post by Nathaniel Smith
Apparently it is not well known that if you have a Python project
source tree (e.g., a numpy checkout), then the correct way to install
it is NOT to type
python setup.py install # bad and broken!
but rather to type
pip install .
Though I haven't studied it exhaustively, it always seems to me that
pip is bad & broken, whereas python setup.py install does what I
expect (even if it's a mess internally).
Unfortunately this is only true if what you expect is for packages to be
installed in subtly corrupted ways, as described in the original email.
Sorry to be the bearer of bad tidings :-/
Post by James E.H. Turner
In particular, when
maintaining a distribution of Python packages, you try to have some
well-defined, reproducible build from source tarballs and then you
find that pip is going off and downloading stuff under the radar
without being asked (etc.). Stopping that can be a pain & I always
groan whenever some package insists on using pip. Maybe I don't
understand it well enough but in this role its dependency handling
is an unnecessary complication with no purpose.
There are two cases where a 'pip install' run might go off and start
downloading packages without asking you:

- if the project is using setuptools with setup_requires=..., then the
setup.py itself will go off and start downloading things without asking.
This has nothing to do with pip. The way Debian prevents this is that they
always define an intentionally invalid http_proxy environment variable
before building any python package.

- if the project has declared that they do not work without some other
package installed via install_requires=... For this case, if you really
know what you're doing and you intentionally want to install a
non-functional configuration (which yeah, a package build tool might indeed
want to do), then just add --no-deps to the pip install command line. Maybe
add --no-index and/or the magic http_proxy setting if you want to be extra
sure.
Post by James E.H. Turner
Just a comment that
not every installation is someone trying to get numpy on their
laptop...
Sure, we're well aware of the importance of downstream packagers -- part of
the point of having this email thread is to smoke out such non-trivial use
cases. (And note that worst case if you decide that you'd rather take your
chances with setup.py install, then that's why the proposal includes an
escape hatch of passing a special --force switch.)

But unless you're somehow planning to disable pip entirely in your
distribution, so that end users have to get upgrades through your tools
rather than using pip, then you do probably want to think about how to
provide accurate pip-style metadata. (And even then it doesn't hurt.)

-n
Nathan Goldbaum
2015-10-27 15:25:30 UTC
Permalink
Interestingly, conda actually does "setup.py install" in the recipe for
numpy:
https://github.com/conda/conda-recipes/blob/master/numpy-openblas/build.sh

I'm not sure if this is the one they use to build the anaconda package, I
think they have internal versions of most of the recipes on conda-recipes.
Post by Nathaniel Smith
Post by James E.H. Turner
Post by Nathaniel Smith
Apparently it is not well known that if you have a Python project
source tree (e.g., a numpy checkout), then the correct way to install
it is NOT to type
python setup.py install # bad and broken!
but rather to type
pip install .
Though I haven't studied it exhaustively, it always seems to me that
pip is bad & broken, whereas python setup.py install does what I
expect (even if it's a mess internally).
Unfortunately this is only true if what you expect is for packages to be
installed in subtly corrupted ways, as described in the original email.
Sorry to be the bearer of bad tidings :-/
Post by James E.H. Turner
In particular, when
maintaining a distribution of Python packages, you try to have some
well-defined, reproducible build from source tarballs and then you
find that pip is going off and downloading stuff under the radar
without being asked (etc.). Stopping that can be a pain & I always
groan whenever some package insists on using pip. Maybe I don't
understand it well enough but in this role its dependency handling
is an unnecessary complication with no purpose.
There are two cases where a 'pip install' run might go off and start
- if the project is using setuptools with setup_requires=..., then the
setup.py itself will go off and start downloading things without asking.
This has nothing to do with pip. The way Debian prevents this is that they
always define an intentionally invalid http_proxy environment variable
before building any python package.
- if the project has declared that they do not work without some other
package installed via install_requires=... For this case, if you really
know what you're doing and you intentionally want to install a
non-functional configuration (which yeah, a package build tool might indeed
want to do), then just add --no-deps to the pip install command line. Maybe
add --no-index and/or the magic http_proxy setting if you want to be extra
sure.
Post by James E.H. Turner
Just a comment that
not every installation is someone trying to get numpy on their
laptop...
Sure, we're well aware of the importance of downstream packagers -- part
of the point of having this email thread is to smoke out such non-trivial
use cases. (And note that worst case if you decide that you'd rather take
your chances with setup.py install, then that's why the proposal includes
an escape hatch of passing a special --force switch.)
But unless you're somehow planning to disable pip entirely in your
distribution, so that end users have to get upgrades through your tools
rather than using pip, then you do probably want to think about how to
provide accurate pip-style metadata. (And even then it doesn't hurt.)
-n
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Chris Barker
2015-11-03 00:04:17 UTC
Permalink
Post by Nathan Goldbaum
Interestingly, conda actually does "setup.py install" in the recipe for
indeed -- many, many conda packages do setup.py install, whihc doesn't mean
it's a good idea --personally, I'm trying hard to switch them all to:

pip install ./

:-)

Which reminds me, the conda skelaton command craes a pip install build.sh
file -- I really need to submit a PR for that ...

There are two cases where a 'pip install' run might go off and start
for my part, regular old setup.py isntall oftem goes off and istalls sutff
too - using easy_install, which really sucks...

This is making me want a setuptools-lite again -- see the distutils SIG if
you're curious.

-CHB
--
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

***@noaa.gov
Sandro Tosi
2015-10-28 23:28:32 UTC
Permalink
please, pretty please, do not disable setup.py install or at least
keep providing a way for distribution (Debian in this case) to be able
to build/install numpy in a temporary location for packaging reasons.
pip is not the solution for us
Post by Nathaniel Smith
Hi all,
Apparently it is not well known that if you have a Python project
source tree (e.g., a numpy checkout), then the correct way to install
it is NOT to type
python setup.py install # bad and broken!
but rather to type
pip install .
(I.e., pip install isn't just for packages on pypi -- you can also
pass it the path to an arbitrary source directory or the URL of a
source tarball and it will do its thing. In this case "install ."
means "install the project in the current directory".)
These don't quite have identical results -- the main difference is
that the latter makes sure that proper metadata gets installed so that
later on it will be possible to upgrade or uninstall correctly. If you
call setup.py directly, and then later you try to upgrade your
package, then it's entirely possible to end up with a mixture of old
and new versions of the package installed in your PYTHONPATH. (One
common effect is in numpy's case is that we get people sending us
mysterious bug reports about failing tests in files don't even exist
(!) -- because nose is finding tests in files from one version of
numpy and running them against a different version of numpy.)
But this isn't the only issue -- using pip also avoids a bunch of
weird corner cases in distutils/setuptools. E.g., if setup.py uses
plain distutils, then it turns out this will mangle numpy version
numbers in ways that cause weird horribleness -- see [1] for a bug
report of the form "matplotlib doesn't build anymore" which turned out
to be because of using 'setup.py install' to install numpy. OTOH if
setup.py uses setuptools then you get different weirdnesses, like you
can easily end up with multiple versions of the same library installed
simultaneously.
And finally, an advantage of getting used to using 'pip install .' now
is that you'll be prepared for the glorious future when we kill
distutils and get rid of setup.py entirely in favor of something less
terrible [2].
So a proposal that came out of the discussion in [1] is that we modify
numpy's setup.py now so that if you try running
python setup.py install
you get
Error: Calling 'setup.py install' directly is NOT SUPPORTED!
pip install .
Alternatively, if you want to proceed at your own risk, you
can try 'setup.py install --force-raw-setup.py'
For more information see http://...
(Other setup.py commands would continue to work as normal.)
I believe that this would also break both 'easy_install numpy', and
attempts to install numpy via the setup_requires= argument to
setuptools.setup (because setup_requires= implicitly calls
easy_install). install_requires= would *not* be affected, and
setup_requires= would still be fine in cases where numpy was already
installed.
This would hopefully cut down on the amount of time everyone spends
trying to track down these stupid weird bugs, but it will also require
some adjustment in people's workflows, so... objections? concerns?
-n
[1] https://github.com/numpy/numpy/issues/6551
[2] https://mail.python.org/pipermail/distutils-sig/2015-October/027360.html
--
Nathaniel J. Smith -- http://vorpus.org
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
--
Sandro Tosi (aka morph, morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi
Ralf Gommers
2015-10-28 23:34:50 UTC
Permalink
Post by Sandro Tosi
please, pretty please, do not disable setup.py install or at least
keep providing a way for distribution (Debian in this case) to be able
to build/install numpy in a temporary location for packaging reasons.
pip is not the solution for us
``python setup.py install --force`` would still work. Would that be OK then?

Ralf
Post by Sandro Tosi
Post by Nathaniel Smith
Hi all,
Apparently it is not well known that if you have a Python project
source tree (e.g., a numpy checkout), then the correct way to install
it is NOT to type
python setup.py install # bad and broken!
but rather to type
pip install .
(I.e., pip install isn't just for packages on pypi -- you can also
pass it the path to an arbitrary source directory or the URL of a
source tarball and it will do its thing. In this case "install ."
means "install the project in the current directory".)
These don't quite have identical results -- the main difference is
that the latter makes sure that proper metadata gets installed so that
later on it will be possible to upgrade or uninstall correctly. If you
call setup.py directly, and then later you try to upgrade your
package, then it's entirely possible to end up with a mixture of old
and new versions of the package installed in your PYTHONPATH. (One
common effect is in numpy's case is that we get people sending us
mysterious bug reports about failing tests in files don't even exist
(!) -- because nose is finding tests in files from one version of
numpy and running them against a different version of numpy.)
But this isn't the only issue -- using pip also avoids a bunch of
weird corner cases in distutils/setuptools. E.g., if setup.py uses
plain distutils, then it turns out this will mangle numpy version
numbers in ways that cause weird horribleness -- see [1] for a bug
report of the form "matplotlib doesn't build anymore" which turned out
to be because of using 'setup.py install' to install numpy. OTOH if
setup.py uses setuptools then you get different weirdnesses, like you
can easily end up with multiple versions of the same library installed
simultaneously.
And finally, an advantage of getting used to using 'pip install .' now
is that you'll be prepared for the glorious future when we kill
distutils and get rid of setup.py entirely in favor of something less
terrible [2].
So a proposal that came out of the discussion in [1] is that we modify
numpy's setup.py now so that if you try running
python setup.py install
you get
Error: Calling 'setup.py install' directly is NOT SUPPORTED!
pip install .
Alternatively, if you want to proceed at your own risk, you
can try 'setup.py install --force-raw-setup.py'
For more information see http://...
(Other setup.py commands would continue to work as normal.)
I believe that this would also break both 'easy_install numpy', and
attempts to install numpy via the setup_requires= argument to
setuptools.setup (because setup_requires= implicitly calls
easy_install). install_requires= would *not* be affected, and
setup_requires= would still be fine in cases where numpy was already
installed.
This would hopefully cut down on the amount of time everyone spends
trying to track down these stupid weird bugs, but it will also require
some adjustment in people's workflows, so... objections? concerns?
-n
[1] https://github.com/numpy/numpy/issues/6551
[2]
https://mail.python.org/pipermail/distutils-sig/2015-October/027360.html
Post by Nathaniel Smith
--
Nathaniel J. Smith -- http://vorpus.org
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
--
Sandro Tosi (aka morph, morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Todd
2015-10-29 15:25:17 UTC
Permalink
Post by Sandro Tosi
please, pretty please, do not disable setup.py install or at least
keep providing a way for distribution (Debian in this case) to be able
to build/install numpy in a temporary location for packaging reasons.
pip is not the solution for us
What is wrong with "pip install --root" ?
Warren Weckesser
2015-10-29 19:11:44 UTC
Permalink
Post by Nathaniel Smith
Hi all,
Apparently it is not well known that if you have a Python project
source tree (e.g., a numpy checkout), then the correct way to install
it is NOT to type
python setup.py install # bad and broken!
but rather to type
pip install .
FWIW, I don't see any mention of this in the numpy docs, but I do see a lot
of instructions involving `setup.py build` and `setup.py install`. See,
for example, INSTALL.txt. Also see
http://docs.scipy.org/doc/numpy/user/install.html#building-from-source
So I guess it is not surprising that it is not well known.

Warren
Post by Nathaniel Smith
(I.e., pip install isn't just for packages on pypi -- you can also
pass it the path to an arbitrary source directory or the URL of a
source tarball and it will do its thing. In this case "install ."
means "install the project in the current directory".)
These don't quite have identical results -- the main difference is
that the latter makes sure that proper metadata gets installed so that
later on it will be possible to upgrade or uninstall correctly. If you
call setup.py directly, and then later you try to upgrade your
package, then it's entirely possible to end up with a mixture of old
and new versions of the package installed in your PYTHONPATH. (One
common effect is in numpy's case is that we get people sending us
mysterious bug reports about failing tests in files don't even exist
(!) -- because nose is finding tests in files from one version of
numpy and running them against a different version of numpy.)
But this isn't the only issue -- using pip also avoids a bunch of
weird corner cases in distutils/setuptools. E.g., if setup.py uses
plain distutils, then it turns out this will mangle numpy version
numbers in ways that cause weird horribleness -- see [1] for a bug
report of the form "matplotlib doesn't build anymore" which turned out
to be because of using 'setup.py install' to install numpy. OTOH if
setup.py uses setuptools then you get different weirdnesses, like you
can easily end up with multiple versions of the same library installed
simultaneously.
And finally, an advantage of getting used to using 'pip install .' now
is that you'll be prepared for the glorious future when we kill
distutils and get rid of setup.py entirely in favor of something less
terrible [2].
So a proposal that came out of the discussion in [1] is that we modify
numpy's setup.py now so that if you try running
python setup.py install
you get
Error: Calling 'setup.py install' directly is NOT SUPPORTED!
pip install .
Alternatively, if you want to proceed at your own risk, you
can try 'setup.py install --force-raw-setup.py'
For more information see http://...
(Other setup.py commands would continue to work as normal.)
I believe that this would also break both 'easy_install numpy', and
attempts to install numpy via the setup_requires= argument to
setuptools.setup (because setup_requires= implicitly calls
easy_install). install_requires= would *not* be affected, and
setup_requires= would still be fine in cases where numpy was already
installed.
This would hopefully cut down on the amount of time everyone spends
trying to track down these stupid weird bugs, but it will also require
some adjustment in people's workflows, so... objections? concerns?
-n
[1] https://github.com/numpy/numpy/issues/6551
[2]
https://mail.python.org/pipermail/distutils-sig/2015-October/027360.html
--
Nathaniel J. Smith -- http://vorpus.org
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion
Ralf Gommers
2015-11-01 00:54:45 UTC
Permalink
On Thu, Oct 29, 2015 at 8:11 PM, Warren Weckesser <
Post by Warren Weckesser
Post by Nathaniel Smith
Hi all,
Apparently it is not well known that if you have a Python project
source tree (e.g., a numpy checkout), then the correct way to install
it is NOT to type
python setup.py install # bad and broken!
but rather to type
pip install .
FWIW, I don't see any mention of this in the numpy docs, but I do see a
lot of instructions involving `setup.py build` and `setup.py install`.
See, for example, INSTALL.txt. Also see
http://docs.scipy.org/doc/numpy/user/install.html#building-from-source
So I guess it is not surprising that it is not well known.
Indeed, install docs are always hopelessly outdated. And we have too many
of them. There's duplicate info in INSTALL.txt and
http://scipy.org/scipylib/building/index.html for example. We should
probably just empty out INSTALL.txt and simply put a link in it to the html
docs.

I've created an issue with a long todo list and a bunch of links:
https://github.com/numpy/numpy/issues/6599. Feel free to add stuff. Or to
go fix something:)

Ralf
Ralf Gommers
2015-11-01 00:59:09 UTC
Permalink
Post by Ralf Gommers
On Thu, Oct 29, 2015 at 8:11 PM, Warren Weckesser <
Post by Warren Weckesser
Post by Nathaniel Smith
Hi all,
Apparently it is not well known that if you have a Python project
source tree (e.g., a numpy checkout), then the correct way to install
it is NOT to type
python setup.py install # bad and broken!
but rather to type
pip install .
FWIW, I don't see any mention of this in the numpy docs, but I do see a
lot of instructions involving `setup.py build` and `setup.py install`.
See, for example, INSTALL.txt. Also see
http://docs.scipy.org/doc/numpy/user/install.html#building-from-source
So I guess it is not surprising that it is not well known.
Indeed, install docs are always hopelessly outdated. And we have too many
of them. There's duplicate info in INSTALL.txt and
http://scipy.org/scipylib/building/index.html for example. We should
probably just empty out INSTALL.txt and simply put a link in it to the html
docs.
https://github.com/numpy/numpy/issues/6599. Feel free to add stuff. Or to
go fix something:)
Oh, and: looking at this thread there haven't been serious unanswered
concerns (at least in my perception), so without more discussion I'd
interpret the current status as "go ahead".

Ralf
Ralf Gommers
2015-11-01 23:16:27 UTC
Permalink
Post by Ralf Gommers
Post by Ralf Gommers
On Thu, Oct 29, 2015 at 8:11 PM, Warren Weckesser <
Post by Warren Weckesser
Post by Nathaniel Smith
Hi all,
Apparently it is not well known that if you have a Python project
source tree (e.g., a numpy checkout), then the correct way to install
it is NOT to type
python setup.py install # bad and broken!
but rather to type
pip install .
FWIW, I don't see any mention of this in the numpy docs, but I do see a
lot of instructions involving `setup.py build` and `setup.py install`.
See, for example, INSTALL.txt. Also see
http://docs.scipy.org/doc/numpy/user/install.html#building-from-source
So I guess it is not surprising that it is not well known.
Indeed, install docs are always hopelessly outdated. And we have too many
of them. There's duplicate info in INSTALL.txt and
http://scipy.org/scipylib/building/index.html for example. We should
probably just empty out INSTALL.txt and simply put a link in it to the html
docs.
https://github.com/numpy/numpy/issues/6599. Feel free to add stuff. Or
to go fix something:)
Oh, and: looking at this thread there haven't been serious unanswered
concerns (at least in my perception), so without more discussion I'd
interpret the current status as "go ahead".
Hmm, after some more testing I'm going to have to bring up a few concerns
myself:

1. ``pip install .`` still has a clear bug; it starts by copying everything
(including .git/ !) to a tempdir with shutil, which is very slow. And the
fix for that will go via ``setup.py sdist``, which is still slow.

2. ``pip install .`` silences build output, which may make sense for some
usecases, but for numpy it just sits there for minutes with no output after
printing "Running setup.py install for numpy". Users will think it hangs
and Ctrl-C it. https://github.com/pypa/pip/issues/2732

3. ``pip install .`` refuses to upgrade an already installed development
version. For released versions that makes sense, but if I'm in a git tree
then I don't want it to refuse because 1.11.0.dev0+githash1 compares equal
to 1.11.0.dev0+githash2. Especially after waiting a few minutes, see (1).


I've sent a (incomplete) fix for the shutil thing (
https://github.com/pypa/pip/pull/3219) and will comment on some open issues
on the pip tracker. But I'm thinking that for now we should go with some
printed message first. Something like "please use ``pip install .`` if you
want reliable uninstall behavior. See <somelink> for more details".

Pip has worked quite well for me in the past, but the above makes me thing
it's not much of an improvement over use of setuptools.....

Ralf
Charles R Harris
2015-11-02 00:12:33 UTC
Permalink
Post by Ralf Gommers
Post by Ralf Gommers
Post by Ralf Gommers
On Thu, Oct 29, 2015 at 8:11 PM, Warren Weckesser <
Post by Warren Weckesser
Post by Nathaniel Smith
Hi all,
Apparently it is not well known that if you have a Python project
source tree (e.g., a numpy checkout), then the correct way to install
it is NOT to type
python setup.py install # bad and broken!
but rather to type
pip install .
FWIW, I don't see any mention of this in the numpy docs, but I do see a
lot of instructions involving `setup.py build` and `setup.py install`.
See, for example, INSTALL.txt. Also see
http://docs.scipy.org/doc/numpy/user/install.html#building-from-source
So I guess it is not surprising that it is not well known.
Indeed, install docs are always hopelessly outdated. And we have too
many of them. There's duplicate info in INSTALL.txt and
http://scipy.org/scipylib/building/index.html for example. We should
probably just empty out INSTALL.txt and simply put a link in it to the html
docs.
https://github.com/numpy/numpy/issues/6599. Feel free to add stuff. Or
to go fix something:)
Oh, and: looking at this thread there haven't been serious unanswered
concerns (at least in my perception), so without more discussion I'd
interpret the current status as "go ahead".
Hmm, after some more testing I'm going to have to bring up a few concerns
1. ``pip install .`` still has a clear bug; it starts by copying
everything (including .git/ !) to a tempdir with shutil, which is very
slow. And the fix for that will go via ``setup.py sdist``, which is still
slow.
2. ``pip install .`` silences build output, which may make sense for some
usecases, but for numpy it just sits there for minutes with no output after
printing "Running setup.py install for numpy". Users will think it hangs
and Ctrl-C it. https://github.com/pypa/pip/issues/2732
3. ``pip install .`` refuses to upgrade an already installed development
version. For released versions that makes sense, but if I'm in a git tree
then I don't want it to refuse because 1.11.0.dev0+githash1 compares equal
to 1.11.0.dev0+githash2. Especially after waiting a few minutes, see (1).
I've sent a (incomplete) fix for the shutil thing (
https://github.com/pypa/pip/pull/3219) and will comment on some open
issues on the pip tracker. But I'm thinking that for now we should go with
some printed message first. Something like "please use ``pip install .`` if
you want reliable uninstall behavior. See <somelink> for more details".
Pip has worked quite well for me in the past, but the above makes me thing
it's not much of an improvement over use of setuptools.....
Which version of pip?

Chuck
Ralf Gommers
2015-11-02 06:47:34 UTC
Permalink
Post by Charles R Harris
Post by Ralf Gommers
Post by Ralf Gommers
Post by Ralf Gommers
On Thu, Oct 29, 2015 at 8:11 PM, Warren Weckesser <
Post by Warren Weckesser
Post by Nathaniel Smith
Hi all,
Apparently it is not well known that if you have a Python project
source tree (e.g., a numpy checkout), then the correct way to install
it is NOT to type
python setup.py install # bad and broken!
but rather to type
pip install .
FWIW, I don't see any mention of this in the numpy docs, but I do see
a lot of instructions involving `setup.py build` and `setup.py install`.
See, for example, INSTALL.txt. Also see
http://docs.scipy.org/doc/numpy/user/install.html#building-from-source
So I guess it is not surprising that it is not well known.
Indeed, install docs are always hopelessly outdated. And we have too
many of them. There's duplicate info in INSTALL.txt and
http://scipy.org/scipylib/building/index.html for example. We should
probably just empty out INSTALL.txt and simply put a link in it to the html
docs.
https://github.com/numpy/numpy/issues/6599. Feel free to add stuff. Or
to go fix something:)
Oh, and: looking at this thread there haven't been serious unanswered
concerns (at least in my perception), so without more discussion I'd
interpret the current status as "go ahead".
Hmm, after some more testing I'm going to have to bring up a few concerns
1. ``pip install .`` still has a clear bug; it starts by copying
everything (including .git/ !) to a tempdir with shutil, which is very
slow. And the fix for that will go via ``setup.py sdist``, which is still
slow.
2. ``pip install .`` silences build output, which may make sense for some
usecases, but for numpy it just sits there for minutes with no output after
printing "Running setup.py install for numpy". Users will think it hangs
and Ctrl-C it. https://github.com/pypa/pip/issues/2732
3. ``pip install .`` refuses to upgrade an already installed development
version. For released versions that makes sense, but if I'm in a git tree
then I don't want it to refuse because 1.11.0.dev0+githash1 compares equal
to 1.11.0.dev0+githash2. Especially after waiting a few minutes, see (1).
I've sent a (incomplete) fix for the shutil thing (
https://github.com/pypa/pip/pull/3219) and will comment on some open
issues on the pip tracker. But I'm thinking that for now we should go with
some printed message first. Something like "please use ``pip install .`` if
you want reliable uninstall behavior. See <somelink> for more details".
Pip has worked quite well for me in the past, but the above makes me
thing it's not much of an improvement over use of setuptools.....
Which version of pip?
Latest master (it's 'develop' branch). Recent released versions will be the
same, because there are open issues for these things.

Ralf
Nathaniel Smith
2015-11-03 01:57:35 UTC
Permalink
[Adding distutils-sig to the CC as a heads-up. The context is that
numpy is looking at deprecating the use of 'python setup.py install'
and enforcing the use of 'pip install .' instead, and running into
some issues that will probably need to be addressed if 'pip install .'
is going to become the standard interface to work with source trees.]

On Sun, Nov 1, 2015 at 3:16 PM, Ralf Gommers <***@gmail.com> wrote:
[...]
Post by Ralf Gommers
Hmm, after some more testing I'm going to have to bring up a few concerns
1. ``pip install .`` still has a clear bug; it starts by copying everything
(including .git/ !) to a tempdir with shutil, which is very slow. And the
fix for that will go via ``setup.py sdist``, which is still slow.
Ugh. If 'pip (install/wheel) .' is supposed to become the standard way
to build things, then it should probably build in-place by default.
Working in a temp dir makes perfect sense for 'pip install
<requirement>' or 'pip install <url>', but if the user supplies an
actual named on-disk directory then presumably the user is expecting
this directory to be used, and to be able to take advantage of
incremental rebuilds etc., no?
Post by Ralf Gommers
2. ``pip install .`` silences build output, which may make sense for some
usecases, but for numpy it just sits there for minutes with no output after
printing "Running setup.py install for numpy". Users will think it hangs and
Ctrl-C it. https://github.com/pypa/pip/issues/2732
I tend to agree with the commentary there that for end users this is
different but no worse than the current situation where we spit out
pages of "errors" that don't mean anything :-). I posted a suggestion
on that bug that might help with the apparent hanging problem.
Post by Ralf Gommers
3. ``pip install .`` refuses to upgrade an already installed development
version. For released versions that makes sense, but if I'm in a git tree
then I don't want it to refuse because 1.11.0.dev0+githash1 compares equal
to 1.11.0.dev0+githash2. Especially after waiting a few minutes, see (1).
Ugh, this is clearly just a bug -- `pip install .` should always
unconditionally install, IMO. (Did you file a bug yet?) At least the
workaround is just 'pip uninstall numpy; pip install .', which is
still better the running 'setup.py install' and having it blithely
overwrite some files and not others.

The first and last issue seem like ones that will mostly only affect
developers, who should mostly have the ability to deal with these
weird issues (or just use setup.py install --force if that's what they
prefer)? This still seems like a reasonable trade-off to me if it also
has the effect of reducing the number of weird broken installs among
our thousands-of-times-larger userbase.

-n
--
Nathaniel J. Smith -- http://vorpus.org
Nathaniel Smith
2015-11-03 03:02:30 UTC
Permalink
Post by Nathaniel Smith
[Adding distutils-sig to the CC as a heads-up. The context is that
numpy is looking at deprecating the use of 'python setup.py install'
and enforcing the use of 'pip install .' instead, and running into
some issues that will probably need to be addressed if 'pip install .'
is going to become the standard interface to work with source trees.]
[...]
Post by Ralf Gommers
Hmm, after some more testing I'm going to have to bring up a few concerns
1. ``pip install .`` still has a clear bug; it starts by copying everything
(including .git/ !) to a tempdir with shutil, which is very slow. And the
fix for that will go via ``setup.py sdist``, which is still slow.
Ugh. If 'pip (install/wheel) .' is supposed to become the standard way
to build things, then it should probably build in-place by default.
Working in a temp dir makes perfect sense for 'pip install
<requirement>' or 'pip install <url>', but if the user supplies an
actual named on-disk directory then presumably the user is expecting
this directory to be used, and to be able to take advantage of
incremental rebuilds etc., no?
Thats what 'pip install -e .' does. 'setup.py develop' -> 'pip install -e
.'

I'm not talking about in place installs, I'm talking about e.g. building a
wheel and then tweaking one file and rebuilding -- traditionally build
systems go to some effort to keep track of intermediate artifacts and reuse
them across builds when possible, but if you always copy the source tree
into a temporary directory before building then there's not much the build
system can do.
Post by Nathaniel Smith
Post by Ralf Gommers
3. ``pip install .`` refuses to upgrade an already installed development
version. For released versions that makes sense, but if I'm in a git tree
then I don't want it to refuse because 1.11.0.dev0+githash1 compares equal
to 1.11.0.dev0+githash2. Especially after waiting a few minutes, see (1).
Ugh, this is clearly just a bug -- `pip install .` should always
unconditionally install, IMO. (Did you file a bug yet?) At least the
workaround is just 'pip uninstall numpy; pip install .', which is
still better the running 'setup.py install' and having it blithely
overwrite some files and not others.
There is a bug open. https://github.com/pypa/pip/issues/536
Thanks!

-n
Chris Barker - NOAA Federal
2015-11-03 17:10:16 UTC
Permalink
Post by Nathaniel Smith
I'm not talking about in place installs, I'm talking about e.g. building a
wheel and then tweaking one file and rebuilding -- traditionally build
systems go to some effort to keep track of intermediate artifacts and reuse
them across builds when possible, but if you always copy the source tree
into a temporary directory before building then there's not much the build
system can do.
This strikes me as an optimization -- is it an important one?

If I'm doing a lot of tweaking and re-running, I'm usually in develop mode.

I can see that when you build a wheel, you may build it, test it,
discover an wheel-specific error, and then need to repeat the cycle --
but is that a major use-case?

That being said, I have been pretty frustrated debugging conda-build
scripts -- there is a lot of overhead setting up the build environment
each time you do a build...

But with wheel building there is much less overhead, and far fewer
complications requiring the edit-build cycle.

And couldn't make-style this-has-already-been-done checking happen
with a copy anyway?

CHB
Ah yes. So I don't think pip should do what it does. It a violation of
the abstractions we all want to see within it. However its not me you
need to convince ;).
-Rob
--
Distinguished Technologist
HP Converged Cloud
_______________________________________________
https://mail.python.org/mailman/listinfo/distutils-sig
Nathaniel Smith
2015-11-06 21:56:45 UTC
Permalink
Post by Nathaniel Smith
Post by Ralf Gommers
2. ``pip install .`` silences build output, which may make sense for some
usecases, but for numpy it just sits there for minutes with no output after
printing "Running setup.py install for numpy". Users will think it hangs and
Ctrl-C it. https://github.com/pypa/pip/issues/2732
I tend to agree with the commentary there that for end users this is
different but no worse than the current situation where we spit out
pages of "errors" that don't mean anything :-). I posted a suggestion
on that bug that might help with the apparent hanging problem.
For the record, this is now fixed in pip's "develop" branch and should
be in the next release. For commands like 'setup.py install', pip now
displays a spinner that ticks over whenever the underlying process
prints to stdout/stderr. So if the underlying process hangs, then the
spinner will stop (it's not just lying to you), but normally it works
nicely.

https://github.com/pypa/pip/pull/3224

-n
--
Nathaniel J. Smith -- http://vorpus.org
Loading...