[Numpy-discussion] [SciPy-Dev] Setting up a dev environment with conda

[Nathaniel Smith]

Hi Luke,

For day-to-day development and testing of numpy, I don't bother with
either inplace builds *or* installing it -- I just use the magical
"runtests.py" script that you'll find in the root of your git
checkout.

E.g., to build and then test the (possibly modified) source in your
current checkout, just do:

./runtests.py

That's all. This builds into a hidden directory and then sets up the
correct PYTHONPATH before running the tests etc. -- you don't have to
worry about any of it, it's magic.

There are also lots of options, see ./runtests.py --help. Try adding
-j for multi-core builds, or you can specify arbitrary options to pass
to nose, or you can run it under gdb (there's an example in --help),
or if you just want an interactive shell to futz around in manually
instead of running the test suite then try passing --ipython.

BTW, numpy has its own mailing list at numpy-***@scipy.org
(CC'ed), which is where numpy development discussions usually take
place -- this list is more for scipy-the-package itself. There's lots
of overlap in readership between the two lists, but numpy-discussion
will probably give you quicker and more useful answers to questions
like this in general :-)

-n

On Sat, Oct 17, 2015 at 4:03 PM, Luke Zoltan Kelley

Thanks Nathan, I'll try that. Both without the inplace build, I'll have to
rebuild and install everytime I want to test something, right?
My best guess is to nuke and reclone Numpy, then do setup.py install without
the inplace build. What you're doing seems like it should work, though, so
I'm not sure what's going on.
Nathan Woods
When trying to do that is when I get the error I described in the OP. i.e.
I get an error when trying to install.
It would get installed into whatever conda environment you had activated.
_______________________________________________
SciPy-Dev mailing list
https://mail.scipy.org/mailman/listinfo/scipy-dev
_______________________________________________
SciPy-Dev mailing list
https://mail.scipy.org/mailman/listinfo/scipy-dev
_______________________________________________
SciPy-Dev mailing list
https://mail.scipy.org/mailman/listinfo/scipy-dev
_______________________________________________
SciPy-Dev mailing list
https://mail.scipy.org/mailman/listinfo/scipy-dev

[Luke Zoltan Kelley]

Thanks for the help Nathaniel --- but building via `./runtests.py` is failing in the same way. Hopefully Numpy-discussion can help me out.

I'm able to build using `python setup.py build_ext --inplace` but both trying to run `python setup.py install` or `./runtests.py` leads to the following error:

(numpy-py27)daedalus-2:numpy lzkelley$ ./runtests.py
Building, see build.log...
Running from numpy source directory.
Traceback (most recent call last):
File "setup.py", line 264, in <module>
setup_package()
File "setup.py", line 248, in setup_package
from numpy.distutils.core import setup
File "/Users/lzkelley/Programs/public/numpy/numpy/distutils/__init__.py", line 21, in <module>
from numpy.testing import Tester
File "/Users/lzkelley/Programs/public/numpy/numpy/testing/__init__.py", line 14, in <module>
from .utils import *
File "/Users/lzkelley/Programs/public/numpy/numpy/testing/utils.py", line 17, in <module>
from numpy.core import float32, empty, arange, array_repr, ndarray
File "/Users/lzkelley/Programs/public/numpy/numpy/core/__init__.py", line 59, in <module>
test = Tester().test
File "/Users/lzkelley/Programs/public/numpy/numpy/testing/nosetester.py", line 180, in __init__
if raise_warnings is None and '.dev0' in np.__version__:
AttributeError: 'module' object has no attribute '__version__'

Build failed!

Has anyone seen something like this before?

Thanks!
Luke

[Feng Yu]

Hi Luke,

Could you check if you have "/Users/lzkelley/Programs/public/numpy/ in
your PYTHONPATH?

I would also suggest you add a print(np) line before the crash in
nosetester.py. I got something like this (which didn't crash):

<module 'numpy' from
'/home/yfeng1/source/numpy/build/testenv/lib64/python2.7/site-packages/numpy/__init__.pyc'>

If you see something not starting with 'numpy/build', then it is again
pointing at PYTHONPATH.

I hope these helps.

Best,

- Yu

Post by Luke Zoltan Kelley
Thanks for the help Nathaniel --- but building via `./runtests.py` is
failing in the same way. Hopefully Numpy-discussion can help me out.
I'm able to build using `python setup.py build_ext --inplace` but both
trying to run `python setup.py install` or `./runtests.py` leads to the
(numpy-py27)daedalus-2:numpy lzkelley$ ./runtests.py
Building, see build.log...
Running from numpy source directory.
File "setup.py", line 264, in <module>
setup_package()
File "setup.py", line 248, in setup_package
from numpy.distutils.core import setup
File "/Users/lzkelley/Programs/public/numpy/numpy/distutils/__init__.py",
line 21, in <module>
from numpy.testing import Tester
File "/Users/lzkelley/Programs/public/numpy/numpy/testing/__init__.py",
line 14, in <module>
from .utils import *
File "/Users/lzkelley/Programs/public/numpy/numpy/testing/utils.py", line 17, in <module>
from numpy.core import float32, empty, arange, array_repr, ndarray
File "/Users/lzkelley/Programs/public/numpy/numpy/core/__init__.py", line 59, in <module>
test = Tester().test
File "/Users/lzkelley/Programs/public/numpy/numpy/testing/nosetester.py",
line 180, in __init__
AttributeError: 'module' object has no attribute '__version__'
Build failed!
Has anyone seen something like this before?
Thanks!
Luke
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Luke Zoltan Kelley]

Thanks Yu,

There was nothing in my PYTHONPATH at first, and adding my numpy directory ('/Users/lzkelley/Programs/public/numpy') didn't help (same error). In both cases, adding 'print(np)' yields:

<module 'numpy' from '/Users/lzkelley/Programs/public/numpy/numpy/__init__.pyc'>

Post by Nathaniel Smith
Hi Luke,
Could you check if you have "/Users/lzkelley/Programs/public/numpy/ in
your PYTHONPATH?
I would also suggest you add a print(np) line before the crash in
<module 'numpy' from
'/home/yfeng1/source/numpy/build/testenv/lib64/python2.7/site-packages/numpy/__init__.pyc'>
If you see something not starting with 'numpy/build', then it is again
pointing at PYTHONPATH.
I hope these helps.
Best,
- Yu

Post by Luke Zoltan Kelley
Thanks for the help Nathaniel --- but building via `./runtests.py` is
failing in the same way. Hopefully Numpy-discussion can help me out.
I'm able to build using `python setup.py build_ext --inplace` but both
trying to run `python setup.py install` or `./runtests.py` leads to the
(numpy-py27)daedalus-2:numpy lzkelley$ ./runtests.py
Building, see build.log...
Running from numpy source directory.
File "setup.py", line 264, in <module>
setup_package()
File "setup.py", line 248, in setup_package
from numpy.distutils.core import setup
File "/Users/lzkelley/Programs/public/numpy/numpy/distutils/__init__.py",
line 21, in <module>
from numpy.testing import Tester
File "/Users/lzkelley/Programs/public/numpy/numpy/testing/__init__.py",
line 14, in <module>
from .utils import *
File "/Users/lzkelley/Programs/public/numpy/numpy/testing/utils.py", line
17, in <module>
from numpy.core import float32, empty, arange, array_repr, ndarray
File "/Users/lzkelley/Programs/public/numpy/numpy/core/__init__.py", line
59, in <module>
test = Tester().test
File "/Users/lzkelley/Programs/public/numpy/numpy/testing/nosetester.py",
line 180, in __init__
AttributeError: 'module' object has no attribute '__version__'
Build failed!
Has anyone seen something like this before?
Thanks!
Luke
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Michael Sarahan]

Running tests in the folder might be causing your problem. If it's trying
to import numpy, and numpy is a folder in your current folder, sometimes
you see errors like this. The confusion is that Python treats folders
(packages) similarly to modules, and the resolution order sometimes bites
you. Try cd'ing to a different folder (importantly, one NOT containing a
numpy folder!) and run the test command from there.

HTH,
Michael

Post by Luke Zoltan Kelley
Thanks Yu,
There was nothing in my PYTHONPATH at first, and adding my numpy directory
('/Users/lzkelley/Programs/public/numpy') didn't help (same error). In
<module 'numpy' from
'/Users/lzkelley/Programs/public/numpy/numpy/__init__.pyc'>

Post by Nathaniel Smith
Hi Luke,
Could you check if you have "/Users/lzkelley/Programs/public/numpy/ in
your PYTHONPATH?
I would also suggest you add a print(np) line before the crash in
<module 'numpy' from

'/home/yfeng1/source/numpy/build/testenv/lib64/python2.7/site-packages/numpy/__init__.pyc'>

Post by Nathaniel Smith
If you see something not starting with 'numpy/build', then it is again
pointing at PYTHONPATH.
I hope these helps.
Best,
- Yu

Post by Luke Zoltan Kelley
Thanks for the help Nathaniel --- but building via `./runtests.py` is
failing in the same way. Hopefully Numpy-discussion can help me out.
I'm able to build using `python setup.py build_ext --inplace` but both
trying to run `python setup.py install` or `./runtests.py` leads to the
(numpy-py27)daedalus-2:numpy lzkelley$ ./runtests.py
Building, see build.log...
Running from numpy source directory.
File "setup.py", line 264, in <module>
setup_package()
File "setup.py", line 248, in setup_package
from numpy.distutils.core import setup
File

"/Users/lzkelley/Programs/public/numpy/numpy/distutils/__init__.py",

Post by Nathaniel Smith

Post by Luke Zoltan Kelley
line 21, in <module>
from numpy.testing import Tester
File "/Users/lzkelley/Programs/public/numpy/numpy/testing/__init__.py",
line 14, in <module>
from .utils import *
File "/Users/lzkelley/Programs/public/numpy/numpy/testing/utils.py",

line

Post by Nathaniel Smith

Post by Luke Zoltan Kelley
17, in <module>
from numpy.core import float32, empty, arange, array_repr, ndarray
File "/Users/lzkelley/Programs/public/numpy/numpy/core/__init__.py",

line

Post by Nathaniel Smith

Post by Luke Zoltan Kelley
59, in <module>
test = Tester().test
File

"/Users/lzkelley/Programs/public/numpy/numpy/testing/nosetester.py",

Post by Nathaniel Smith

Post by Luke Zoltan Kelley
line 180, in __init__
AttributeError: 'module' object has no attribute '__version__'
Build failed!
Has anyone seen something like this before?
Thanks!
Luke
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Nathaniel Smith]

Post by Michael Sarahan
Running tests in the folder might be causing your problem. If it's trying
to import numpy, and numpy is a folder in your current folder, sometimes you
see errors like this. The confusion is that Python treats folders
(packages) similarly to modules, and the resolution order sometimes bites
you. Try cd'ing to a different folder (importantly, one NOT containing a
numpy folder!) and run the test command from there.

This isn't the problem -- ./runtests.py is designed to work fine when
run from the root of the numpy checkout.

You might try nuking your checkout and environment and starting over
just in case your earlier attempts left behind some broken detritus
somewhere. 'git clean -xdf' will clear everything out of a git
directory aside from tracked files (so make sure to add any new files
you want to keep first!).

-n

[Luke Zoltan Kelley]

I tried cleaning the git dir, and trying again. It still didn't work giving me the report:

======================================================================
ERROR: test_scripts.test_f2py
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/lzkelley/anaconda/envs/numpy-py27/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/Users/lzkelley/Programs/public/numpy/build/testenv/lib/python2.7/site-packages/numpy/testing/decorators.py", line 146, in skipper_func
return f(*args, **kwargs)
File "/Users/lzkelley/Programs/public/numpy/build/testenv/lib/python2.7/site-packages/numpy/tests/test_scripts.py", line 68, in test_f2py
code, stdout, stderr = run_command([f2py_cmd, '-v'])
File "/Users/lzkelley/Programs/public/numpy/build/testenv/lib/python2.7/site-packages/numpy/tests/test_scripts.py", line 48, in run_command
proc = Popen(cmd, stdout=PIPE, stderr=PIPE)
File "/Users/lzkelley/anaconda/envs/numpy-py27/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/Users/lzkelley/anaconda/envs/numpy-py27/lib/python2.7/subprocess.py", line 1335, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

----------------------------------------------------------------------
Ran 6029 tests in 82.132s

FAILED (KNOWNFAIL=6, SKIP=10, errors=2)

f2py itself did seem to work fine from the command-line...

I did get things (seemingly) working, by cleaning the dir again, and then running:

python setupegg.py develop --user

This built properly, and now lets me make modification to the source files and have them take effect immediately. This is really all I need for now, but I will try to get the `./runtests.py` working for the future. Perhaps the problem is something to do with my previous python environment installed via macports...

Post by Nathaniel Smith

Post by Michael Sarahan
Running tests in the folder might be causing your problem. If it's trying
to import numpy, and numpy is a folder in your current folder, sometimes you
see errors like this. The confusion is that Python treats folders
(packages) similarly to modules, and the resolution order sometimes bites
you. Try cd'ing to a different folder (importantly, one NOT containing a
numpy folder!) and run the test command from there.

This isn't the problem -- ./runtests.py is designed to work fine when
run from the root of the numpy checkout.
You might try nuking your checkout and environment and starting over
just in case your earlier attempts left behind some broken detritus
somewhere. 'git clean -xdf' will clear everything out of a git
directory aside from tracked files (so make sure to add any new files
you want to keep first!).
-n
--
Nathaniel J. Smith -- http://vorpus.org
_______________________________________________
NumPy-Discussion mailing list
https://mail.scipy.org/mailman/listinfo/numpy-discussion