[Numpy-discussion] I can't tell if Numpy is configured properly with show

Discussion:

[Numpy-discussion] I can't tell if Numpy is configured properly with show_config()

Elliot Hallmark

2015-06-19 21:19:56 UTC

Debian Sid, 64-bit. I was trying to fix the problem of np.dot running very
slow.

I ended up uninstalling numpy, installing libatlas3-base through apt-get
and re-installing numpy. The performance of dot is greatly improved! But
I can't tell from any other method whether numpy is set up correctly.
Consider comparing the faster one to another in a virtual env that is still
slow:

###
fast one
###

In [1]: import time, numpy

In [2]: n=1000

In [3]: A = numpy.random.rand(n,n)

In [4]: B = numpy.random.rand(n,n)

In [5]: then = time.time(); C=numpy.dot(A,B); print time.time()-then
0.306427001953

In [6]: numpy.show_config()
blas_info:
libraries = ['blas']
library_dirs = ['/usr/lib']
language = f77
lapack_info:
libraries = ['lapack']
library_dirs = ['/usr/lib']
language = f77
atlas_threads_info:
NOT AVAILABLE
blas_opt_info:
libraries = ['blas']
library_dirs = ['/usr/lib']
language = f77
define_macros = [('NO_ATLAS_INFO', 1)]
atlas_blas_threads_info:
NOT AVAILABLE
openblas_info:
NOT AVAILABLE
lapack_opt_info:
libraries = ['lapack', 'blas']
library_dirs = ['/usr/lib']
language = f77
define_macros = [('NO_ATLAS_INFO', 1)]
atlas_info:
NOT AVAILABLE
lapack_mkl_info:
NOT AVAILABLE
blas_mkl_info:
NOT AVAILABLE
atlas_blas_info:
NOT AVAILABLE
mkl_info:
NOT AVAILABLE

###
slow one
###

In [1]: import time, numpy

In [2]: n=1000

In [3]: A = numpy.random.rand(n,n)

In [4]: B = numpy.random.rand(n,n)

In [5]: then = time.time(); C=numpy.dot(A,B); print time.time()-then
7.88430500031

In [6]: numpy.show_config()
blas_info:
libraries = ['blas']
library_dirs = ['/usr/lib']
language = f77
lapack_info:
libraries = ['lapack']
library_dirs = ['/usr/lib']
language = f77
atlas_threads_info:
NOT AVAILABLE
blas_opt_info:
libraries = ['blas']
library_dirs = ['/usr/lib']
language = f77
define_macros = [('NO_ATLAS_INFO', 1)]
atlas_blas_threads_info:
NOT AVAILABLE
openblas_info:
NOT AVAILABLE
lapack_opt_info:
libraries = ['lapack', 'blas']
library_dirs = ['/usr/lib']
language = f77
define_macros = [('NO_ATLAS_INFO', 1)]
atlas_info:
NOT AVAILABLE
lapack_mkl_info:
NOT AVAILABLE
blas_mkl_info:
NOT AVAILABLE
atlas_blas_info:
NOT AVAILABLE
mkl_info:
NOT AVAILABLE

#####

Further, in the following comparison between Cpython and converting to
numpy array for one operation, I get Cpython being faster by the same
amount in both environments. But another user got numpy being faster.

In [1]: import numpy as np

In [2]: pts = range(100,1000)

In [3]: pts[100] = 0

In [4]: %timeit pts_arr = np.array(pts); mini = np.argmin(pts_arr)
10000 loops, best of 3: 129 Âµs per loop

In [5]: %timeit mini = sorted(enumerate(pts))[0][1]
10000 loops, best of 3: 89.2 Âµs per loop

The other user got

In [29]: %timeit pts_arr = np.array(pts); mini = np.argmin(pts_arr)
10000 loops, best of 3: 37.7 Âµs per loop

In [30]: %timeit mini = sorted(enumerate(pts))[0][1]
10000 loops, best of 3: 69.2 Âµs per loop

And I can't help but wonder if there is further configuration I need
to make numpy faster, or if this is just a difference between out
machines

In the future, should I ignore show_config() and just do this dot product
test?

Any guidance would be appreciated.

Thanks,
Elliot

Sturla Molden

2015-06-19 21:33:56 UTC

Permalink

Post by Elliot Hallmark
And I can't help but wonder if there is further configuration I need
to make numpy faster, or if this is just a difference between out
machines

Try to build NumPy with Intel MKL or OpenBLAS instead.

ATLAS is only efficient on the host computer on which it is built, and even
there it is not very fast (but far better than the reference BLAS).

Sturla

Sebastian Berg

2015-06-20 09:09:58 UTC

Permalink

Post by Elliot Hallmark
Debian Sid, 64-bit. I was trying to fix the problem of np.dot running
very slow.
I ended up uninstalling numpy, installing libatlas3-base through
apt-get and re-installing numpy. The performance of dot is greatly
improved! But I can't tell from any other method whether numpy is set
up correctly. Consider comparing the faster one to another in a

Not that I really know this stuff, but one thing to be sure is probably
checking `ldd /usr/lib/python2.7/dist-packages/numpy/core/_dotblas.so`.
That is probably silly (I really never cared to learn this stuff), but I
think it can't go wrong....

About the other difference. Aside from CPU, etc. differences, I expect
you got a newer numpy version then the other user. Not sure which part
got much faster, but there were for example quite a few speedups in the
code converting to array, so I expect it is very likely that this is the
reason.

- Sebastian

Post by Elliot Hallmark
###
fast one
###
In [1]: import time, numpy
In [2]: n=1000
In [3]: A = numpy.random.rand(n,n)
In [4]: B = numpy.random.rand(n,n)
In [5]: then = time.time(); C=numpy.dot(A,B); print time.time()-then
0.306427001953
In [6]: numpy.show_config()
libraries = ['blas']
library_dirs = ['/usr/lib']
language = f77
libraries = ['lapack']
library_dirs = ['/usr/lib']
language = f77
NOT AVAILABLE
libraries = ['blas']
library_dirs = ['/usr/lib']
language = f77
define_macros = [('NO_ATLAS_INFO', 1)]
NOT AVAILABLE
NOT AVAILABLE
libraries = ['lapack', 'blas']
library_dirs = ['/usr/lib']
language = f77
define_macros = [('NO_ATLAS_INFO', 1)]
NOT AVAILABLE
NOT AVAILABLE
NOT AVAILABLE
NOT AVAILABLE
NOT AVAILABLE
###
slow one
###
In [1]: import time, numpy
In [2]: n=1000
In [3]: A = numpy.random.rand(n,n)
In [4]: B = numpy.random.rand(n,n)
In [5]: then = time.time(); C=numpy.dot(A,B); print time.time()-then
7.88430500031
In [6]: numpy.show_config()
libraries = ['blas']
library_dirs = ['/usr/lib']
language = f77
libraries = ['lapack']
library_dirs = ['/usr/lib']
language = f77
NOT AVAILABLE
libraries = ['blas']
library_dirs = ['/usr/lib']
language = f77
define_macros = [('NO_ATLAS_INFO', 1)]
NOT AVAILABLE
NOT AVAILABLE
libraries = ['lapack', 'blas']
library_dirs = ['/usr/lib']
language = f77
define_macros = [('NO_ATLAS_INFO', 1)]
NOT AVAILABLE
NOT AVAILABLE
NOT AVAILABLE
NOT AVAILABLE
NOT AVAILABLE
#####
Further, in the following comparison between Cpython and converting to
numpy array for one operation, I get Cpython being faster by the same
amount in both environments. But another user got numpy being faster.
In [1]: import numpy as np
In [2]: pts = range(100,1000)
In [3]: pts[100] = 0
In [4]: %timeit pts_arr = np.array(pts); mini = np.argmin(pts_arr)
10000 loops, best of 3: 129 Âµs per loop
In [5]: %timeit mini = sorted(enumerate(pts))[0][1]
10000 loops, best of 3: 89.2 Âµs per loop
The other user got
In [29]: %timeit pts_arr = np.array(pts); mini = np.argmin(pts_arr)
10000 loops, best of 3: 37.7 Âµs per loop
In [30]: %timeit mini = sorted(enumerate(pts))[0][1]
10000 loops, best of 3: 69.2 Âµs per loop
And I can't help but wonder if there is further configuration I need to make numpy faster, or if this is just a difference between out machines
In the future, should I ignore show_config() and just do this dot
product test?
Any guidance would be appreciated.
Thanks,
Elliot
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Elliot Hallmark

2015-06-20 20:02:45 UTC

Permalink

Well, here is the question that started this all. In the slow environment,
blas seems to be there and work well, but numpy doesn't use it!

In [1]: import time, numpy, scipy

In [2]: from scipy import linalg

In [3]: n=1000

In [4]: A = numpy.random.rand(n,n)

In [5]: B = numpy.random.rand(n,n)

In [6]: then = time.time(); C=scipy.dot(A,B); print time.time()-then
7.62005901337

In [7]: begin = time.time(); C=linalg.blas.dgemm(1.0,A,B);print time.time()
- begin
0.325305938721

In [8]: begin = time.time(); C=linalg.blas.ddot(A,B);print time.time() -
begin
0.0363020896912

Post by Sebastian Berg

Not that I really know this stuff, but one thing to be sure is probably
checking `ldd /usr/lib/python2.7/dist-packages/numpy/core/_dotblas.so`.
That is probably silly (I really never cared to learn this stuff), but I
think it can't go wrong....
About the other difference. Aside from CPU, etc. differences, I expect
you got a newer numpy version then the other user. Not sure which part
got much faster, but there were for example quite a few speedups in the
code converting to array, so I expect it is very likely that this is the
reason.
- Sebastian

make numpy faster, or if this is just a difference between out machines

Post by Elliot Hallmark
In the future, should I ignore show_config() and just do this dot
product test?
Any guidance would be appreciated.
Thanks,
Elliot
_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion

_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Elliot Hallmark

2015-06-20 20:08:40 UTC

Permalink

Sebastian, in the slow virtual-env, _dotblas.so isn't there. I only have
_dummy.so

Post by Elliot Hallmark
Well, here is the question that started this all. In the slow
environment, blas seems to be there and work well, but numpy doesn't use
it!
In [1]: import time, numpy, scipy
In [2]: from scipy import linalg
In [3]: n=1000
In [4]: A = numpy.random.rand(n,n)
In [5]: B = numpy.random.rand(n,n)
In [6]: then = time.time(); C=scipy.dot(A,B); print time.time()-then
7.62005901337
In [7]: begin = time.time(); C=linalg.blas.dgemm(1.0,A,B);print
time.time() - begin
0.325305938721
In [8]: begin = time.time(); C=linalg.blas.ddot(A,B);print time.time() -
begin
0.0363020896912
On Sat, Jun 20, 2015 at 4:09 AM, Sebastian Berg <

Post by Sebastian Berg

Not that I really know this stuff, but one thing to be sure is probably
checking `ldd /usr/lib/python2.7/dist-packages/numpy/core/_dotblas.so`.
That is probably silly (I really never cared to learn this stuff), but I
think it can't go wrong....
About the other difference. Aside from CPU, etc. differences, I expect
you got a newer numpy version then the other user. Not sure which part
got much faster, but there were for example quite a few speedups in the
code converting to array, so I expect it is very likely that this is the
reason.
- Sebastian

make numpy faster, or if this is just a difference between out machines

_______________________________________________
NumPy-Discussion mailing list
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Charles R Harris

2015-06-20 20:45:42 UTC

Permalink

Post by Elliot Hallmark
Sebastian, in the slow virtual-env, _dotblas.so isn't there. I only have
_dummy.so

What numpy version?

<snip>

Chuck

Elliot Hallmark

2015-06-20 21:06:00 UTC

Permalink

Post by Charles R Harris
What numpy version?

1.8.1