[PYTHON] Use OpenBLAS with numpy, scipy

If you feel that numpy and scipy are slow, you should check the blas settings. The environment I tried was ubuntu 14.04.

To check, launch python

>>> import numpy
>>> numpy.show_config()

lapack_opt_info:
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    extra_compile_args = ['-msse3']
    define_macros = [('NO_ATLAS_INFO', 3)]
blas_opt_info:
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
    define_macros = [('NO_ATLAS_INFO', 3)]

Check at.

As a result of investigation, various methods such as using update-alternatives came out, but since that one had strange behavior, the method of directly setting OpenBLAS and compiling numpy is described as a memorandum.

reference

--Using BLAS on ubuntu 14.04 (link)

Advance preparation

Install python-dev and gfortran as preparations before installing OpenBLAS.

~$ sudo apt-get install git python-dev gfortran

OpenBLAS installation

Download OpenBLAS from git, make and install with gfortran. Make sure it is installed in / opt / openblas /.

~$ git clone https://github.com/xianyi/OpenBLAS
~$ cd OpenBLAS
~$ make FC=gfortran
~$ sudo make PREFIX=/opt/openblas install

After that, set the path to OpenBLAS. It is good to describe it in ~ / .bashrc etc.

~$ export LD_LIBRARY_PATH=/opt/OpenBLAS/lib:$LD_LIBRARY_PATH

When using with multiple users

~$ sudo -s "echo '/opt/OpenBLAS/lib' > /etc/ld.so.conf.d/openblas.conf"

After the setting is completed

~$ sudo ldconfig

Is executed.

numpy installation

Download numpy from git and describe openblas settings in site.cfg

~$ git clone https://github.com/numpy/numpy
~$ cd numpy
~$ cp site.cfg.example site.cfg
~$ vi site.cfg

Uncomment the following part of site.cfg.

[openblas]
libraries = openblas
library_dirs = /opt/OpenBLAS/lib
include_dirs = /opt/OpenBLAS/include

After that, check if the setting change is successful with ~ $ python setup.py config.

openblas_info:
  FOUND:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/opt/openblas/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]

  FOUND:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/opt/openblas/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]

After checking the above, build and install with ~ $ python setup.py build && python setup.py install.

Operation check

Check the speed with Code here. The number of threads to be used is specified by ʻOMP_NUM_THREADS` (the example below is a matrix calculation of 5000 * 5000).

~$ OMP_NUM_THREADS=1 python test_numpy.py
version: 1.12.0.dev0+3c394f7
maxint:  9223372036854775807

BLAS info:
 * libraries ['openblas', 'openblas']
 * library_dirs ['/opt/openblas/lib']
 * define_macros [('HAVE_CBLAS', None)]
 * language c

dot: 4.537050 sec
~$ OMP_NUM_THREADS=8 python test_numpy.py
version: 1.12.0.dev0+3c394f7
maxint:  9223372036854775807

BLAS info:
 * libraries ['openblas', 'openblas']
 * library_dirs ['/opt/openblas/lib']
 * define_macros [('HAVE_CBLAS', None)]
 * language c

dot: 1.855611 sec

scipy installation

If you can install numpy above, all you have to do is install scipy with pip.

pip install scipy

Recommended Posts

Use OpenBLAS with numpy, scipy
Use multithreaded BLAS / LAPACK with numpy / scipy
Use Numpy
Use Numpy, Scipy, scikit-learn on Heroku
LPC with Scipy
Use Numpy, Scipy, scikit-learn on Amazon Linux
CORDIC with Scipy
Using Intel MKL with NumPy / SciPy (November 2019 version)
Use mecab-ipadic-neologd with igo-python
Use RTX 3090 with PyTorch
Use ansible with cygwin
[Python] Use JSON with Python
Use Mock with pytest
Use indicator with pd.merge
Use Gentelella with django
Moving average with numpy
Use mecab with Python3
Use tensorboard with Chainer
Use DynamoDB with Python
Use pip with MSYS2
Use Python 3.8 with Anaconda
Use pyright with Spacemacs
Use python with docker
Use TypeScript with django-compressor
Create filter with scipy
How to use numpy
Use LESS with Django
Getting Started with Numpy
Use MySQL with Django
Learn with Cheminformatics NumPy
Matrix concatenation with Numpy
Hamming code with numpy
Normarize data with Scipy
Regression analysis with NumPy
Use Enums with SQLAlchemy
Use tensorboard with NNabla
Use GPS with Edison
Extend NumPy with Rust
Use nim with Jupyter
FFT processing with numpy and scipy and low pass filter
Kernel regression with Numpy only
Use Trello API with python
I wrote GP with numpy
Use shared memory with shared libraries
Use "$ in" operator with mongo-go-driver
Use custom tags with PyYAML
Use directional graphs with networkx
CNN implementation with just numpy
Artificial data generation with numpy
Use Twitter API with Python
Use pip with Jupyter Notebook
[Python] Calculation method with numpy
Try matrix operation with NumPy
Use DATE_FORMAT with SQLAlchemy filter
Use TUN / TAP with Python
Diffusion equation animation with NumPy
Use sqlite3 with NAO (Pepper)
See the power of speeding up with NumPy and SciPy
Implemented SMO with Python + NumPy
Stick strings together with Numpy
[python] A note when trying to use numpy with Cython