RFC: building numpy against OpenBLAS.

[Ricardo Wurmus]

Hi Guix,

python-numpy currently depends on Atlas, which means that it cannot be
substituted with a binary built elsewhere.  OpenBLAS is an alternative
to Atlas and the binary can be used on all supported CPUs at runtime.
This makes it possible for us to make numpy substitutable.

We currently do not have a working OpenBLAS on MIPS, so the attached
patch selects OpenBLAS as an input only when not on MIPS.  Some
additional configuration happens only unless "atlas" is among the
inputs.

I have successfully compiled numpy and python-scikit-image for both
versions of Python on x86_64 with these changes.  I should say, however,
that there's a note in the numpy sources that warns about a bug under
certain conditions:

    # **Warning**: OpenBLAS, by default, is built in multithreaded mode. Due to 
the
    # way Python's multiprocessing is implemented, a multithreaded OpenBLAS can
    # cause programs using both to hang as soon as a worker process is forked on
    # POSIX systems (Linux, Mac).
    # This is fixed in Openblas 0.2.9 for the pthread build, the OpenMP build 
using
    # GNU openmp is as of gcc-4.9 not fixed yet.
    # Python 3.4 will introduce a new feature in multiprocessing, called the
    # "forkserver", which solves this problem. For older versions, make sure
    # OpenBLAS is built using pthreads or use Python threads instead of
    # multiprocessing.
    # (This problem does not exist with multithreaded ATLAS.)

I do not know if this is still a problem and I haven't yet tried to
reproduce this.  Our OpenBLAS is not built with openmp, so I think this
problem does not affect us.

Anyway, I just wanted to post this here to ask for opinions.  Maybe this
is a bad idea.  (In my case it makes sense not to use Atlas, because the
compile-time tuning is useless when a shared store is used and clients
use Atlas on machines other than the build host.)

Comments are very welcome!

~~ Ricardo

>From 4b56608674a9c7977cad88ce3f03a9fcb72c93bc Mon Sep 17 00:00:00 2001
From: Ricardo Wurmus <address@hidden>
Date: Fri, 22 May 2015 16:48:05 +0200
Subject: [PATCH] gnu: python-numpy: Build against OpenBLAS.

* gnu/packages/python.scm (python-numpy)[inputs]: Use "openblas" instead of
  "atlas" when not on MIPS.
* gnu/packages/python.scm (python-numpy)[arguments]: Configure build against
  OpenBLAS unless "atlas" is among the inputs.
---
 gnu/packages/python.scm | 38 ++++++++++++++++++++++++++++----------
 1 file changed, 28 insertions(+), 10 deletions(-)

diff --git a/gnu/packages/python.scm b/gnu/packages/python.scm
index 0916a75..c96296f 100644
--- a/gnu/packages/python.scm
+++ b/gnu/packages/python.scm
@@ -2283,7 +2283,9 @@ writing C extensions for Python as easy as Python 
itself.")
     (build-system python-build-system)
     (inputs
      `(("python-nose" ,python-nose)
-       ("atlas" ,atlas)))
+       ,(if (string-prefix? "mips" (%current-system))
+            `("atlas" ,atlas)
+            `("openblas" ,openblas))))
     (native-inputs
      `(("gfortran" ,gfortran-4.8)))
     (arguments
@@ -2291,16 +2293,32 @@ writing C extensions for Python as easy as Python 
itself.")
        (alist-cons-before
         'build 'set-environment-variables
         (lambda* (#:key inputs #:allow-other-keys)
-          (let* ((atlas-threaded
-                  (string-append (assoc-ref inputs "atlas")
-                                 "/lib/libtatlas.so"))
-                 ;; On single core CPUs only the serial library is created.
-                 (atlas-lib
-                  (if (file-exists? atlas-threaded)
-                      atlas-threaded
+          (if (assoc-ref inputs "atlas")
+              ;; Link against Atlas
+              (let* ((atlas-threaded
                       (string-append (assoc-ref inputs "atlas")
-                                     "/lib/libsatlas.so"))))
-            (setenv "ATLAS" atlas-lib)))
+                                     "/lib/libtatlas.so"))
+                     ;; On single core CPUs only the serial library is created.
+                     (atlas-lib
+                      (if (file-exists? atlas-threaded)
+                          atlas-threaded
+                          (string-append (assoc-ref inputs "atlas")
+                                         "/lib/libsatlas.so"))))
+                (setenv "ATLAS" atlas-lib))
+              ;; Link against OpenBLAS
+              (begin
+                (call-with-output-file "site.cfg"
+                  (lambda (port)
+                    (format port "[openblas]
+libraries = openblas
+library_dirs = ~a/lib
+include_dirs = ~a/include
+" (assoc-ref inputs "openblas") (assoc-ref inputs "openblas"))))
+                ;; Use "gcc" executable, not "cc".
+                (substitute* "numpy/distutils/system_info.py"
+                  (("c = distutils\\.ccompiler\\.new_compiler\\(\\)")
+                   "c = distutils.ccompiler.new_compiler(); 
c.set_executables(compiler='gcc',compiler_so='gcc',linker_exe='gcc',linker_so='gcc
 -shared')"))))
+          #t)
         ;; Tests can only be run after the library has been installed and not
         ;; within the source directory.
         (alist-cons-after
-- 
2.1.0