WIP pypy3.3 - tests still fail? Anyone see why?

[Danny Milosavljevic]

Hi,

here's a work-in-progress package for pypy3.3. It builds - but the tests fail. 
A build takes about 6 GB of RAM and about 9 hours.

Maybe someone can spot how to fix one of the tests (usually they don't find 
libraries etc). (I'll work it out eventually but I thought I'd ask)

Also if guix build does fail executing some test - can I continue there somehow?

What do we do about setupterm? Disable the test? If hydra builds it there's 
probably no terminal available, right? The test it just testing readline's 
history manipulation so it's not like it's using the terminal...
I've already disabled test_curses.

(define-public pypy3.3
  (package
    (name "pypy3.3")
    (version "5.2.0")
    (source (origin
              (method url-fetch)
              (uri (string-append "https://bitbucket.org/pypy/pypy/downloads/";
                                  "pypy3.3-v" version "-alpha1-src.tar.bz2"))
              (sha256
               (base32
                "09hllsxds9ck22m92gy0ii9rwgqdp1d51c2bjrs15sl2ih42yk1l"))
              (patches (search-patches "pypy3.3-fix-compiler.patch"))
              (file-name (string-append "pypy3.3-v"
                                        version
                                        "-alpha1-src.tar.bz2"))))
    (build-system gnu-build-system)
    ;; TODO libgc for the untranslated tests
    (native-inputs
     ;; Note: python2-minimal doesn't work because ctypes.c_long is missing.
     `(("python-2" ,python-2)
       ("pkg-config" ,pkg-config)
       ("tar" ,tar)
       ("gzip" ,gzip)))
    (inputs
     `( ; for some reason this is not needed. ("bash" ,bash) ; actually /bin/sh
       ("libffi" ,libffi)
       ("zlib" ,zlib)
       ("ncurses" ,ncurses)
       ("openssl" ,openssl)
       ("expat" ,expat)
       ("bzip2" ,bzip2)
       ("sqlite" ,sqlite)
       ("gdbm" ,gdbm)
       ("tcl" ,tcl)
       ("tk" ,tk)
       ("xz" ,xz))) ; lzma
    (arguments
      `( ;#:modules ((srfi srfi-1)
          ;         (guix build utils))
        #:phases (modify-phases %standard-phases
         (delete 'configure)
         (replace 'build
            (lambda* (#:key inputs #:allow-other-keys)
              ;; distutils is used to verify extensions.
              ;; However, distutils build_ext uses a compiler that can only
              ;; be specified via CC environment variable.
              ;; Should have been used by customize_compiler() but wasn't.
              ;; Therefore we patch as well (in addition to setting it here).
              (setenv "CC" "gcc")
              ;; TODO set CPPFLAGS to something useful.
              ;; TODO set LDFLAGS rpath and other stuff
              (setenv "CCSHARED" "-fPIC")
               ;; TODO set LDFLAGS -L for tcl, tk, sqlite and rpath (mostly for 
the verification)
              (substitute* '("rpython/rtyper/lltypesystem/ll2ctypes.py")
                ;; Python ctypes.util.find_library documentation specifies
                ;; that it returns a file path, however all the actual
                ;; implementation cases return a SONAME.
                ;; Pypy expects it to return a file path.
                ;; Therefore, hotpatch ctypes.util to disable SONAME
                ;; determination.
                (("    import ctypes.util") "    import ctypes.util
    ctypes.util._get_soname = lambda x: x"))
              (substitute* '("lib-python/3/subprocess.py")
                (("\"/bin/sh\"") (string-append "\"" (which "sh") "\"")))
              (substitute* '("lib-python/3/distutils/log.py") ; FIXME remove
                (("__init__.self, threshold=WARN.") "__init__(self, 
threshold=INFO)")
                (("_global_log.threshold = level") "_x = level"))
              (substitute* '("lib_pypy/_tkinter/tklib_build.py")
                (("/usr/include/tcl") (string-append (assoc-ref inputs "tcl") 
"/include"))
                (("'tcl' + _ver, 'tk' + _ver") "'tcl8.6', 'tk8.6'") ; FIXME 
dyna version
)
              (substitute* '("lib_pypy/_sqlite3_build.py")
                (("libraries=.'sqlite3.$") (string-append "libraries= 
['sqlite3'], "
                                                           "include_dirs='" 
(assoc-ref inputs "sqlite")  "/include', "
                                                           "library_dirs='" 
(assoc-ref inputs "sqlite") "/lib'")))
              ;; FIXME just use main Makefile
              (chdir "pypy/goal") ; FIXME error checking
              (and
                (zero? (system* "python" "../../rpython/bin/rpython"
                                "-Ojit" "targetpypystandalone")))))
         (replace 'check
            (lambda* (#:key inputs outputs #:allow-other-keys)
              (chdir "../..")
              (system* "pwd")
              (system* "ls") ; FIXME remove
              (display (getcwd))
              (setenv "LC_ALL" "en_US.utf8") ; make sure Python Unicode works
              (substitute* '("lib-python/3/test/test_shutil.py")
                ;; Make sure that the tests don't try to use uid / grp
                ;; (because we have no passwd entries in the container).
                (("UID_GID_SUPPORT = True") "UID_GID_SUPPORT = False")
                (("#!/bin/sh") (string-append "#!" (which "sh"))))
              (substitute* '("lib-python/3/test/test_posixpath.py")
                ;; Make sure that the tests don't try to use $HOME
                (("def test_expanduser") "
@unittest.skipIf(True, "Disabled on GNU Guix")
def test_expanduser"))
              (setenv "PYTHONPATH" (string-append (getcwd) ":"
                                                  (getcwd) "/lib_pypy:" ; for 
'_testcapi'
                                                  (getenv "PYTHONPATH")))
              (and
                ;; TODO: " ./pypy/test_all.py"
                ;;       "lib-python/2.7.0/test/test_complex.py"
                ;;       "--pypy=foobar"
                (zero? (system* "mkdir" "/tmp/pytest"))
                (zero? (system* "python" "-S", "testrunner/runner.py"
                                "--config=pypy/pytest-A.cfg" ; FIXME logfile is 
mandatory!
                                "--config=pypy/pytest-A.py"
                                "--logfile=/tmp/pytest/runner.log"
                                (string-append "--root=" (getcwd))
                                "--timeout=3600")) ; --config=~/machine-A_cfg.py
                (zero? (system* "python" "-S", "pypy/test_all.py"
                                "--pypy=pypy/goal/pypy-c" "--timeout=3600"
                                "lib-python"))
                (zero? (system* "python" "-S", "pypy/test_all.py"
                                "--pypy=pypy/goal/pypy-c"
                                "pypy/module/pypyjit/test"))
                (zero? (system* "pypy/goal/pypy-c" "pypy/test_all.py"
                                "pypy/module/pypyjit/test_pypy_c")))))
         (replace 'install
            (lambda* (#:key outputs #:allow-other-keys)
              (chdir "../tool/release")
              ; TODO package.py --archive-name foobar
              (zero? (system* "./package.py pypy-VER-PLATFORM")))))))
    (home-page "http://www.pypy.org/";)
    (synopsis "Python 3.3 JIT")
    (description "This package provides a Python 3.3 just-in-time compiler.")
    (license (list license:expat
                   license:psfl
                   license:asl2.0
                   license:gpl2+
                   license:bsd-2
                   license:bsd-3
                   (license:non-copyleft
                    "http://www.unicode.org/copyright.html";)))))

TODO:
        Make it find '_testcapi' which is in
./pypy3.3-v5.2.0-alpha1-src/lib_pypy/__pycache__/_pypy_testcapi.pypy3-52.pyc
./pypy3.3-v5.2.0-alpha1-src/lib_pypy/_testcapi.py
        Make it find sqlite (still).

FAIL lib-python/3/test/test_capi.py::unmodified
        can't find module '_testcapi'
FAIL lib-python/3/test/test_compileall.py::unmodified
        pyo vs. pyc
FAIL lib-python/3/test/test_ctypes.py::unmodified
        can't find module 'ctypes_test'
FAIL lib-python/3/test/test_decimal.py::unmodified
        MemoryError (MPD_Malloc_error) in test_context_subclassing, others.
        Invalid Operation in _eval_equation
FAIL lib-python/3/test/test_faulthandler.py::unmodified
        Traceback capturing
FAIL lib-python/3/test/test_imp.py::unmodified
        test_imp: test_load_dynamic_ImportError_path 'NoneType' object is not 
iterable
FAIL lib-python/3/test/test_keywordonlyarg.py::unmodified
        test.test_keywordonlyarg.KeywordOnlyArgTestCase
                Captured error message isn't nice
FAIL lib-python/3/test/test_numeric_tower.py::unmodified
        test_integers fails (equal hash: InvalidOperation)
        test_numeric_tower.ComparisonTest: MemoryError
FAIL lib-python/3/test/test_peepholer.py::unmodified
        test_binary_subscr_on_unicode IR
FAIL lib-python/3/test/test_posixpath.py::unmodified
        Fixed; skipping now
FAIL lib-python/3/test/test_readline.py::unmodified
        _minimal_curses.error: setupterm: could not find terminfo database
FAIL lib-python/3/test/test_runpy.py::unmodified
        Main recursion (overflow) test fails