autocython 1.1¶
Chris Billington, Mar 23, 2018
View on PyPI | View on BitBucket | Read the docs
Installation¶
to install autocython
, run:
$ pip3 install autocython
or to install from source:
$ python3 setup.py install
Note
Works with Python 2.7 or Python 3.4+
Introduction¶
Like pyx_import
, but when you want to write your own setup.py still, and you want to keep the compiled extensions in the package directory.
“How is that at all like pyx_import
?” I hear you ask. “Why would I want that?”.
Well, autocython
records a hash of the .pyx
files and the resulting .so
or
.pyd
files whenever it compiles anything, and recompiles automatically (by
running your setup.py
) if it detects that there is a mismatch. So that’s how it’s
like pyx_import
.
The similarities end there. As mentioned, you have to write your own setup.py
. I
don’t see this as much of a drawback, it’s rare that I have a Cython extension that
doesn’t need at least some customisation, and that customisation
might as well go in a separate file than in a call to pyx_import
. See the example
below for how to write a setup.py that works with autocython
.
autocython
expects you to keep all the compiled extensions in the same directory,
even for different versions of Python and platforms. Whilst keeping multiple
versions of extensions for different platforms in the same directory is easy in
Python 3 (since extensions get a platform-specific suffix), it is less easy in
Python 2. So autocython
provides a platform-specific suffix that you can add to
the names of your extensions in your setup.py
(see example below), and an import
function that uses the same suffix to import the right version at run time. This
allows distributing fat packages with all the supported compiled versions of the
extension in the same folder. Whilst it is bad practice to distribute packages like
this, it is often what is most convenient if you’re a research group doing
numerical simulations or lab control systems and sharing code with each other
without wanting to think too hard about packaging or build servers.
But perhaps most importantly, autocython
serves as a reminder to myself as to
the current state of compiling cython extensions on all platform. It calls your
setup.py
as: python setup.py build_ext --inplace
, or on Windows as python
setup.py build_ext --inplace --compiler=msvc
. So long as your setup.py
imports
setuptools
, on Windows this means you actually get a meaningful error about where
to download the correct compiler from Microsoft. Even if you get these steps wrong,
autocython
prints a big fat error message describing what you need to make sure
you’ve done, which I intend to update whenever the state of Windows compiling
changes.
Note
As of March 2018, the compilers needed on Windows are available at: Python 2.7: http://aka.ms/vcpython27. Python 3.5+:”Microsoft Visual C++ Build Tools”, from http://aka.ms/BuildTools
Example usage¶
Below is an example of how to use autocython from a script. All recompilation is triggered by imports.
Note
You can also check and trigger recompilation by running python -m
autocython
in the directory containing the .pyx
files and your
setup.py
. This can be a more convenient way to compile for a specific
program than having to actually run the program doing the imports.
This example has a top-level script example.py
which imports the hello
function from a package hello_package
:
# example.py
from hello_package import hello
hello()
That package’s __init__.py
imports the hello
function from a Cython
extension hello_module.pyx``using ``autocython
, after using all extensions in
the folder are up to date:
# hello_package/__init__.py
import os
from autocython import ensure_extensions_compiled, import_extension
this_folder = os.path.dirname(os.path.abspath(__file__))
ensure_extensions_compiled(this_folder)
module = import_extension('hello_package.hello_module')
hello = module.hello
Note that it is important that the import line given to
import_extension
is a fully qualified, absolute import. If you are only
using Python 3, import_extension()
is unnecessary and you can just use
a normal import line (though you should still use it if your code needs to run on
both Python 2 and 3)
The Cython extension in the package is:
# hello_package/hello_module.pyx
def hello():
print('hello from cython!')
And the package directory also contains the setup.py
for compiling the
extension:
# hello_package/setup.py
from setuptools import setup
from setuptools.extension import Extension
from Cython.Distutils import build_ext
from autocython import PLATFORM_SUFFIX
ext_modules = [Extension("hello_module" + PLATFORM_SUFFIX, ["hello_module.pyx"])]
setup(
name = "hello_package",
cmdclass = {"build_ext": build_ext},
ext_modules = ext_modules,
)
Use of setuptools
is crucial on Windows, otherwise distutils
will not be
able to find the Microsoft compilers. Importing PLATFORM_SUFFIX
and appending
it to the extension name allows each version of the extension to have a platform-
specific unique name on Python 2 (import_extension()
makes sure it gets the
right one at import time). If you are only using Python 3, you don’t need to add
this suffix, but you still should if your code needs to run on both Pytohn 2 and 3
(in Python 3 PLATFORM_SUFFIX
is just an empty string)
The result of all this is:
$ python example.py
Extension(s) out of date, recompiling...
<compilation output>
hello from cython!
$ python example.py # again, no compilation output this time:
hello from cython!
$ python3 example.py # different Python version:
Extension(s) out of date, recompiling...
<compilation output>
hello from cython!
$ python example.py # original Python again, still no recompilation neccesary:
hello from cython!
$ ls hello_package/ # See what files have been generated:
autocython_compile_state.json hello_module.pyx __pycache__
hello_module.cpython-36m-x86_64-linux-gnu.so __init__.py setup.py
hello_module_py27_linux2_64bit.so __init__.pyc
Module reference¶
There are two functions and a constant:
-
autocython.
ensure_extensions_compiled
(folder, names=None)¶ Ensure the Cython extensions in the given folder with the given list of names are compiled, and if not (or if they are in need of recompilation), compile them by running
setup.py
(assumed to be in the same folder). If no names are given, they will be inferred from any.pyx
files in the folder. It is assumed that each cython file is called<name>.pyx
, and that each extension (as specified insetup.py
) is called<name><PLATFORM_SUFFIX>
, wherePLATFORM_SUFFIX
is a constant defined in this module that specifies the platform details for Python 2, allowingimport_extension()
to import the correct version of the extension if multiple versions exist for different platforms. In Python 3PLATFORM_SUFFIX
is the empty string since Python 3 does a similar thing automatically.
-
autocython.
import_extension
(fullname)¶ Import the extension, after appending
PLATFORM_SUFFIX
in order to ensure we get the right version for our platform. This is not neccesary on Python 3, which does a similar thing automatically if you use an ordinary import (On Python 3PLATFORM_SUFFIX
is an empty string).fullname
must be a fully qualified, absolute import. This function also inserts the module into sys.modules under the name fullname, and hence it will be available for ordinary import without this function, so long as this function is called once first (say in the__init__.py
of the package)
-
autocython.
PLATFORM_SUFFIX
¶ A platform-specific string that should be appended to extension names in
setup.py
(see above example) in order to make them uniquely named on a per- platform basis in Python 2. On Python 3 this is the empty string. On Python 2 it isb'_py27_{}_{}'.format(sys.platform, platform.architecture()[0])
, leading to extensions with names likehello_module_py27_linux2_64bit.so