Reference resources Cython introduction - cloud + Community - Tencent cloud
setup I met many times this year , With python The constant depth of programming learning is important for python The understanding of is also increasing , Here is a simple section .
Related tools ：distutils,cython
1.Cython brief introduction
We usually use python, Also called CPython, Because he uses C Written language , Generally speaking , our python Source code (.py Watson piece ), First compile it into bytecode (.pyc file ), And then .pyc Files in python Running on a virtual machine , there python Virtual machines are so-called “python Interpreter ”. To make a long story short , pure python The running speed of the code is between the traditional compilation language and the traditional interpretation language .
2.1 What is bytecode ？
Bytecode stay python The virtual machine program corresponds to PyCodeObject object ; .pyc File is the representation of bytecode on disk .
Every one with py At the end of the python Source code files are all modules , The file that can run the whole program after startup is called the top-level file . The top-level file is imported into other modules （ file ）, You must find the file and compile it into bytecode , And run bytecode .
The bytecode compiled when the file is imported is automatically saved , The timestamp is also saved . If they exist at the same time .py and .pyc,python Will use .pyc function , If .pyc The compilation time of is earlier than .py Time for , Then recompile .py, And update the .pyc file .
If python Unable to write bytecode on machine , The program still works , Bytecode is generated in memory and discarded at the end of the program .（ Strictly speaking , The bytecode is saved only when the file is imported , Not for top-level files ）.
If you want to generate test.pyc, We can use python Built-in module py_compile To compile the . You can also execute commands python -m test.py such , It was generated test.pyc.
typedef struct {
PyObject_HEAD
int co_argcount; /* Number of position parameters */
int co_nlocals; /* Number of local variables */
int co_stacksize; /* Stack size */
int co_flags;
PyObject *co_code; /* Bytecode instruction sequence */
PyObject *co_consts; /* All constant sets */
PyObject *co_names; /* A collection of all symbol names */
PyObject *co_varnames; /* Set of local variable names */
PyObject *co_freevars; /* Collection of variable names for closures */
PyObject *co_cellvars; /* A collection of variable names referenced by internally nested functions */
/* The rest doesn’t count for hash/cmp */
PyObject *co_filename; /* The file name where the code is located */
PyObject *co_name; /* Module name | Function name | Class name */
int co_firstlineno; /* The starting line number of the code block in the file */
PyObject *co_lnotab; /* Correspondence between bytecode instruction and line number */
void *co_zombieframe; /* for optimization only (see frameobject.c) */
} PyCodeObject;2.2 Execution bytecode ？
Python The principle of virtual machine is to simulate executable program and then X86 Operation on the machine ,X86 The runtime stack frame of is shown in the figure below ： When a function call occurs , Create a new stack frame , Corresponding Python The realization of PyFrameObject object .PyFrameObject Object to create dynamic information when the program runs , That is, the execution environment , The relevant source code is as follows ：
typedef struct _frame{
PyObject_VAR_HEAD //" Runtime stack " The size of is uncertain
struct _frame *f_back; // Execute the previous in the environment chain frame, Many PyFrameObject Connected to form a linked list of execution environments
PyCodeObject *f_code; //PyCodeObject object , This frame This is this. PyCodeObject Object context
PyObject *f_builtins; //builtin Namespace
PyObject *f_globals; //global Namespace
PyObject *f_locals; //local Namespace
PyObject **f_valuestack; //" Runtime stack " At the bottom of the stack
PyObject **f_stacktop; //" Runtime stack " The top of the stack
//...
int f_lasti; // The last bytecode instruction is in f_code Offset position in
int f_lineno; // The source code line corresponding to the current bytecode
//...
// Dynamic memory , maintain ( local variable +cell A collection of objects +free A collection of objects + Runtime stack ) The space needed
PyObject *f_localsplus[1];
} PyFrameObject;every last PyFrameObject Objects maintain a PyCodeObject object , This shows that every PyFrameObject The dynamic memory space objects in are the same as a section in the source code Code Corresponding .
2.2 What is? Cython？
Cython yes Python Language extension module , His aim is to be called python The superset of language (superset), by python Provide advanced , Object oriented , Functional and dynamic programming . Its main function is to support （ Optional ） Some static types are declared as Cython Part of the language . such cython The source code can be transformed into optimized C/C++ Code , You can then program this code as python Expansion module for .Cython Code in CPython Execute in the runtime environment , But in compiled C Speed execution of , And can directly call C library . meanwhile , It retains Python The original interface of the source code , This allows it to be directly from Python The code uses .
2.3 structure Cython
Cython The code must be compiled , It includes two steps ：
First step , take A.pyx For documents Cython Compile to a .c In file , It contains python Extension module code
The second step , take .c File compiled into .so file （Windows for .pyd file ）, This file can direct import Get into Python session.Distutils or setuptools Responsible for this part . although Cython They can be called for you in some cases .
Specific examples , stay faser rcnn in ,bbox_overlaps Function is OK cython Written .
Specific construction Cython Several ways to ：
1. Write a distutils / setuptools setup.py.
setyp.py:
from distutils.core import setup
from Cython.Build import cythonize
setup(
ext_modules = cythonize("helloworld.pyx")
)Command line ：
$ python setup.py build_ext --inplace2. Use Pyximport, Import Cython .pyx Files are like they are .py file （ Use distutils Compile and build in the background ）. This method is better than writing a setup.py It's easier . But not very flexible , such as , You need some compilation options .（ Actually , I don't need any compilation options without learning the principles of compilation , Obviously this method is excellent ）. however , When my computer uses this method , failed , May be VS Environmental reasons , So I used the first method , Directly compiled a pyd Files use .
import pyximport; pyximport.install()
import helloworld# Hello World3.Jupyter notebook Allow inline Cython Code . This is the beginning of writing Cython The easiest way to code and run it .
2.distutils brief introduction
except Cython Used in setup.py, To write python Third party library , It is also necessary to write setup.py Of . In fact, if we downloaded some source code files of the third Library , After opening, there is usually a setup.py, perform python setup.py install You can install this library .setup.py How to write a lot of content , Please refer to the official documents ：https://wiki.python.org/moin/Distutils/Tutorial?highlight=%28setup.py%29. A typical setup.py Let's write it as follows ( Refer to the official document ):
The file structure is ： top |-- package | |-- __init__.py # Empty | |-- module.py # This package Module files in | `-- things | |-- cross.png | |-- fplogo.png | `-- tick.png |-- README `-- setup.py
setup.py：
from distutils.core import setup
#This is a list of files to install, and where
#(relative to the 'root' dir, where setup.py is)
#You could be more specific.
files = ["things/*"]
setup(name = "appname",
version = "100",
description = "yadda yadda",
author = "myself and I",
author_email = "[email protected]",
url = "whatever",
#Name the folder where your packages live:
#(If you have other packages (dirs) or modules (py files) then
#put them into the package directory - they will be found
#recursively.)
packages = ['package'],
#'package' package must contain files (see list above)
#I called the package 'package' thus cleverly confusing the whole issue...
#This dict maps the package name =to=> directories
#It says, package *needs* these files.
package_data = {'package' : files },
#'runner' is in the root.
scripts = ["runner"],
long_description = """Really long text here."""
#
#This next part it for the Cheese Shop, look a little down the page.
#classifiers = []
) Making the installation tarball
python setup.py sdistMy compilation process ：
setup.py:
import sys
import numpy as np
A=sys.path.insert(0, "..")
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
from Cython.Distutils import build_ext
# ext_module = cythonize("TestOMP.pyx")
ext_module = Extension(
"bbox",
["bbox.pyx"],
extra_compile_args=["/openmp"],
extra_link_args=["/openmp"],
)
setup(
cmdclass = {'build_ext': build_ext},
ext_modules = [ext_module],
# Note that this sentence must have , Otherwise, it will only be compiled into C Code , Cannot compile to pyd file
include_dirs=[np.get_include()]
) 
