How to compile Python

Hello, Habr!

I want to tell you about an amazing event that I learned about a couple of months ago. It turns out that one popular python utility has been distributed for over a year as binaries that are compiled directly from python. And it's not about banal packaging by some PyInstaller , but about honest Ahead-of-time compilation of the whole python package. If you are as surprised as I am, welcome to cat.

Let me explain why I think this event is truly amazing. There are two types of compilation:  Ahead-of-time (AOT) , when all the code is compiled before starting the program, and Just in time compiler (JIT) , when the program is directly compiled for the required processor architecture during its execution. In the second case, the initial launch of the program is carried out by a virtual machine or interpreter.

If we group popular programming languages ​​by compilation type, we get the following list:

• Ahead-of-time compiler: C, C ++, Rust, Kotlin, Nim, D, Go, Dart;

  
  
  
  
  

• Just in time compiler: Lua, C #, Groovy, Dart.

  
  
  
  
  

There is no JIT compiler out of the box in python, but separate libraries that provide such an opportunity have been around for a long time

, : . : Kotlin JIT JavaVM, AOT Kotlin/Native. Dart ( 2). A JIT-, . 

, , . .

JIT , . JIT . AOT , ? , .

, , . mypy - python-.

2019 , . — mypyc. , “ 4 Python-” mypy ( : 1, 2, 3). mypyc: mypy python- Dropbox, , . , : go cython. — AOT python-.

, mypy , . mypy “” python, , mypyc .

, , python-. Python c 3.4 , mypy . , python , AOT . , mypyc !

bubble_sort

“”.  lib.py:

def bubble_sort(data):
   n = len(data)
   for i in range(n - 1):
       for j in range(n - i - 1):
           if data[j] > data[j + 1]:
               buff = data[j]
               data[j] = data[j + 1]
               data[j + 1] = buff
   return data
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

, mypyc . , mypyc. , mypy, mypyc ! mypyc, :

> mypyc lib.py
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

:

• .mypy_cache
  
  
  
    — mypy , mypyc mypy AST;

  
  
  
  
  

• build
  
  
  
  — ;

  
  
  
  
  

• lib.cpython-38-x86_64-linux-gnu.so
  
  
  
  — . CPython Extension.

  
  
  
  
  

CPython Extension — CPython , /C++. , CPython lib. , python.

:

1. python ;

   
   
   
   
   

2. .so , mypyc gcc (gcc python-dev ).

   
   
   
   
   

lib.cpython-38-x86_64-linux-gnu.so



lib.py , .

. main.py :

import lib


data = lib.bubble_sort(list(range(5000, 0, -1)))

assert data == list(range(1, 5001))
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

:

(~ 2 ), , . .

“ ”, . , .  

sum(a, b)

:

def sum(a, b):
  return a + b
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

:

int sum(int a, int b) {
   return a + b;
}
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

c ( ):

PyObject *CPyDef_sum(PyObject *cpy_r_a, PyObject *cpy_r_b){
    return PyNumber_Add(cpy_r_a, cpy_r_b);
}
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

, . -, , PyObject, CPython . , , : , , , , . mypyc? 

, : CPython . PyNumber_Add — Python, , Python .

CPython c Extension :

 — - sum A, B;

 — , , A + B;

 — ;

 — , - .

: , , . 

, , , mypyc , .

sum(a: int, b: int)

, python, , , . , . , CPython - Extension. ? 

, , , , CPython. mypyc , . mypyc , , sum. , , . , -, :

def sum(a: int, b: int):
  return a + b
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

C ( ):

PyObject *CPyDef_sum(CPyTagged cpy_r_a, CPyTagged cpy_r_b) {
   CPyTagged cpy_r_r0;
   PyObject *cpy_r_r1;
   cpy_r_r0 = CPyTagged_Add(cpy_r_a, cpy_r_b);
   cpy_r_r1 = CPyTagged_StealAsObject(cpy_r_r0);

   return cpy_r_r1;
}
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

, : , , . , . 

CPyDef_sum PyObject, CPyTagged. int, CPython, mypyc, . , sum int .

CPyTaggetAdd PyNumber_Add. mypyc. CPyTaggetAdd, , a b, int, , :

if (likely(CPyTagged_CheckShort(left) && CPyTagged_CheckShort(right))) {
    CPyTagged sum = left + right;

    if (likely(!CPyTagged_IsAddOverflow(sum, left, right))) {
        return sum;
    }
}
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

, CPython - Extension :

 — - sum A, B;

 — , .

bubble_sort(data: List[int])

, . , data:

def bubble_sort(data: List[int]):
    …
      
      

        
        
        
      

    
        
        
        
      
      

        
        
        
      

    
    

:

, , , !

mypyc

, , . mypyc : , , , mypy. mypy . python-, , .

, , :

• ;

  
  
  
  
  

• monkey patching;

  
  
  
  
  

• Mypy , .

  
  
  
  
  

. , , abc. , . , gcc , , , . , , 20 % , .

, Roadmap , .

Nuitka

, . Nuitka . , Nuitka Python ++ , Python Extension. , CPython libpython.

Nuitka , . mypy .

, mypy : , “ ”, PyCharm . , mypy. , . , python. mypy — , . , CPython , , . (, mypyc ). , mypyc , , , - , mypyc, , , mypy.

P.S.

, python, . , mypyc, , , .

UPD

, python - Cython, python ( cython-). cython , (real 1.82) mypyc . .