This is my participation 2022 For the first time, the third challenge is 8 God , Check out the activity details ：2022 For the first time, it's a challenge
Python What has been criticized by everyone is the slow execution speed , But there's no denying it Python It is still a sharp weapon in our study and work . therefore , We are right. Python What is it? “ Love and hate ”.
This article summarizes some small tips Help to improve Python Execution speed 、 Optimize performance . All of the following techniques have been verified by me , It's safe to eat .
Come to the conclusion first ：
map() Function mapping set() Find the intersection sort() or sorted() Sort collections.Counter() Count join() Connection string x, y = y, x Exchange variables while 1 replace while True.) Use for Circulation replaces while loop Numba.jit Accelerate Computing Numpy Vectorized array in Check list members itertools Library iteration About Python How to accurately measure the execution time of the program , The problem seems simple, but it's actually very complicated , Because the execution time of the program is affected by many factors , For example, operating system 、Python Version and related hardware （CPU performance 、 Memory read and write speed ） etc. . When running the same version of the language on the same computer , The above factors are certain , But the sleep time of the program is still changing , And other programs running on the computer will also interfere with the experiment , So strictly speaking, this is 《 The experiment cannot be repeated 》.
The two representative libraries I learned about timing are time and timeit.
among ,time In the library time()、perf_counter() as well as process_time() Three functions can be used to time （ In seconds ）, suffix _ns Time in nanoseconds （ since Python3.7 beginning ）. Before that clock() function , But in Python3.3 Then it was removed . The differences between the above three are as follows ：
time() The accuracy is relatively not that high , And affected by the system , Suitable for representing date and time or timing of large programs .perf_counter() Suitable for smaller program testing , Calculation sleep() Time .process_time() Suitable for smaller program testing , Don't count sleep() Time . And time Coop ,timeit There are two advantages ：
timeit According to your operating system and Python Select the best timer for the version .timeit Garbage collection is temporarily disabled during timing .timeit.timeit(stmt='pass', setup='pass', timer=<default timer>, number=1000000, globals=None) Parameter description ：
stmt='pass'： Statements or functions that need timing .setup='pass'： perform stmt Code to run before . Usually , It is used to import some modules or declare some necessary variables .timer=<default timer>： Timer function , The default is time.perf_counter().number=1000000： The number of times the timed statement was executed , Default is one million times .globals=None： Specifies the namespace in which the code is executed . All timing in this paper adopts timeit Method , And the default execution times are one million times .
Why a million times ？ Because our test program is very short , If you don't do it so many times , There is no difference at all .
map() Function mapping Exp1： Convert lowercase letters to uppercase letters in the string array .
The test array is oldlist = ['life', 'is', 'short', 'i', 'choose', 'python'].
newlist = []
for word in oldlist:
newlist.append(word.upper())
Copy code list(map(str.upper, oldlist))
Copy code Method one takes time 0.5267724000000005s, Method two takes time 0.41462569999999843s, Performance improvement 21.29%
set() Find the intersection Exp2： Two, please list Intersection .
Test array ：a = [1,2,3,4,5],b = [2,4,6,8,10].
overlaps = []
for x in a:
for y in b:
if x == y:
overlaps.append(x)
Copy code list(set(a) & set(b))
Copy code Method one takes time 0.9507264000000006s, Method two takes time 0.6148200999999993s, Performance improvement 35.33%
About set() The grammar of ：|、&、- Respectively represent the union set 、 intersection 、 Difference set .
sort() or sorted() Sort We can sort the sequence in many ways , But in fact, the method of writing sorting algorithm by yourself is not worth the loss . Because of the built-in sort() or sorted() The method is good enough , And use parameters key Different functions can be realized , Very flexible . The difference between them is sort() Methods are defined only in list in , and sorted() The global method is effective for all iterative sequences .
Exp3： Use fast platoon and..., respectively sort() Method to sort the same list .
Test array ：lists = [2,1,4,3,0].
def quick_sort(lists,i,j):
if i >= j:
return list
pivot = lists[i]
low = i
high = j
while i < j:
while i < j and lists[j] >= pivot:
j -= 1
lists[i]=lists[j]
while i < j and lists[i] <=pivot:
i += 1
lists[j]=lists[i]
lists[j] = pivot
quick_sort(lists,low,i-1)
quick_sort(lists,i+1,high)
return lists
Copy code lists.sort()
Copy code Method one takes time 2.4796975000000003s, Method two takes time 0.05551999999999424s, Performance improvement 97.76%
By the way ,sorted() The method is time consuming 0.1339823999987857s.
It can be seen that ,sort() As list The exclusive sorting method is still very strong ,sorted() Although a little slower than the former , But the victory is in it “ No picky eaters ”, It works for all iteratable sequences .
Expand ： How to define sort() or sorted() Methodical key
lambda Definition # Student ：（ full name , achievement , Age ）
students = [('john', 'A', 15),('jane', 'B', 12),('dave', 'B', 10)]
students.sort(key = lambda student: student[0]) # Sort by name
sorted(students, key = lambda student: student[0])
Copy code operator Definition import operator
students = [('john', 'A', 15),('jane', 'B', 12),('dave', 'B', 10)]
students.sort(key=operator.itemgetter(0))
sorted(students, key = operator.itemgetter(1, 0)) # Sort your grades first , Then sort the names
Copy code
operatorOfitemgetter()Suitable for ordinary array sorting ,attrgetter()Applicable to object array sorting
cmp_to_key() Definition , Most flexible import functools
def cmp(a,b):
if a[1] != b[1]:
return -1 if a[1] < b[1] else 1 # First sort them in ascending order
elif a[0] != b[0]:
return -1 if a[0] < b[0] else 1 # Same grades , Sort by name in ascending order
else:
return -1 if a[2] > b[2] else 1 # The grades and names are the same , Sort by age
students = [('john', 'A', 15),('john', 'A', 14),('jane', 'B', 12),('dave', 'B', 10)]
sorted(students, key = functools.cmp_to_key(cmp))
Copy code collections.Counter() Count Exp4： Count the number of occurrences of each character in the string .
Test array ：sentence='life is short, i choose python'.
counts = {}
for char in sentence:
counts[char] = counts.get(char, 0) + 1
Copy code from collections import Counter
Counter(sentence)
Copy code Method one takes time 2.8105250000000055s, Method two takes time 1.6317423000000062s, Performance improvement 41.94%
The list of deduction （list comprehension） short . In small code snippets , It may not make much difference . But in large-scale development , It can save some time .
Exp5： Square the odd numbers in the list , Even numbers don't change .
Test array ：oldlist = range(10).
newlist = []
for x in oldlist:
if x % 2 == 1:
newlist.append(x**2)
Copy code [x**2 for x in oldlist if x%2 == 1]
Copy code Method one takes time 1.5342976000000021s, Method two takes time 1.4181957999999923s, Performance improvement 7.57%
join() Connection string Most people are used to using + To connect strings . But in fact , This method is very inefficient . because ,+ In each step of the operation, a new string is created and the old string is copied . A better way is to use join() To connect strings . Other operations on strings , Also try to use built-in functions , Such as isalpha()、isdigit()、startswith()、endswith() etc. .
Exp6： Connect the elements in the string list .
Test array ：oldlist = ['life', 'is', 'short', 'i', 'choose', 'python'].
sentence = ""
for word in oldlist:
sentence += word
Copy code "".join(oldlist)
Copy code Method one takes time 0.27489080000000854s, Method two takes time 0.08166570000000206s, Performance improvement 70.29%
join There is also a very comfortable point , That is, it can specify the separator of the connection , for instance
oldlist = ['life', 'is', 'short', 'i', 'choose', 'python']
sentence = "//".join(oldlist)
print(sentence)
Copy code life//is//short//i//choose//python
x, y = y, x Exchange variables Exp6： In exchange for x,y Value .
Test data ：x, y = 100, 200.
temp = x
x = y
y = temp
Copy code x, y = y, x
Copy code Method one takes time 0.027853900000010867s, Method two takes time 0.02398730000000171s, Performance improvement 13.88%
while 1 replace while True When you don't know the exact number of cycles , The conventional method is to use while True Make an infinite loop , Determine whether the loop termination condition is met in the code block . Although there is no problem doing so , but while 1 The execution speed ratio of while True faster . Because it is a numerical conversion , Output can be generated faster .
Exp8： Use them separately while 1 and while True loop 100 Time .
i = 0
while True:
i += 1
if i > 100:
break
Copy code i = 0
while 1:
i += 1
if i > 100:
break
Copy code Method one takes time 3.679268300000004s, Method two takes time 3.607847499999991s, Performance improvement 1.94%
Storing files in a cache helps to quickly restore functionality .Python Support decorator cache , The cache maintains a specific type of cache in memory , To achieve the best software driven speed . We use lru_cache Decorator to provide caching for Fibonacci functions , In the use of fibonacci When recursive functions , There's a lot of double counting , for example fibonacci(1)、fibonacci(2) It ran many times . And in use lru_cache after , All double counting will be performed only once , So as to greatly improve the execution efficiency of the program .
Exp9： Find the Fibonacci sequence .
Test data ：fibonacci(7).
def fibonacci(n):
if n == 0:
return 0
elif n == 1:
return 1
return fibonacci(n - 1) + fibonacci(n-2)
Copy code import functools
@functools.lru_cache(maxsize=128)
def fibonacci(n):
if n == 0:
return 0
elif n == 1:
return 1
return fibonacci(n - 1) + fibonacci(n-2)
Copy code Method one takes time 3.955014900000009s, Method two takes time 0.05077979999998661s, Performance improvement 98.72%
matters needing attention ：
lru_cache The decorated function will only execute once .list Cannot be used as lru_cache The parameters of the decorated function .import functools
@functools.lru_cache(maxsize=100)
def demo(a, b):
print(' I was executed ')
return a + b
if __name__ == '__main__':
demo(1, 2)
demo(1, 2)
Copy code I was executed （ Twice
demo(1, 2), But only output once ）
from functools import lru_cache
@lru_cache(maxsize=100)
def list_sum(nums: list):
return sum(nums)
if __name__ == '__main__':
list_sum([1, 2, 3, 4, 5])
Copy code TypeError: unhashable type: 'list'
functools.lru_cache(maxsize=128, typed=False) Two optional parameters of ：
maxsize Represents the memory usage value of the cache , After exceeding this value , The result will be released , Then cache the new calculation results , Its value should be set to 2 The power of .typed if True, The results of different parameter types will be saved separately ..) Use Dot operator (.) The property or method used to access the object , This will cause the program to use __getattribute__() and __getattr__() Look up the dictionary , This brings unnecessary expenses . Pay special attention to , In the cycle , It is also necessary to reduce the use of point operators , It should be moved out of the loop .
This suggests that we should try to use from ... import ... This way to guide the package , Instead of using the dot operator to get... When a method needs to be used . It's not just dot operators , We try to move many other unnecessary operations out of the loop .
Exp10： Convert lowercase letters to uppercase letters in the string array .
The test array is oldlist = ['life', 'is', 'short', 'i', 'choose', 'python'].
newlist = []
for word in oldlist:
newlist.append(str.upper(word))
Copy code newlist = []
upper = str.upper
for word in oldlist:
newlist.append(upper(word))
Copy code Method one takes time 0.7235491999999795s, Method two takes time 0.5475435999999831s, Performance improvement 24.33%
for Circulation replaces while loop When we know exactly how many times to cycle , Use for Cycle ratio use while Better cycle .
Exp12： Use for and while Separate cycles 100 Time .
i = 0
while i < 100:
i += 1
Copy code for _ in range(100):
pass
Copy code Method one takes time 3.894683299999997s, Method two takes time 1.0198077999999953s, Performance improvement 73.82%
Numba.jit Accelerate Computing Numba Can be Python Function encoding and decoding is executed for machine code , Greatly improve the speed of code execution , Even close C or FORTRAN The speed of . It can match Numpy In combination with , stay for It can significantly improve the execution efficiency when there are a lot of calculations in the loop .
Exp12： Seek from 1 Add to 100 And .
def my_sum(n):
x = 0
for i in range(1, n+1):
x += i
return x
Copy code from numba import jit
@jit(nopython=True)
def numba_sum(n):
x = 0
for i in range(1, n+1):
x += i
return x
Copy code Method one takes time 3.7199997000000167s, Method two takes time 0.23769430000001535s, Performance improvement 93.61%
Numpy Vectorized array Vectorization is NumPy A powerful feature of , The operation can be expressed as taking place on the entire array rather than on individual elements . This method of replacing explicit loops with array expressions is often called vectorization .
stay Python When looping through an array or any data structure , It will involve a lot of expenses .NumPy Vectorization in delegates internal loops to highly optimized C and Fortran function , So that Python Code is faster .
Exp13： Two sequences of the same length are multiplied element by element .
Test array ：a = [1,2,3,4,5], b = [2,4,6,8,10]
[a[i]*b[i] for i in range(len(a))]
Copy code import numpy as np
a = np.array([1,2,3,4,5])
b = np.array([2,4,6,8,10])
a*b
Copy code Method one takes time 0.6706845000000214s, Method two takes time 0.3070132000000001s, Performance improvement 54.22%
in Check list members To check whether a member is included in the list , Usually use in Keywords faster .
Exp14： Check whether the list contains a member .
Test array ：lists = ['life', 'is', 'short', 'i', 'choose', 'python']
def check_member(target, lists):
for member in lists:
if member == target:
return True
return False
Copy code if target in lists:
pass
Copy code Method one takes time 0.16038449999999216s, Method two takes time 0.04139250000000061s, Performance improvement 74.19%
itertools Library iteration itertools Is a module used to operate iterators , Its functions can be divided into three categories ： Infinite iterator 、 Finite iterator 、 Combined iterators .
Exp15： Returns the full arrangement of the list .
Test array ：["Alice", "Bob", "Carol"]
def permutations(lst):
if len(lst) == 1 or len(lst) == 0:
return [lst]
result = []
for i in lst:
temp_lst = lst[:]
temp_lst.remove(i)
temp = permutations(temp_lst)
for j in temp:
j.insert(0, i)
result.append(j)
return result
Copy code import itertools
itertools.permutations(["Alice", "Bob", "Carol"])
Copy code Method one takes time 3.867292899999484s, Method two takes time 0.3875405000007959s, Performance improvement 89.98%
Expand ：itertools Detailed explanation of the library ： Click on this link
According to the test data above , I drew the following picture of the experimental results , You can more intuitively see the performance differences brought by different methods .
As you can see from the diagram , The performance increase brought by most techniques is still considerable , But there are also a few skills that have a small increase （ For example, number 5、7、8, among , The first 8 There is little difference between the two methods ）.
Sum up , I think it's actually the following two principles ：
The built-in library functions are written by professional developers and have been tested for many times , The bottom layer of many library functions is C Language development . therefore , These functions are generally very efficient （ such as sort()、join() etc. ）, It's hard to write your own methods beyond them , You might as well save your time , Don't make wheels again , Besides, the wheels you make may be worse . therefore , If the function already exists in the function library , Just use it .
There are many excellent third-party libraries , Their bottom layer may be C and Fortran To achieve , A library like this will never suffer a loss when used , For example, as mentioned above Numpy and Numba, The improvements they bring are amazing . There are many libraries like this , such as Cython、PyPy etc. , I'm just throwing a brick at you .
In fact, speed up Python There are many ways to speed up code execution , such as Avoid using global variables 、 Use the latest version 、 Use the right data structure 、 utilize if The inertia of conditions wait , I won't cite one by one here . These methods need us to practice in person before we can have a deep feeling and understanding , But the most fundamental way is to keep our passion for programming and the pursuit of best practices , This is how we can constantly break through ourselves 、 The inexhaustible power source of climbing the peak ！
Thank you for seeing this , If you think this is going to help you ：
Like it and support it , So that more people can see the content .
Welcome to share your thoughts with me in the message area , You are also welcome to record your thinking process in the message area .
If you are interested, you can also read some of my previous articles ：
