One 、 Introduction to processes and threads
1. Concept
1.1 multitasking
The operation of the program is CPU The result of working with memory
Operating system, such as Mac OS X,UNIX,Linux,Windows etc. , All support. “ multitasking ” Operating system of
problem 1： What is multitasking ？
The operating system can run multiple tasks at the same time . For example , You're surfing the Internet with your browser , While listening MP3, I'm using it on one side Word Catch up on homework , This is multitasking , At least at the same time 3 A task is running . There are still many tasks running quietly in the background at the same time , It's just that there's no display on the desktop
problem 2： Multicore CPU It's very popular , however , Even the past single core CPU, You can also perform multiple tasks . because CPU The execution code is executed in sequence , that , Single core CPU How to perform multitasking ？
answer ： The operating system alternates tasks , Mission 1 perform 0.01 second , Switch to task 2, Mission 2 perform 0.01 second , Switch to task again 3, perform 0.01 second …… This is repeated . actually , Each task is performed alternately , however , On the surface , because CPU The execution speed of is too fast , We feel like all tasks are being performed at the same time 【 Today's computers at least 4 Nucleation 】
parallel ： Really execute together , The number of tasks is less than cpu The number of cores 【 ideal type 】
Concurrent ： Looks like we're doing it together , The number of tasks is greater than cpu The number of cores 【 Realistic 】
Multicore cpu The principle of multitasking ： Real parallel multitasking can only be performed in multicore cpu Implemented on , But because the number of tasks is far more than cpu The number of cores , So the operating system will automatically schedule many tasks to each core in turn
For the operating system , A task is a process 【process】, An operating system can start multiple processes
Some processes start up , It may be necessary to handle multiple subtasks , These subtasks are threads 【thread】
1.2 process
It is the running state and resource occupation of a program （ Memory ,CPU） Description of
A process is a dynamic process of a program , It refers to a complete process from code loading to execution
【 Interview questions 】 Process characteristics ：
a. independence ： Different processes are independent , Resources are not shared with each other （ give an example ： The two classrooms in class have their own property , Don't share with each other ）
b. dynamic ： Processes are not static in the system , It's all the time in the system （ give an example ： The classroom has been giving lectures ）
c. concurrency ： Multiple processes can run simultaneously on a single processor , And they don't affect each other （ give an example ： If tianfengli is a processor , Having classes in multiple classrooms at the same time is equivalent to multiple processes ,c Do not affect each other ）
Multi process ： An operating system can run multiple applications
1.3 Threads
Threads are part of a process , A process can have multiple threads , Each thread handles a specific subtask
Thread execution is preemptive , Multiple threads can execute concurrently in the same process , In fact, that is CPU Quickly switch between different threads , in other words , The currently running thread is likely to be suspended at any time , So that another thread can run
for example ： Turn on cool dog music ————》 It's a process
Play songs and refresh lyrics ————》 These are two threads （ Two threads execute alternately at the same time , So it is concurrent ）
Different threads in a process share resources
Multithreading ：
In a process , Multiple threads can be performed simultaneously
Multithreading is an effective means to realize concurrency mechanism , A process can contain multiple threads , Different threads can share resources , Running at the same time can improve the execution efficiency of the program , Can complete multiple tasks at the same time
application ： A browser can download multiple pictures and videos at the same time
A server can respond to multiple user requests at the same time
【 Interview questions 】 Briefly describe the differences and relationships between processes and threads
a. After a program starts , It will certainly start a process
b. After a process starts , Multiple threads may be started , however , The process requires at least one thread , Otherwise, the process is meaningless
c. Resources are not shared among multiple processes , Resource sharing among multiple threads
d. The system creation process needs to reallocate system resources for this process , Creating threads is much easier , So in the actual development , Use more threads than multiple processes
Two 、 process
1. Single task phenomenon
# Mission 2
def func():
print("11111111")
# Mission 1
if __name__ == '__main__':
while True:
print("main~~~")
func()
2. Multi process realizes multi task
2.1 Start the process to realize multitasking
Unix/Linux The operating system provides a fork() system call ,fork() Exist in os Under module 【os.fork()】, however ,Windows No, fork call
because Python It's cross platform , Provides a cross platform multi process support .multiprocessing Module is a multi process module of cross platform version
multiprocessing The module provides one Process Class to represent a process object
# 1. The import module
from multiprocessing import Process
import os,time
# Tasks of subprocesses
def func():
# os.getppid() Indicates to get the process number of the parent process of the current process
print(f" Subprocess start , Process number ：%s, The corresponding parent process is ：%s" % (os.getpid(),os.getppid()))
while True:
print("11111111")
time.sleep(0.5)
def func1(a,b):
print(a + b)
print(f" Subprocess start , Process number ：%s, The corresponding parent process is ：%s" % (os.getpid(),os.getppid()))
while True:
print("11111111")
time.sleep(0.5)
if __name__ == '__main__':
# 2. stay __main__ The code executed in represents the main process / The parent process
# Be careful ： A program starts , It means that a parent process starts , The system will automatically assign a process number to the process , adopt os.getpid() Get process number
print(f" The parent process starts , Process number ：%s" % (os.getpid()))
# 3. Create a child process in the parent process
"""
Process(target,args)
target: The function name of the task to be executed by the current process
args: Represents the parameters of the task function , The value must be a tuple
"""
# Be careful ： After the process object is created , Be sure to start it manually
# a. The task function has no parameters
# p = Process(target=func)
# p.start()
# b. The task function has parameters
p = Process(target=func1,args=(23,10))
p.start()
# Tasks of the main process
while True:
print("main~~~")
time.sleep(0.5)
2.2 The sequence of parent-child processes
from multiprocessing import Process
import os,time
def run():
print(f" Subprocess start ：%s" % (os.getpid()))
print('112431513531')
time.sleep(5)
print(f" End of subprocess ~~~~：%s" % (os.getpid()))
if __name__ == '__main__':
print(f" The parent process starts ：%s" % (os.getpid()))
p = Process(target=run)
p.start()
# The merger process 【 Queue the specified child process , Will give priority to the consolidation process 】
"""
By default , According to the scheduling of the system , The main process may end before the child process
But when you merge child processes 【join】 after , The main process will wait for the execution of the child process before ending
"""
p.join()
print('main~~~~~~')
time.sleep(0.5)
print(f" End of parent process ~~~~~：%s" % (os.getpid()))
"""
join Before
The parent process starts ：63157
main~~~~~~
Subprocess start ：63158
112431513531
End of parent process ~~~~~：63157
End of subprocess ~~~~：63158
join after ：
The parent process starts ：63138
Subprocess start ：63139
112431513531
End of subprocess ~~~~：63139
main~~~~~~
End of parent process ~~~~~：63138
"""
2.3 Global variables in multiple processes
from multiprocessing import Process
import os,time
# demand ： Modify global variables in child processes , View the value of the global variable accessed in the parent process
num = 100
def run():
print(f" Subprocess start ~~~~")
global num
num += 50
print(f" End of subprocess ~~~~：{num}") # 150
if __name__ == '__main__':
print(f" The parent process starts ：%s" % (os.getpid()))
p = Process(target=run)
p.start()
# Give priority to sub processes
p.join()
print(f" End of parent process ~~~~~：%s" % (os.getpid()))
print(f" Global variables num={num}") # 100
"""
summary ：
It is verified that the processes are independent , Resources are not shared with each other
working principle ： When creating child process objects , All global data will be backed up ,
Each process has its own code snippet , Data segments and stack segments , So the processes are independent , Resources are not shared
"""
2.4 Start a large number of child processes
If you want to start a large number of subprocesses , You can create subprocesses in batch in the way of process pool
from multiprocessing import Process,Pool
import os,time
def func(n):
print(f" Subprocesses {n} start-up ")
print("111111")
time.sleep(2)
print(f" Subprocesses {n} end ~~~~~")
if __name__ == '__main__':
print(f" The parent process starts ")
# Create multiple child processes
# Mode one ：Process
# p1 = Process(target=func)
# p1.start()
# p2 = Process(target=func)
# p2.start()
# p3 = Process(target=func)
# p3.start()
# Mode two ：Pool
# Create the object of the process pool
# Pool(num):num Indicates the number of processes that can be executed simultaneously ,num Omit , The default value is... Of the current operating system cpu The core number
pool = Pool(2)
# Create multiple process objects , Unified management through process pool
for i in range(10):
pool.apply_async(func,args=(i,))
# When the child process is added , Need to close the process pool , call close After that, it means that new process objects cannot be added to the process pool
pool.close()
# When using the process pool , Recommend calling join, The main process will be blocked here ,
# When all the child processes in the process pool are completed , Will continue to execute the tasks in the main process
pool.join()
print(f" End of parent process ~~~~")
2.5 Communication between processes
Process Communication is required between , The operating system provides many mechanisms for interprocess communication ,Python Of multiprocessing Modules wrap the underlying mechanism , Provides Queue、Pipes And so on
demand ; Create two child processes in the parent process , A to Queue Write the data , A from Queue Read in the data
# Communication between processes is required , No direct communication , So you can communicate through a queue
from multiprocessing import Process,Queue
from time import sleep
# demand ： Create two child processes in the parent process , A process writes data to the queue , Another process gets data from the queue
def write(queue):
print(" The process of writing data starts ")
for ch in "ABC":
print(f"put {ch} to queue")
# Add data to the queue
queue.put(ch)
sleep(1)
print(" The process of writing data is over ~~~~")
def read(queue):
print(" The process of reading data starts ")
for _ in range(3):
# Get data from the queue
value = queue.get(True)
print(f"get {value} from queue")
print(" The process of reading data ends ~~~~")
if __name__ == '__main__':
print(" The parent process starts ")
# Create a queue object
queue = Queue()
# Create an object for the process
write_process = Process(target=write,args=(queue,))
read_process = Process(target=read,args=(queue,))
write_process.start()
read_process.start()
write_process.join()
read_process.join()
print(" End of parent process ~~~")
3、 ... and 、 Threads
Multitasking can be done by multiple processes , It can also be done by multiple threads in a process
Inside a process , Do many things at the same time , You need to run multiple “ The subtasks ”, We put these in the process “ The subtasks ” It's called thread
Threads are often called lightweight processes . Threads are multitasks that are executed concurrently in shared memory space , Each thread shares resources in a process
Threads are the smallest unit of execution , And the process is made up of at least one thread . How to schedule processes and threads , It's entirely up to the operating system , The program itself can't decide when to execute , Nor can it be decided how long it will take
Moreover, thread is the execution unit directly supported by the operating system , therefore , High level languages usually have built-in multithreading support ,Python No exception , also ,Python The thread of is real Posix Thread, Instead of a simulated thread
modular
1、_thread modular ： Provides a low level of 、 Original thread 【 Low level is not bad , But the function is limited , The bottom layer is c Encapsulation of language 】
2、threading modular ： Advanced modules , Yes _thre
ad It was packaged , And provides _thread Features not available in
In most cases , We just need to use threading This advanced module
1. Create thread
To create and start a thread is to pass in a function and create Thread example , And then call start() Start execution
"""
Multiple tasks can be performed through processes , Multitasking can also be performed through threads
Create a process ：p = Process(target=xxx,args=(xx,xx..)) p.start() Process number
Create thread ：t = Thread(target=xxx,args=(xx,xx..)) t.start() Thread name
"""
# 1. The import module
# import multiprocessing # Process module
import threading,os,time # Thread module
def run():
print(f" The child thread starts ：{threading.current_thread().name}")
time.sleep(2)
print(f" Child thread end ~~~~：{threading.current_thread().name}")
def run1(num1,num2):
print(num1,num2)
print(f" The child thread starts ：{threading.current_thread().name}")
time.sleep(2)
print(f" Child thread end ~~~~：{threading.current_thread().name}")
if __name__ == '__main__':
# 2. After any program starts , Will start a process ,
# This process is called the master process , The process will start a thread by default , It is called the main thread
# threading.current_thread().name Get the name of the thread currently executing , The name of the main thread is MainThread
print(f" The main process {os.getpid()} The main thread in ：{threading.current_thread().name}")
# 3. Create child threads
"""
Thread(target,args,name)
target: The function name of the task to be executed by the current thread
args: Represents the parameters of the task function , The value must be a tuple
name: Name the thread
"""
# Be careful ： The default name of the child thread is Thread-1,Thread-2....
# a. The task function has no parameters
# t = threading.Thread(target=run,name='hello')
# t.start()
# b. The task function has parameters
t = threading.Thread(target=run1, name='hello',args=(34,6))
t.start()
# Similar to process usage , You can also merge threads , Then the thread executes first
t.join()
print(f" The main process {os.getpid()} The main thread in ends ~~~~：{threading.current_thread().name}")
2. Global variables in threads
import threading
# Suppose the bank card balance
balance = 0
def change(n):
global balance
balance += n
balance -= n # 0
def run(m):
for _ in range(1000000):
change(m)
if __name__ == '__main__':
t1 = threading.Thread(target=run,args=(5,))
t2 = threading.Thread(target=run, args=(8,))
t1.start()
t2.start()
t1.join()
t2.join()
print(f"balance={balance}") # Theoretically, the result is 0
"""
Multiple processes are independent , Resources are not shared , So when a process modifies a global variable , The results accessed by another process have no effect
Multiple threads share resources , So when multiple threads access a global variable at the same time , This will cause errors in the results of global variables
reason ：
balance += n-----》 First step ：x = balance + n The second step ：balance = x
analysis ：
t1:x1 = balance + n x1 = 5
t2:x2 = balance + n x2 = 8
t2:balance = x2 balance = 8
t1:balance = x1 balance = 5
t1:x1 = balance -n x1 = 0
t1:balance = x1 balance = 0
t2:x2 = balance - n x2 = -8
t2:balance = x2 balance = -8
solve ：
Just make sure that a thread grabs the time slice , Be able to execute all the code , Other threads are waiting
Realization ：
Lock critical resources , Which thread grabbed the time slice , Then the thread holds the lock , When the code of critical resources is executed , Then release the lock
"""
3. Thread lock
import threading
# Create a lock object
lock = threading.Lock()
# Suppose the bank card balance
balance = 0
def change(n):
global balance
balance += n
balance -= n # 0
def run(m):
for _ in range(1000000):
# Get the lock
lock.acquire()
try:
change(m)
finally:
# Release the lock
lock.release()
if __name__ == '__main__':
t1 = threading.Thread(target=run,args=(5,))
t2 = threading.Thread(target=run, args=(8,))
t1.start()
t2.start()
t1.join()
t2.join()
print(f"balance={balance}") # Theoretically, the result is 0
import threading
# Create a lock object
lock = threading.Lock()
# Suppose the bank card balance
balance = 0
def change(n):
global balance
balance += n
balance -= n # 0
def run(m):
for _ in range(1000000):
# Get into with Code block , Will get the lock , When with Code block execution complete , Will automatically release the lock
with lock:
change(m)
if __name__ == '__main__':
t1 = threading.Thread(target=run,args=(5,))
t2 = threading.Thread(target=run, args=(8,))
t1.start()
t2.start()
t1.join()
t2.join()
print(f"balance={balance}") # Theoretically, the result is 0
4. Timing of the thread
import threading
# 【 Interview questions 】
def run():
print("hello~~~~~")
if __name__ == '__main__':
print(" The main thread starts ")
# Timer(seds,target)
t = threading.Timer(5,run)
t.start()
t.join()
print(" End of main thread ")
5. Communication between threads
import threading,time
def func():
# Create an event object
event = threading.Event()
def run():
for i in range(5):
# wait(): wait for clear(): Reset notify()： Wake up the notifyAll(): Wake up all
# wait for / Blocking , Wait for the event to trigger
print("hello~~~~~~start")
event.wait()
print("hello~~~~~~end")
threading.Thread(target=run).start()
return event
e = func()
# Triggering event
for i in range(5):
time.sleep(1)
e.set()
6. Producer consumer design pattern
"""
The producer consumer design pattern solves the strong coupling problem through a container ,, There is no direct communication between producers and consumers , Instead, they communicate through queues
Therefore, the producer does not need to wait for the consumer to process the data after production , Throw the data directly to the queue , Consumers do not need to find producers , Instead, get directly from the queue
The queue is equivalent to a buffer , Balancing the processing capacity of producers and consumers
"""
import threading,time,queue,random
# producer : Add data to the queue
def product(id,queue):
while True:
num = random.randint(1,100)
queue.put(num)
print(f" producer {id} Added... To the queue {num} data ")
time.sleep(1)
# Task to complete
queue.task_done()
# consumer ： Get data from the queue
def customer(id,queue):
while True:
num = queue.get()
if num is None:
break
print(f" consumer {id} Got... From the queue {num} data ")
time.sleep(0.5)
queue.task_done()
if __name__ == '__main__':
# Create a queue
queue = queue.Queue()
# Create and start the producer's thread
for i in range(2):
threading.Thread(target=product,args=(i,queue)).start()
# Create and start the consumer's thread
for j in range(5):
threading.Thread(target=customer, args=(j, queue)).start()
