python Multi process programming is very convenient to manage processes by using process pool , But sometimes child processes will preempt some exclusive resources , such as consol Or, for example, the write permission of the log file , In this case, we usually need to share one Lock To lock exclusive resources .
lock As a resource that cannot be directly packaged, there is no way to directly give it as a parameter Pool Of map The function in the method passes parameters .
To solve this problem , There are two solutions , One is to use a multi process manager Manager(), And use the method of partial function to transfer objects Manager.Lock(). The second method is to pass... When the process pool is created multiprocessing.Lock() object .
Example ：
For example, I now have a data list, and I want to send the data in it to the specified through multiple processes API And record the time taken for each request in the log file .
The easiest solution we can think of is to pass the lock as a parameter ：
from multiprocessing import Pool, Lock
import urllib2
from time import clock
from functools import partial
def send_request(lock, data):
api_url = 'http://api.xxxx.com/?data=%s'
start_time = clock()
print urllib2.urlopen(api_url % data).read()
end_time = clock()
lock.acquire()
whit open('request.log', 'a+') as logs:
logs.write('request %s cost: %s\n' % (data, end_time - start_time))
lock.release()
if __name__ == '__main__':
data_list = ['data1', 'data2', 'data3']
pool = Pool(8)
lock = Lock()
partial_send_request(send_request, lock=lock)
pool.map(partial_send_request, data_list)
pool.close()
pool.join()
In this case ,lock As a resource that cannot be directly packaged, there is no way to directly give it as a parameter Pool Of map The function in the method passes parameters .
A runtime error will occur ：Runtime Error: Lock objects should only be shared between processes through inheritance.
According to the initial idea, we can change the code to the following ：
The first way of thinking , Use Manager.
send_request The function doesn't have to be changed , Only change main The content in ：
''' No one answers the problems encountered in learning ？ Xiaobian created a Python Exchange of learning QQ Group ：153708845 Looking for small partners who share the same aspiration , Help each other , There are also good video tutorials and PDF e-book ！ '''
if __name__ == '__main__':
from multiprocessing import Manager
data_list = ['data1', 'data2', 'data3']
pool = Pool(8)
manager = Manager()
lock = manager.Lock()
partial_send_request(send_request, lock=lock)
pool.map(partial_send_request, data_list)
pool.close()
pool.join()
That's the first way , But for a log write lock that only needs one Manager It's very heavy . This method actually requires a special process to handle Manager service . All lock adding and lock releasing operations are performed through IPC Pass to Manager Service .
The second solution is through initializer Parameter in Pool Objects are passed when they are created Lock object . This way will Lock Object becomes a global object for all child processes .
The code can be modified as follows ：
def send_request(data):
api_url = 'http://api.xxxx.com/?data=%s'
start_time = clock()
print urllib2.urlopen(api_url % data).read()
end_time = clock()
lock.acquire()
whit open('request.log', 'a+') as logs:
logs.write('request %s cost: %s\n' % (data, end_time - start_time))
lock.release()
def init(l):
global lock
lock = l
if __name__ == '__main__':
data_list = ['data1', 'data2', 'data3']
lock = Lock()
pool = Pool(8, initializer=init, initargs=(lock,))
pool.map(send_request, data_list)
pool.close()
pool.join()
This modification eliminates the need to use partial functions .
