The problem that multithreaded Django programs run out of database connections

Django Of ORM It's very easy to use , Even if it's not doing Web The project is also worth using , So you can also find many uses on the Internet Django Develop non Web Project information , Because except for ORM Of , Command line 、 Components such as configuration files are also very easy to use .

Recently, a non Web project , And it is multithreaded . Yes N Worker threads from DB In order to get jobs, And write back the results DB. In short, that's it .

After the project runs for a period of time , Found that the database connection is exhausted , Fortunately, the memory is large , Then keep going up , Finally, the number of connections is more than 9000, 10000 . When you run out of connections ,PostgreSQL There will be errors like this ：

FATAL: remaining connection slots are reserved for non-replication superuser connections

Then look at all kinds of documents 、 Code , To find problems , The difficulties are slightly different , Finally, there are some knowledge points ：

1. Django The database connection in is on the thread local() In the instance .
2. anytime , If you need a database connection ,Django Will create one , Or use the one that this thread already has .
3. If it is Web project , At the end of the request ,Django Will close the connection . Yes , No connection pool .
4. Because we are right and wrong Web project , So there is no request end event , So never close the connection . But this should not have caused problems , Because it has been used since it was not closed , But I don't know what went wrong , There will be connection leakage , So the connection data will continue to grow .

The final solution is to find an opportunity to actively close the database connection , Specific to our project , Every time the worker thread finishes a task , Just turn off the relevant connection , Because we use ThreadPoolExecutor, therefore Django It's easy to do this .

The key codes are as follows ：

from django.db import connections
def on_done(future):
# Because every thread has one connections, So you can call close_all(), Close all connections under the name of this thread .
connections.close_all()
def main():
# ...
with ThreadPoolExecutor() as executor:
while True:
future = executor.submit(do, get_a_job())
future.add_done_callback(on_done)

After active shutdown , The number of database connections fell to close to the number of worker threads , And stay stable .