1. 問題:日志中出現下面的警告:
警告: The web application [ROOT] appears to have started a thread named [HouseKeeper] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Thread.sleep(Native Method)
org.logicalcobwebs.proxool.HouseKeeperThread.run(HouseKeeperThread.java:46)
2. 原因調查:
使用的是 proxool 連接池,查看源碼,分析如下:
在項目某種情況下,ProxoolFacade.shutdown(); 被調用,他會調用:
protected static void shutdown(String finalizer, int delay) { ConnectionPool[] cps = ConnectionPoolManager.getInstance().getConnectionPools(); for (int i = 0; i < cps.length; i++) { removeConnectionPool(finalizer, cps[i], delay); } // If a shutdown hook was registered then remove it try { if (shutdownHook != null) { ShutdownHook.remove(shutdownHook); } } catch (Throwable t) { if (LOG.isDebugEnabled()) { LOG.debug("Unanticipated error during removal of ShutdownHook. Ignoring it.", t); } } // Stop threads PrototyperController.shutdown(); HouseKeeperController.shutdown(); }
這裡 首先 關閉連接池,然後在最後面關閉了一個 housekeeper 的 demaon線程:
HouseKeeperController.shutdown();
/** * Stop all house keeper threads. */ protected static void shutdown() { synchronized(LOCK) { Iterator i = houseKeeperThreads.iterator(); while (i.hasNext()) { HouseKeeperThread hkt = (HouseKeeperThread) i.next(); LOG.info("Stopping " + hkt.getName() + " thread"); hkt.cancel(); } houseKeeperThreads.clear(); } }
對每一個 housekeeper 線程調用 cancel() 方法,一般只有一個 housekeeper線程:
hkt.cancel();
public class HouseKeeperThread extends Thread { private static final Log LOG = LogFactory.getLog(HouseKeeperThread.class); private boolean stop; public HouseKeeperThread(String name) { setDaemon(true); setName(name); } public void run() { while (!stop) { HouseKeeper hk = HouseKeeperController.getHouseKeeperToRun(); while (hk != null && !stop) { try { // if (LOG.isDebugEnabled()) { // LOG.debug("About to sweep " + hk.getAlias()); // } hk.sweep(); } catch (ProxoolException e) { LOG.error("Couldn't sweep " + hk.getAlias(), e); } hk = HouseKeeperController.getHouseKeeperToRun(); } try { Thread.sleep(5000); } catch (InterruptedException e) { LOG.error("Interrupted", e); } } } protected void cancel() { stop = true; } }
尼瑪 stop 盡然是一個 普通 的 boolean 變量,感覺應該使用 volatile 修飾!
保錯的原因是,housekeeper 線程因為每隔5秒執行一次,而系統 想要 stop 它的時候,它正在 sleep ,所以拋出了異常:
警告: The web application [ROOT] appears to have started a thread named [HouseKeeper] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: java.lang.Thread.sleep(Native Method) org.logicalcobwebs.proxool.HouseKeeperThread.run(HouseKeeperThread.java:46)
但是,因為 設置了 stop = true,所以,當它 sleep 完成之後,也可能會 判斷 stop == true, 也可能判斷 stop == false,因為這裡沒有使用 volatile 修飾!
所以最壞的結果,就是 有一個名叫 "HouseKeeper" 的 deamon jvm 線程沒有被成功關閉。
3. 日志顯示,數據庫連接池,確實被正確關閉了:
ContainerBackgroundProcessor[StandardEngine[Catalina]] - Proxool shoutdown [INFO ] org.logicalcobwebs.proxool.ConnectionPool.shutdown(ConnectionPool.java:484):ContainerBackgroundProcessor[StandardEngine[Catalina]] - Shutting down 'autumn' pool immediately [ContainerBackgroundProcessor[StandardEngine[Catalina]]] [DEBUG] org.logicalcobwebs.proxool.ConnectionPool.removeProxyConnection(ConnectionPool.java:441):ContainerBackgroundProcessor[StandardEngine[Catalina]] - 000003 (00/05/00) - #0006 removed because of shutdown. [DEBUG] org.logicalcobwebs.proxool.ConnectionPool.shutdown(ConnectionPool.java:547):ContainerBackgroundProcessor[StandardEngine[Catalina]] - Connection #6 closed [DEBUG] org.logicalcobwebs.proxool.ConnectionPool.removeProxyConnection(ConnectionPool.java:441):ContainerBackgroundProcessor[StandardEngine[Catalina]] - 000003 (00/04/00) - #0005 removed because of shutdown. [DEBUG] org.logicalcobwebs.proxool.ConnectionPool.shutdown(ConnectionPool.java:547):ContainerBackgroundProcessor[StandardEngine[Catalina]] - Connection #5 closed [DEBUG] org.logicalcobwebs.proxool.ConnectionPool.removeProxyConnection(ConnectionPool.java:441):ContainerBackgroundProcessor[StandardEngine[Catalina]] - 000003 (00/03/00) - #0004 removed because of shutdown. [DEBUG] org.logicalcobwebs.proxool.ConnectionPool.shutdown(ConnectionPool.java:547):ContainerBackgroundProcessor[StandardEngine[Catalina]] - Connection #4 closed [DEBUG] org.logicalcobwebs.proxool.ConnectionPool.removeProxyConnection(ConnectionPool.java:441):ContainerBackgroundProcessor[StandardEngine[Catalina]] - 000003 (00/02/00) - #0003 removed because of shutdown. [DEBUG] org.logicalcobwebs.proxool.ConnectionPool.shutdown(ConnectionPool.java:547):ContainerBackgroundProcessor[StandardEngine[Catalina]] - Connection #3 closed [DEBUG] org.logicalcobwebs.proxool.ConnectionPool.removeProxyConnection(ConnectionPool.java:441):ContainerBackgroundProcessor[StandardEngine[Catalina]] - 000003 (00/01/00) - #0002 removed because of shutdown. [DEBUG] org.logicalcobwebs.proxool.ConnectionPool.shutdown(ConnectionPool.java:547):ContainerBackgroundProcessor[StandardEngine[Catalina]] - Connection #2 closed [DEBUG] org.logicalcobwebs.proxool.ConnectionPool.removeProxyConnection(ConnectionPool.java:441):ContainerBackgroundProcessor[StandardEngine[Catalina]] - 000003 (00/00/00) - #0001 removed because of shutdown. [DEBUG] org.logicalcobwebs.proxool.ConnectionPool.shutdown(ConnectionPool.java:547):ContainerBackgroundProcessor[StandardEngine[Catalina]] - Connection #1 closed [INFO ] org.logicalcobwebs.proxool.ConnectionPool.shutdown(ConnectionPool.java:564):ContainerBackgroundProcessor[StandardEngine[Catalina]] - 'autumn' pool has been closed down by ContainerBackgroundProcessor[StandardEngine[Catalina]] in 6 milliseconds. [DEBUG] org.logicalcobwebs.proxool.ShutdownHook.remove(ShutdownHook.java:41):ContainerBackgroundProcessor[StandardEngine[Catalina]] - Removed shutdownHook [INFO ] org.logicalcobwebs.proxool.PrototyperController.shutdown(PrototyperController.java:100):ContainerBackgroundProcessor[StandardEngine[Catalina]] - Stopping Prototyper thread [INFO ] org.logicalcobwebs.proxool.HouseKeeperController.shutdown(HouseKeeperController.java:107):ContainerBackgroundProcessor[StandardEngine[Catalina]] - Stopping HouseKeeper thread
排除了 poxool 導致 數據庫連接洩露的問題。