Status/Resolution/Reason: Closed/Fixed/
Reporter/Name(from Bugbase): Anthony Galano / Anthony Galano (WebPexDev)
Created: 01/08/2014
Components: Performance
Versions: 9.0.1
Failure Type: Memory Leak
Found In Build/Fixed In Build: 9.0.1 /
Priority/Frequency: Major / All users will encounter
Locale/System: English / Win 2003 Server x64
Vote Count: 0
When a cfc is called directly with no method argument the server begins to chew up memory.
Calling a cfc directly like http://www.domain.com/controller/LoginController.cfc will run in the browser until it times out.
The /CFIDE has been locked down and is not publicly accessible so the cfexplorer is not (or should not) be available.
Monitoring the active threads using FusionReactor show there are no threads running event though the page is still running the browser.
Even though no threads are shown as actively running, monitoring the memory usage shows that is slowly being consumed.
Using FusionReactor and inspecting the PermGen memory space found that it was continuing to grow.
Forced garbage collection recovers no memory even though there is no traffic on the server other than this single request.
The memory usage eventually tops out, presumably because the thread has timed our been killed but memory still cannot be recovered.
Performing a memory dump and analyzing it with MAP shows the following object in memory consuming 1.7GB
Class Name | Shallow Heap | Retained Heap | Percentage
---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- --------------------------------------------------------
byte[1769628928] @ 0x4d963b198 ...128.................POST......../controller/LoginController.cfc... ...../controller/LoginController.cfc........173.14.93.66........173.14 .93.66........www.domain.com........443........HTTP/1.1.......;D:\webs ites\domain\system\controller\Lo...| 1,769,628,944 | 1,769,628,944 | 8.60%
---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- ---------------------------------------------------------------------- --------------------------------------------------------
The only way to recover this memory is a server restart.
Calling an cfc via URL with a method argument (http://www.domain.com/controller/LoginController.cfc?method=foo) invokes the onMissingMethod handler and redirects to the appropriate error page with no server effect.
If I make multiple requests in separate tabs to the URL with no method call I can bring down the server fairly quickly. Even with a dozen requests to
http://www.domain.com/controller/LoginController.cfc FusionReactor still shows no active requests while memory is quickly exhausted.
All the huge objects are instances of jrun.servlet.jrpp.ProxyEndpoint.
I've included a screenshot of the MAP analysis.
This is causing consistent service downtime. CFC's are used to handle submissions and application flow in our app. Something (bots possibly) pick up the direct CFC URL and hit it. Over time these requests chew up and hold onto memory until is finally goes down. As can be seen in the screenshot of the MAP analysis there are 10 objects holding 1.7GB each using over 18GB of the 20GB allotted to the heap.
----------------------------- Additional Watson Details -----------------------------
Watson Bug ID: 3690526
Deployment Phase: Release Candidate
External Customer Info:
External Company:
External Customer Name: WebPexDev
External Customer Email:
External Test Config: My Hardware and Environment details:
This is CF9.01 Standard on Win2003x64 SP2 Server running Java 1.7.0_25.
Attachments:
- January 09, 2014 00:00:00: 1_Screen_Shot_2014-01-08_at_1.28.49_AM.JPG
- January 15, 2014 00:00:00: 2_Bug_3690526_-_Server_Info.txt
- January 15, 2014 00:00:00: 3_bug3690526.zip
Comments: