tracker issue : CF-4167008

select a category, or use search below
(searches all categories and all time range)
Title:

SEVERE: Error in getRealPathFromConn under load

| View in Tracker

Status/Resolution/Reason: Closed/Withdrawn/NotABug

Reporter/Name(from Bugbase): Henry Ho / Henry Ho (Henry Ho)

Created: 06/22/2016

Components: Installation/Config, Connector

Versions: 11.0

Failure Type: Crash

Found In Build/Fixed In Build: CF11_Final /

Priority/Frequency: Critical / Most users will encounter

Locale/System: English / Win All

Vote Count: 0

Problem Description:

org.apache.coyote.ajp.AjpProcessor getRealPathFromServer
SEVERE: Error in getRealPathFromConn

I got many of these in coldfusion-error.log and CF was unresponsive with Sandbox Security on.  There were a few rules on Files/Dirs and I use coldbox 3.8.  I'm already using latest hotfix update 9.


Steps to Reproduce:

Render pages under different paths under load, with sandbox security on with some file/dir paths restrictions.

Actual Result: Unresponsive CF

Expected Result: Working CF

Any Workarounds:  Disable sandbox security..


Stack Trace: 

org.apache.coyote.ajp.AjpProcessor getRealPathFromServer
SEVERE: Error in getRealPathFromConn
java.net.SocketException: Connection reset by peer: socket write error
	at java.net.SocketOutputStream.socketWrite0(Native Method)
	at java.net.SocketOutputStream.socketWrite(Unknown Source)
	at java.net.SocketOutputStream.write(Unknown Source)
	at org.apache.coyote.ajp.AjpProcessor.getRealPathFromServer(AjpProcessor.java:407)
	at org.apache.coyote.ajp.AbstractAjpProcessor.getRealPath(AbstractAjpProcessor.java:1361)
	at org.apache.naming.resources.FileDirContext.doGetRealPath(FileDirContext.java:202)
	at org.apache.naming.resources.BaseDirContext.getRealPath(BaseDirContext.java:424)
	at org.apache.catalina.core.StandardContext.getRealPath(StandardContext.java:4760)
	at org.apache.catalina.core.ApplicationContext.getRealPath(ApplicationContext.java:459)
	at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.apache.catalina.core.ApplicationContextFacade$1.run(ApplicationContextFacade.java:854)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.apache.catalina.core.ApplicationContextFacade.executeMethod(ApplicationContextFacade.java:851)
	at org.apache.catalina.core.ApplicationContextFacade.invokeMethod(ApplicationContextFacade.java:799)
	at org.apache.catalina.core.ApplicationContextFacade.doPrivileged(ApplicationContextFacade.java:770)
	at org.apache.catalina.core.ApplicationContextFacade.getRealPath(ApplicationContextFacade.java:332)
	at coldfusion.runtime.ServletContextWrapper.doGetRealPath(ServletContextWrapper.java:184)
	at coldfusion.runtime.ServletContextWrapper._doGetRealPath(ServletContextWrapper.java:116)
	at coldfusion.runtime.ServletContextWrapper.access$000(ServletContextWrapper.java:29)
	at coldfusion.runtime.ServletContextWrapper$1.run(ServletContextWrapper.java:92)
	at java.security.AccessController.doPrivileged(Native Method)
	at coldfusion.runtime.ServletContextWrapper.getRealPath(ServletContextWrapper.java:88)
	at coldfusion.runtime.RuntimeServiceImpl.getRealPath(RuntimeServiceImpl.java:1151)
	at coldfusion.runtime.RuntimeServiceImpl.getRealPath(RuntimeServiceImpl.java:1165)
	at coldfusion.filter.FusionContext.getRealPath(FusionContext.java:967)
	at coldfusion.filter.FusionContext.getRealPath(FusionContext.java:939)
	at coldfusion.runtime.RuntimeServiceImpl.resolveTemplatePath(RuntimeServiceImpl.java:898)
	at coldfusion.tagext.lang.IncludeTag$ResolveTemplatePathPrivilege.run(IncludeTag.java:339)
	at java.security.AccessController.doPrivileged(Native Method)
	at coldfusion.tagext.lang.IncludeTag.setTemplate(IncludeTag.java:426)
        ...

----------------------------- Additional Watson Details -----------------------------

Watson Bug ID:	4167008

Reason:	PRHaveInfo

External Customer Info:
External Company:  
External Customer Name: Henry Ho
External Customer Email:  
External Test Config: My Hardware and Environment details:



Windows Server 2012

IIS

CF11 Enterprise on JRE 1.8

Attachments:

Comments:

Did you recreate the connector, after applying the Update?
Comment by Anit K.
2298 | June 23, 2016 03:42:33 AM GMT
yes, isapi_redirect.dll file version: 1.2.41.0 product version: 1.2.41
Comment by External U.
2299 | June 23, 2016 03:57:08 AM GMT
Henry, I have a few scenarios where getRealPathFromConn gets invoked with load, on security sandbox, but do not see the issue yet. Could you please give us pointers on, - Whether you were able to narrow the issue down to a specific code snippet or operation. - The load you experience on your servers. - Does this issue occur ONLY with the latest update applied?
Comment by Immanuel N.
2300 | July 02, 2016 06:32:32 AM GMT
1.) not yet. With JMeter we cannot reproduce it so far but we do notice that sandbox security slows things down significantly. Shall we setup a Adobe Connect and let your engineers take a look? We have sent in a thread dump before to cfinstal 2.) the load was live traffic that used to hit CF10 server with resource security with no problem. 3.) Cannot uninstall update 9 without ruining the server. Once uninstalled, the instance cannot be started.
Comment by External U.
2301 | July 02, 2016 09:38:00 AM GMT
We're experiencing this bug as well on both CF11 and CF16. It seems to be causing, or perhaps the result of a frequently occuring HTTP Error 502 "Bad Gateway". Here's info from IIS failed connection tracing: ModuleName="IsapiModule", Notification="EXECUTE_REQUEST_HANDLER", HttpStatus="502", HttpReason="Bad Gateway", HttpSubStatus="0", ErrorCode="The operation completed successfully. (0x0)", ConfigExceptionInfo="" This is occuring on production servers under load, numerous times each day (i.e. hundreds of times). Users receive this error in the browser: 502 - Web server received an invalid response while acting as a gateway or proxy server. There is a problem with the page you are looking for, and it cannot be displayed. When the Web server (while acting as a gateway or proxy) contacted the upstream content server, it received an invalid response from the content server. Adobe please escalate this issue.
Comment by External U.
2302 | July 31, 2016 07:29:30 AM GMT
Henry, Claire, we are trying to reproduce this issue at our end. Will provide an update soon.
Comment by Immanuel N.
2303 | August 02, 2016 03:33:13 AM GMT
We discovered what the problem is related to. An activity logging job was being spawned at the end of various pages, as a separate thread. So the spawning page would end before the thread would end, somehow leaving the thread orphaned and causing Tomcat or ColdFusion to malfunction. We believe it was Tomcat malfunctioning. We have resolved the problem by not spawning the activity logging as a separate thread. We also noticed that the problem only seemed to occur on Windows 8 servers, but not Windows 12 servers, but it occurred in both CF11 and CF16.
Comment by External U.
2304 | August 02, 2016 08:25:16 AM GMT
@Claire would you please share more about that activity logging job? How did you disable that? Thanks.
Comment by External U.
2305 | August 02, 2016 11:29:28 AM GMT
We simply restructured it so that the activity logging was NOT performed in a spawned thread, but was instead performed in line with the web page's actual script (i.e. in the page thread). Also note that when we were spawning the activity logging in a separate thread, we were also NOT including in the page thread a "wait" for the spawned thread to join back to the page thread before continuing. In other cases where we spawn separate threads in the middle of script somewhere without a wait for the spawned thread to rejoin the page thread, and there is additional script following the thread spawn and the spawned thread ends before the page thread, there's no problem. But in this case, since the spawned thread happened at the very end of the page script, and there was absolutely nothing folowing it, the page thread ended before the spawned thread ended, causing this strange intermittent problem identified in this bug report, as well as significant numbers of 502 "bad gateway" errors with the TomCat isapi_redirector.dll. We were not including a thread wait/join, because we truly wanted the page thread to end as soon as possible and just have the spawned thread running in the background separately. But this apparently was entirely what was causing the problem. Hope this helps...
Comment by External U.
2306 | August 02, 2016 11:41:15 AM GMT
@Claire oh, it's a cfthread in your app. I thought it is some tomcat default logging thread that I can disable. Thanks for clearing that up.
Comment by External U.
2307 | August 02, 2016 11:44:36 AM GMT
Claire, Would it be possible to give us a repo of your code that results in 502's? We would like to determine if the issue can be handled by ColdFusion.
Comment by Immanuel N.
2308 | August 03, 2016 09:07:43 PM GMT
Unfortunately it's no longer available in the CFTHREAD structure, since that was eliminated. You can duplicate it by simply launching any CFTHREAD call at the end of any script, and in the thread-level script, make it do some database insert that will take like one second or so, and you should be able to easily view the problem immediately in the TomCat logs and CF logs, especially if you can simulate some light traffic on the server.
Comment by External U.
2309 | August 03, 2016 10:18:43 PM GMT
Hi Henry, Can you let us know, if you are still facing this issue? Is your use-case is also similar to what Claire has reported? This bug has been open for more than 4 months, so can we go-ahead and close it, if it's not an issue? Regards, Hari
Comment by HariKrishna K.
29586 | January 08, 2018 09:30:44 AM GMT
Seems like the user contacted supported, was in touch for sometime. Looks like the issue is resolved and even support haven't  heard back from him. Hence closing the bug.  If in case the issue is still not resolved, please contact support with the info requested.
Comment by HariKrishna K.
29600 | August 22, 2018 11:09:02 AM GMT