tracker issue : CF-3222748

select a category, or use search below
(searches all categories and all time range)
Title:

Server Non-responseive when receiving a POST with a type XML param

| View in Tracker

Status/Resolution/Reason: Closed/Fixed/

Reporter/Name(from Bugbase): Daniel Short / Daniel Short (Daniel Short)

Created: 06/25/2012

Components: Installation/Config, Connector

Versions: 10.0

Failure Type: Crash

Found In Build/Fixed In Build: Final / latest

Priority/Frequency: Critical / All users will encounter

Locale/System: English / Win 2008 Server R2 64 bit

Vote Count: 21

Related Bugs:
CF-3138249 - Similar to
CF-3300889 - Similar to


Problem Description:

When sending a post contain an XML body to a ColdFusion 10 server, the server hangs receiving the response, and in some cases, completely stops responding.

Steps to Reproduce:

Download the attached zip file, which contains an Application.cfc file, poster.cfm, receiver.cfm, and a recent extract of the CNET RSS feed. Drop all of the files into a single folder, and load poster.cfm.


Actual Result:
The log file generated, which should contain just two entries, contains a number of entries, one right after the other, even though poster.cfm is only loaded once (see XMLTest.log in the attached zip).

I then receive an error on the Windows site that the IIS Work Process has stopped working, along with the following text (see IISWorkProcessDialog.txt):

The Windows Application log has the following information (see EventViewer.txt):

ColdFusion then becomes completely non-responsive, returning Service Unavailable on any subsequent requests. Only a restart of IIS (no need to restart ColdFusion) will bring everything back up.

Expected Result:

I should see two entires in the log file, and the poster.cfm file should display the HTTP request data, including the originally submitted RSS file.

Any Workarounds:

None.

----------------------------- Additional Watson Details -----------------------------

Watson Bug ID:	3222748

Keywords:
FixTested


External Customer Info:
External Company:  
External Customer Name: webshorts
External Customer Email:  
External Test Config: My Hardware and Environment details:

Attachments:

  1. June 25, 2012 00:00:00: 1_XMLCFHTTP.zip
  2. June 25, 2012 00:00:00: 2_isapi_redirect.zip

Comments:

I can cause this to happen every time on two different installations, both on IIS on Windows Server 2008.
Comment by External U.
18884 | June 25, 2012 07:30:54 AM GMT
It looks like this might be relevant: https://issues.apache.org/bugzilla/show_bug.cgi?id=51767 The suggestion in the post is to use version 1.2.35.
Comment by External U.
18885 | June 25, 2012 07:52:47 AM GMT
I've attached a log from isapi_redirect showing the error that causes the dll to fail. The log ends with: [Mon Jun 25 09:22:00.951 2012] [1804:3224] [error] ajp_connection_tcp_send_message::jk_ajp_common.c (1222): (cfusion) unknown protocol 795177583, supported are AJP13/AJP14 [Mon Jun 25 09:22:00.951 2012] [1804:3224] [error] ajp_get_reply::jk_ajp_common.c (2240): (cfusion) Tomcat is down or network problems [Mon Jun 25 09:22:00.952 2012] [1804:3224] [error] ajp_service::jk_ajp_common.c (2622): (cfusion) sending request to tomcat failed (unrecoverable), (attempt=1) [Mon Jun 25 09:22:00.952 2012] [1804:3224] [error] HttpExtensionProc::jk_isapi_plugin.c (2272): service() failed with http error 502
Comment by External U.
18886 | June 25, 2012 08:54:15 AM GMT
I've discovered that the cutoff for the crash is 8k. If I send a body xml with 8200 bytes, it crashes. 8062: Processes fine 8108: Bad Gateway message 8200: Worker process crashes
Comment by External U.
18887 | June 25, 2012 09:33:18 AM GMT
There is an easy workaround for this. You can enable "Cache Web Server Paths" setting in the Caching page of the ColdFusion Administrator.
Comment by Vamseekrishna N.
18888 | August 09, 2012 04:24:59 AM GMT
Hi Vamseekrishna, How does caching the web server paths correct the issue? That option causes other problems that can't be worked around on a server with more than one website running. Dan
Comment by External U.
18889 | August 15, 2012 06:54:38 AM GMT
We have just encountered this issue. We can also conform the work-around gets things back to working. We have a single IIS web site / Application pool. But we actually create multiple CF applications from the same code - but using part of the URL to create the cfapplication name. Is this considered multi-site or single site? Is the setting getting to adversely effect our application / customers using it? As always - thanks. Gavin Baumanis
Comment by External U.
18890 | August 19, 2012 06:56:32 PM GMT
+1 . This has caught us out, too.
Vote by External U.
18921 | August 19, 2012 09:41:44 PM GMT
Gavin, It's likely that you will have templates from Site A that will get executed under Site B if they have the same names. I've been digging to try and find out exactly what the cache web server path option does, but can't seem to find anything other than "It caches the web server paths". Unfortunately, that doesn't tell me anything...
Comment by External U.
18891 | August 21, 2012 08:22:48 AM GMT
we are investigating this. It looks like something is getting messed up between the tomcat connector and the web server in the webservice request when POST size is exceeding a certain limit. We found a temporary workaround which prevents this - and that is enable ’cache web server path’ setting in the admin. I know the workaround sounds completely unrelated but it is indeed the case. We also understand that workaround might not be acceptable if you have multiple websites and would like to assure you that we are working on it. Has anyone faced this issue with non-webservice request also?
Comment by Rupesh K.
18892 | August 22, 2012 11:09:17 AM GMT
I've only seen it in the example previously supplied.
Comment by External U.
18893 | August 23, 2012 01:58:01 PM GMT
This issue happens in our application as well
Vote by External U.
18922 | September 04, 2012 04:56:24 PM GMT
When the POST size is greater than 8,090 characters, I get a 502 error. Subsequent requests generate error 500 until the IIS Application Pool is recycled. If the post size is significantly larger, the IIS worker process crashes isapi_redirect.log shows: ajp_connection_tcp_send_message::jk_ajp_common.c (1222): (cfusion) unknown protocol 1701995878, supported are AJP13/AJP14 Application Log: Event ID: 1000 Description: Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2 Faulting module name: isapi_redirect.dll, version: 1.2.32.0, time stamp: 0x4f745a78 Exception code: 0xc0000005 Fault offset: 0x0000000000018b93 Faulting process id: 0x20f4 Faulting application start time: 0x01cd8b8219e1ff58 Faulting application path: c:\windows\system32\inetsrv\w3wp.exe Faulting module path: C:\ColdFusion10\config\wsconfig\1\isapi_redirect.dll
Comment by External U.
18894 | September 06, 2012 09:31:08 AM GMT
Sorry, I basically confirmed what webshorts mentioned back on 6/25.
Comment by External U.
18895 | September 06, 2012 09:42:26 AM GMT
I can confirm that enabling the "Cache web server paths" setting under Server Settings --> Caching in the ColdFusion Administrator will fix the issue. Unfortunately, I can't enable this setting on our development server because it hosts multiple websites. (Also, probably not an option if you're hosting in a shared environment.)
Comment by External U.
18896 | September 06, 2012 11:08:40 AM GMT
WEB SERVICES ARE BROKEN We rely heavily on web services (a major feature of ColdFusion) to communicate with other applications. Because of this bug, XML SOAP requests greater than 8KB crash the server. We cannot deploy CF10 to our clients until this is fixed.
Vote by External U.
18923 | September 07, 2012 08:18:57 AM GMT
We do alot of these requests and our Production CF10 server has been crashing because of this. Reverted to 9
Vote by External U.
18924 | September 11, 2012 08:39:23 AM GMT
Also experiencing the issue. Where is Adobe's commentary on this?
Vote by External U.
18925 | September 18, 2012 10:58:56 AM GMT
I want to get update for this so voting for this bug.
Vote by External U.
18926 | September 18, 2012 11:28:53 AM GMT
We are currently in the process of fixing this issue.It will be available in the next update.
Comment by Vamseekrishna N.
18897 | September 18, 2012 02:23:36 PM GMT
Just in case the impact of this bug wasn't immediately obvious - this also affects web service endpoints. If you are hosting any SOAP web services on CF10, this bug impacts you. If you're hosting REST web services with the content type set to XML, this affects you. This is potentially a large number of users.
Comment by External U.
18898 | September 19, 2012 03:29:19 PM GMT
After upgrading cf 9 to 10 all our webbservices crashed. we had to set up a seperate server with the 9 version. The error loggs showed up exactly as the entries described in this thread, so I voute for a quick fix.
Vote by External U.
18927 | September 23, 2012 02:50:39 AM GMT
Adobe, In case you were wondering, this is now a security vulnerability too. This can be used to DOS any server running CF10. The lack of a prompt response to this is baffling, and you should be embarrassed.
Comment by External U.
18899 | September 25, 2012 01:14:38 PM GMT
isapi connector crashing IIS app pool
Vote by External U.
18928 | September 25, 2012 05:51:55 PM GMT
This bug just caused us to stop draining connections from our CF9 servers to our CF10 servers. We had to reverse that process and go back to CF9. This is terrible. This bug didn't show up in any of our unit or functional testing. I've already merged our CF10 changes into our trunk and had all developers migrate their boxes to CF10. All development is at a standstill until we figure out what we are going to do.
Vote by External U.
18929 | September 25, 2012 08:25:19 PM GMT
We have fixed this bug and we are currently testing it. Could you please help us in testing this before we make it public? If you are interested, please reach out to me at 'rukumar' @t adobe so that I can provide you the fix offline.
Comment by Rupesh K.
18900 | September 26, 2012 05:34:42 AM GMT
Rupesh. Sent an email to you at 7:22 am PST. When can I expect to be provided with the fix? This issue is preventing us from using multiple enterprise licenses that we paid thousands for on servers that we just built. I'm sorry if I sound grumpy. But, I'm not happy about a 3 month old issue that will SHUT DOWN IIS sites.
Comment by External U.
18901 | September 26, 2012 10:41:54 AM GMT
I'm with you on this cf_chris. I have > $15,000 in licenses gathering dust, and a failed production deployment since I opened this bug back in June. I'm hoping that Adobe can do something to make up for the lost time here, considering they've known about the issue for months. I'm well outside my 90 days of support, so I'm going to be expecting some increased level of direct support once this is resolved and I can push our servers into production.
Comment by External U.
18902 | September 26, 2012 11:12:46 AM GMT
The jars that I received from Rupesh have fixed the one test I had that was able to consistently reproduce this error. I would be really interested to hear from anyone else that is able to test the jars. Does it fix your test cases too?
Comment by External U.
18903 | September 26, 2012 11:59:53 AM GMT
With the updated jars installed, we have been routing live traffic to 2 of our new CF10 instances for over an hour. I am happy to say [knock on=wood/] that they are humming along at about 10 requests/second each. We have served tens of thousands of requests so far and not experienced the issue yet. Still eager to hear if anyone else has gotten a chance to use the new jars.
Comment by External U.
18904 | September 26, 2012 03:04:44 PM GMT
Unfortunately the wood I knocked on earlier turned out to be plastic laminate. And shortly after I posted, our instances became unresponsive. Upon investigation of the log files, the "coldfusion-out" log was being filled to bursting with entries like "Chunk len is 6050 [60, 116, 97, 98,...". So, the current "fixed" jars are not yet production ready. I think they may be close. It seems probable that this constant logging of hundreds of megabytes of entries is what locked up the instances.
Comment by External U.
18905 | September 26, 2012 05:40:34 PM GMT
I've not seen anyone else propose this as a workaround: I helped someone resolve this same problem. We changed the POST to a GET, and the problem was solved. May not suit everyone, especially if you have no control over the caller, but if you can try, don't miss considering this potentially simple solution. Feel free to chime in if it does or does not work for you, in case it may help others (or we may find that it helps in some cases and not in others).
Comment by External U.
18906 | September 29, 2012 03:57:57 PM GMT
We are encountering this problem on a production server, has there been any updates as to when a proper fix will be available? This is greatly affecting our CF applications
Comment by External U.
18907 | October 02, 2012 12:30:53 PM GMT
This stops us from entering large amounts of user content into our system which is the corner stone of our application
Vote by External U.
18930 | October 02, 2012 12:32:15 PM GMT
This is affecting our production server - requiring workarounds or service restarts.
Vote by External U.
18931 | October 05, 2012 05:47:50 AM GMT
Same problem! Just bought CF10 and this is killing our production server! We need our web services working again!!! PLEASE FIX ASAP!!
Vote by External U.
18932 | October 08, 2012 10:17:27 PM GMT
The fix for this issue will be available in an upcoming update very soon
Comment by Vamseekrishna N.
18908 | October 08, 2012 11:08:03 PM GMT
Please Adobe, fix that soon! We really need this fix. We have a lot of trouble over this!!!!!!!!!!!!!!
Vote by External U.
18933 | October 11, 2012 06:21:17 AM GMT
When the next update comes? Rupesh Kumar, Can you send it to me, offline? etnrlz@gmail.com
Comment by External U.
18909 | October 11, 2012 06:22:29 AM GMT
FYI, the update works fine on our W2K8 servers and dev laptops. However, if you are running an "unsupported" install on Server 2K12 or Win8, then do NOT apply this patch. This patch will cause the service to be unable to start, with an event log entry similar to the following: Faulting application name: coldfusion.exe, version: 10.0.0.0, time stamp: 0x5052ed25 Faulting module name: ntdll.dll, version: 6.2.9200.16420, time stamp: 0x505ab405 Exception code: 0xc0000374 Fault offset: 0x00000000000ea485 Faulting process id: 0x1cec Faulting application start time: 0x01cdac210d49b69e Faulting application path: C:\ColdFusion10\cfusion\bin\coldfusion.exe Faulting module path: C:\Windows\SYSTEM32\ntdll.dll Report Id: 4c5276c4-1814-11e2-be85-d4bed93d400a Faulting package full name: Faulting package-relative application ID: Given how long *this* bug has been open, I suspect we will see official support for W2K12 and Win8 sometime around MARCH of next year.
Comment by External U.
18910 | October 16, 2012 11:14:12 PM GMT
When official patch will be issued?
Vote by External U.
18934 | October 22, 2012 11:34:04 AM GMT
oof da! What the hell is going on over there at Adobe? When will this "FIX" be made public??? On Oct 8th, we were told "very soon". Hello?
Comment by External U.
18911 | October 22, 2012 01:36:31 PM GMT
The fix for this bug is available in update 4 released on 2nd Nov 2012.
Comment by Rupesh K.
18912 | November 02, 2012 01:26:05 PM GMT
Would it be possible that this bug got re-introduced in update 7? Or that they forgot to include this particular fix in the cumulative update 7? It re-appeared in CF10 servers freshly installed with only the mandatory update and immediately patched to update 7.
Comment by External U.
18913 | February 14, 2013 03:33:47 AM GMT
I am currently on update 8, and I recieve the same problem. Running on Windows 2003 32-bit. Has anyone had any luck with Adobe getting this resolved.
Comment by External U.
18914 | March 29, 2013 01:45:47 PM GMT
could you please confirm whether you re-configured the connector after applying the update?
Comment by Rupesh K.
18915 | March 31, 2013 09:09:17 PM GMT
I am having this issue - we try to upload an image and 503 error - locks up the server, have to re-boot - is this still a problem with everyone?
Vote by External U.
18935 | August 20, 2013 09:31:48 AM GMT
I've updated to 4, then to 11 then back down to 7 all don't work and bomb things out with 503 errors - why do we still have this issue?
Comment by External U.
18916 | August 20, 2013 09:32:45 AM GMT
@PaulG, could you please confirm that you re-configured the connector after applying the update?
Comment by Rupesh K.
18917 | August 21, 2013 09:06:21 AM GMT
Still seeing this issue with one of our sites.
Vote by External U.
18936 | September 19, 2013 10:30:15 AM GMT
We experience more or less the same problem while sending JSON packages larger than +/- 7500 byters with contentType="application/json". The server stops responding and fails before sending any response headers. [Thu Sep 26 15:06:44 2013] [12854:140735210590592] [error] ajp_connection_tcp_send_message::jk_ajp_common.c (1222): (cfusion) unknown protocol 538970668, supported are AJP13/AJP14 [Thu Sep 26 15:06:44 2013] [12854:140735210590592] [error] ajp_get_reply::jk_ajp_common.c (2240): (cfusion) Tomcat is down or network problems [Thu Sep 26 15:06:44 2013] [12854:140735210590592] [error] ajp_service::jk_ajp_common.c (2622): (cfusion) sending request to tomcat failed (unrecoverable), (attempt=1) I can reproduce it on several development machines. Environment: Max OSX 10.8.4 ColdFusion 10 Update 11 Apache/2.2.22 (Unix) mod_jk 1.2.37 (the same error occurs with the original connector version that comes with CF10).
Comment by External U.
18918 | September 26, 2013 07:43:51 AM GMT
This bug renders our application unusable.
Vote by External U.
18937 | September 26, 2013 07:45:23 AM GMT
+1 I have seen this issues as well.
Vote by External U.
18938 | September 26, 2013 07:50:56 AM GMT
We experiencing the same issue. Could you please fix it or provide a work arround!
Vote by External U.
18939 | September 26, 2013 07:54:50 AM GMT
I am experiencing a JSON post content length limit of exactly 8090 characters... I am trying to post JSON with content-length of 100-200K and cannot. Any fixes??? var upload= function (data) { var options = { url: app.baseUrl + 'upload.cfm', type: 'POST', data: JSON.stringify(data), contentType: "application/json; charset=utf-8", mimeType: "application/json; charset=utf-8", dataType: "json", beforeSend: util.loaderStart() }; $.ajax(options).always(util.loaderEnd()).then(success).fail(fail); } Environment: Windows Server 2008 R2 64bit IIS 7.5 Coldfusion 10 update 11 [I have reconfigured connector]
Comment by External U.
18919 | October 17, 2013 04:41:13 PM GMT
Using Coldfusion 10 update 11 and the problem still exists. Is there a fix in sight?
Vote by External U.
18940 | October 17, 2013 04:52:02 PM GMT
To everyone posting comments here saying that things "still don't work", can you please note the observation Rupesh has made multiple times: have you updated the web server connector? (Cory, I see you say you did. Are you positive that the connector file's date was updated?) Updating that connector is so very important for some of the bugs fixed in updates over the past year, yet it's easily missed. For more info, I did a blog entry on it a few weeks ago with a lot more detail: CF911: Why/when you MUST update the web server connector for #ColdFusion 10, and may have missed it http://www.carehart.org/blog/client/index.cfm/2013/9/13/why_you_must_update_cf10_webserver_connector If someone does that carefully, confirming the updated date of the connector file, and then STILL has a problem, then do report here in case somehow your problem remains. (Most votes and comments are from 2012, so hopefully they have long since taken care of this.) Finally, Cory, besides double-checking that connector update date, I'm curious: your problem of posting json doesn't seem quite to fit with the point of this bug, about posting xml. You may want to create another one, if you don't get any specific response here. Some people who might look for the problem may not think to look at a bug entry with a name like this one. Just a suggestion.
Comment by External U.
18920 | October 17, 2013 10:25:12 PM GMT
Our SOAP and REST web services have been rendered useless by this bug. This is a major problem!!!
Vote by External U.
18941 | November 19, 2013 08:29:18 AM GMT