tracker issue : CF-3835411

Here is the event log info: A process serving application pool 'RadioSparx' suffered a fatal communication error with the Windows Process Activation Service. The process id was '8912'. The data field contains the error number. Details: EventData AppPoolID RadioSparx ProcessID 8912 6D000780 Binary data: In Words 0000: 8007006D In Bytes 0000: 6D 00 07 80 m..x80; Simultaneously getting an Event ID 1000: Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2 Faulting module name: isapi_redirect.dll, version: 1.2.37.0, time stamp: 0x54041b95 Exception code: 0xc0000409 Fault offset: 0x000000000003d7ad Faulting process id: 0x22d0 Faulting application start time: 0x01cfe10b8e88880f Faulting application path: c:\windows\system32\inetsrv\w3wp.exe Faulting module path: C:\ColdFusion11\config\wsconfig\1\isapi_redirect.dll Report Id: 00e2b56f-4cff-11e4-8e77-001b2176929d EventData: w3wp.exe 7.5.7601.17514 4ce7afa2 isapi_redirect.dll 1.2.37.0 54041b95 c0000409 000000000003d7ad 22d0 01cfe10b8e88880f c:\windows\system32\inetsrv\w3wp.exe C:\ColdFusion11\config\wsconfig\1\isapi_redirect.dll 00e2b56f-4cff-11e4-8e77-001b2176929d

Comment by External U.

Thanks for raising the issue! Could you please give us a few more specifics about your environment, to help us narrow down to the cause of failure? 1) Does the ColdFusion server have the Hotfix2 applied? 2) After approximately how much time, does the application pool come down 3) Connector logs, with the debug log level turned on. 4) IIS Failed Request Tracing logs

Comment by Immanuel N.

See additional answers and info below: 1) Does the ColdFusion server have the Hotfix2 applied? >> Yes 2) After approximately how much time, does the application pool come down >> Generally within 30 to 120 seconds at full load. 3) Connector logs, with the debug log level turned on. >> See attached file, which I created this a.m. by reintroducing isapi_redirect.dll version 1.2.37 to our production server and running it for a bit. The problem does not occur with isapi_redirect.dll 1.2.32. 4) IIS Failed Request Tracing logs >> Sorry this is not available. Please note that when I ran it this morning to create the connector log file on our production server, I did not run it long enough or with enough traffic to cause the application pool to crash. However I ran it long enough for the related errors to start logging in the event log: Event 5011, WAS A process serving application pool 'RadioSparx2' suffered a fatal communication error with the Windows Process Activation Service. The process id was '2688'. The data field contains the error number. + System - Provider [ Name] Microsoft-Windows-WAS [ Guid] {524B5D04-133C-4A62-8362-64E8EDB9CE40} [ EventSourceName] WAS - EventID 5011 [ Qualifiers] 32768 Version 0 Level 3 Task 0 Opcode 0 Keywords 0x80000000000000 - TimeCreated [ SystemTime] 2014-10-28T11:43:35.000000000Z EventRecordID 1253804 Correlation - Execution [ ProcessID] 0 [ ThreadID] 0 Channel System Computer AXC2.navarr.net Security - EventData AppPoolID RadioSparx2 ProcessID 2688 6D000780 -------------------------------------------------------------------------------- Binary data: In Words 0000: 8007006D In Bytes 0000: 6D 00 07 80 m..x80; ___________________________________________________________________________ This error is occuring when trying to download an mp3 file to clients of our commercial background music service. It occurs when either downloading the whole file at once with a CFHEADER/CFCONTENT method, and also when downloading it by streaming it to the client in chunks. The entire problem does not occur with isapi_redirect.dll 1.2.32, only with the 1.2.37 version. Also, I'm not sure if it is related but with the 1.2.32 version, we are getting thousands of these "errors" in our connector log files: [Tue Oct 28 07:54:23.768 2014] [12348:14084] [error] isapi_write_client::jk_isapi_plugin.c (1400): WriteClient failed with 534 (0x00000216) [Tue Oct 28 07:54:23.770 2014] [12348:14084] [info] ajp_process_callback::jk_ajp_common.c (2089): Writing to client aborted or client network problems [Tue Oct 28 07:54:23.771 2014] [12348:14084] [info] ajp_service::jk_ajp_common.c (2795): (cfusion) sending request to tomcat failed (unrecoverable), because of client write error (attempt=1) [Tue Oct 28 07:54:23.772 2014] [12348:14084] [info] HttpExtensionProc::jk_isapi_plugin.c (2603): service() failed because client aborted connection Hope this helps. Regards, Lee

Comment by External U.

Additionally, we are still experience frequent realtime ColdFusion errors that have the following error info: The cause of this output exception was that: ClientAbortException: java.net.SocketException: Connection reset by peer: socket write error. This problem NEVER occurred when we were running CF 9, it only started when we installed CF11. This is causing actual web page or music download failures for our clients using our commercial background music service, costing us substantial credibility and client disatisfaction. Please escalate this. Thanks, Lee

Comment by External U.

Hi Lee, Thanks for all the additional information you provided. We do see a issue, but could you please send us a couple of more logs for us to pinpoint the exact cause. > coldfusion-error.log > coldfusion-out.log > exception.log We basically want to see more details about the "ClientAbortException: java.net.SocketException" that you mentioned. Thanks.

Comment by Immanuel N.

Unfortunately those log files are not available from when we were trying to use the CF-supplied Tomcat Connector. We are now trying to use the Boncode connector, but that is not without its own set of issues. We are still seeing strange disconnects. I've attached a set of log files that you requested, plus the related Boncode log files. Hopefully this will help. I don't have a production server available now with the CF-supplied Tomcat Connector enabled to be able to generate the specific files you are requesting. Thanks for your help. Regards, Quinn

Comment by External U.

Thanks for all the information provided. We will be investigating this bug.

Comment by Immanuel N.

Just fyi, the Boncode connector is running fine for us in general now after tweaking the configuration. We are using CF 11 now in production and it is working well. There's no need to investigate the Boncode connector, just please investigate the issue with your own version of the Tomcat connector. That's the software that caused us so much grief when first deploying CF11 into production. It was an unexpected glitch when deployed to the production servers, we had been running CF11 for months in our test lab with no problems observed, except of course it did not have the traffic load that the production environment has so did not exhibit the problem symptoms.

Comment by External U.

Please expedite this Tomcat Connector fix. There are significant problems in the Boncode connector as well that are causing production problems for us.

Comment by External U.

We just put a CF11 enterprise server into production last night and are experiencing the same problems - we are advising our customer to roll back to their CF9 server. Is there any ETA on a fix for this? This is critical. From the logs: Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x5215df96 Faulting module name: isapi_redirect.dll, version: 1.2.37.0, time stamp: 0x532bd775 Exception code: 0xc0000409 Fault offset: 0x0000000000045c64 Faulting process id: 0x14e0 Faulting application start time: 0x01d003b58677070b Faulting application path: c:\windows\system32\inetsrv\w3wp.exe Faulting module path: C:\ColdFusion11\config\wsconfig\1\isapi_redirect.dll Report Id: cac9e896-6fa8-11e4-80cb-00155d01c806 Faulting package full name: Faulting package-relative application ID:

Comment by External U.

You know Adobe, you seriously could have F*$^ING mentioned that there was already a new version of the Tomcat connector available since November 7 with the CF 11 version 3 early release, here: http://blogs.coldfusion.com/post.cfm/coldfusion-11-update-3-is-available-for-early-access This has solved our connector problem. Would that really have been too hard to mention??

Comment by External U.

@Quinnster - can you confirm that the pre-release update 3 does actually fix the issue? I looked at the release notes and did not see it addressed so I advised our client to hold off on the update until the final is released - but I'll have them apply the patch now if it solves the issue. Thanks!

Comment by External U.

It solves the connector problems, everything running 100%, zero crashes since installing it.

Comment by External U.

Thanks for confirming that the issue is fixed, @Quinnster. We did release a handful of fixes on the connector with HF3 early release, but this particular bug, had to be investigated further to pinpoint the exact issue, and was in our queue. If HF3 does solve your issue completely, can this bug be closed?

Comment by Immanuel N.

The connector problem is not totally fixed, we are getting MANY MANY of these types of errors in the tomcat log: [Sat Nov 22 10:17:30.053 2014] [12496:3772] [info] ajp_service::jk_ajp_common.c (2795): (cfusion) sending request to tomcat failed (unrecoverable), because of client write error (attempt=1) [Sat Nov 22 10:17:30.054 2014] [12496:3772] [info] HttpExtensionProc::jk_isapi_plugin.c (2603): service() failed because client aborted connection [Sat Nov 22 10:17:38.176 2014] [12496:8860] [error] isapi_write_client::jk_isapi_plugin.c (1400): WriteClient failed with 995 (0x000003e3) [Sat Nov 22 10:17:38.182 2014] [12496:8860] [info] ajp_process_callback::jk_ajp_common.c (2089): Writing to client aborted or client network problems [Sat Nov 22 10:17:38.182 2014] [12496:8860] [info] ajp_service::jk_ajp_common.c (2795): (cfusion) sending request to tomcat failed (unrecoverable), because of client write error (attempt=1) [Sat Nov 22 10:17:38.183 2014] [12496:8860] [info] HttpExtensionProc::jk_isapi_plugin.c (2603): service() failed because client aborted connection [Sat Nov 22 10:17:40.219 2014] [12496:11024] [error] isapi_write_client::jk_isapi_plugin.c (1400): WriteClient failed with 1229 (0x000004cd) [Sat Nov 22 10:17:40.220 2014] [12496:11024] [info] ajp_process_callback::jk_ajp_common.c (2089): Writing to client aborted or client network problems [Sat Nov 22 10:17:40.220 2014] [12496:11024] [info] ajp_service::jk_ajp_common.c (2795): (cfusion) sending request to tomcat failed (unrecoverable), because of client write error (attempt=1) [Sat Nov 22 10:17:40.220 2014] [12496:11024] [info] HttpExtensionProc::jk_isapi_plugin.c (2603): service() failed because client aborted connection [Sat Nov 22 10:17:43.750 2014] [12496:11024] [error] isapi_write_client::jk_isapi_plugin.c (1400): WriteClient failed with 1229 (0x000004cd) [Sat Nov 22 10:17:43.750 2014] [12496:11024] [info] ajp_process_callback::jk_ajp_common.c (2089): Writing to client aborted or client network problems [Sat Nov 22 10:17:43.751 2014] [12496:11024] [info] ajp_service::jk_ajp_common.c (2795): (cfusion) sending request to tomcat failed (unrecoverable), because of client write error (attempt=1) [Sat Nov 22 10:17:43.751 2014] [12496:11024] [info] HttpExtensionProc::jk_isapi_plugin.c (2603): service() failed because client aborted connection [Sat Nov 22 10:17:47.505 2014] [12496:11532] [error] isapi_write_client::jk_isapi_plugin.c (1400): WriteClient failed with 1229 (0x000004cd) [Sat Nov 22 10:17:47.506 2014] [12496:11532] [info] ajp_process_callback::jk_ajp_common.c (2089): Writing to client aborted or client network problems [Sat Nov 22 10:17:47.506 2014] [12496:11532] [info] ajp_service::jk_ajp_common.c (2795): (cfusion) sending request to tomcat failed (unrecoverable), because of client write error (attempt=1) [Sat Nov 22 10:17:47.507 2014] [12496:11532] [info] HttpExtensionProc::jk_isapi_plugin.c (2603): service() failed because client aborted connection [Sat Nov 22 10:17:51.692 2014] [12496:5472] [error] isapi_write_client::jk_isapi_plugin.c (1400): WriteClient failed with 995 (0x000003e3) [Sat Nov 22 10:17:51.693 2014] [12496:5472] [info] ajp_process_callback::jk_ajp_common.c (2089): Writing to client aborted or client network problems [Sat Nov 22 10:17:51.693 2014] [12496:5472] [info] ajp_service::jk_ajp_common.c (2795): (cfusion) sending request to tomcat failed (unrecoverable), because of client write error (attempt=1) [Sat Nov 22 10:17:51.693 2014] [12496:5472] [info] HttpExtensionProc::jk_isapi_plugin.c (2603): service() failed because client aborted connection [Sat Nov 22 10:18:02.606 2014] [11288:11436] [info] ajp_send_request::jk_ajp_common.c (1701): (cfusion) all endpoints are disconnected, detected by connect check (1), cping (0), send (0) This is causing randomly the graphics to not appear on the web site properly. Please review and fix.

Comment by External U.

More: This is with ColdFusion 11, update 3 (re-release of update 3), as explained here: http://blogs.coldfusion.com/post.cfm/coldfusion-11-update-3-is-available-for-early-access

Comment by External U.

We are still experiencing a disfunctional ClientAbortException: java.net.SocketException bug with Tomcat 12.40 when trying to download/deliver an mp3 file to the browser with a standard CFHEADER + CFCONTENT, like this: <CFHEADER NAME='Content-Length' value='#variables.fileSize#'> <CFHEADER NAME='Content-Disposition' value='inline; filename="#variables.tfilename#"'> <CFCONTENT TYPE='audio/mpeg' FILE='#variables.filepath#'> ERROR THAT RESULTS: Failure detected running track.cfm in CFCONTENT command Type: Application Message: The cause of this output exception was that: ClientAbortException: java.net.SocketException: Connection reset by peer: socket write error. Detail: URL: http://www.somedomain.com/rsc/url.... PLEASE NOTE: THE CLIENT IS NOT ABORTING THE CONNECTION!! TOMCAT IS SCREWING UP SOMEHOW AND ERRORING OUT.

Comment by External U.

We are still experiencing issues with Tomcat 12.40 when trying to download large files. This happens with MP3 files as previously reported on Nov 29 below, and is happening when our clients try to download other file types. We are getting these errors frequently: The cause of this output exception was that: ClientAbortException: java.net.SocketException: Connection reset by peer: socket write error. Error message: The cause of this output exception was that: ClientAbortException: java.net.SocketException: Connection reset by peer: socket write error. Root cause: coldfusion.tagext.OutputException: The cause of this output exception was that: ClientAbortException: java.net.SocketException: Connection reset by peer: socket write error. Tag context: Exception type: coldfusion.runtime.CfErrorWrapper

Comment by External U.

Since the pool crash issue has been resolved, marking this bug as ToTest and opening another one for the "connection reset" error in the logs - CF-4046444

Comment by Rupesh K.

Closing this bug since the Application Pool crash has been resolved. The issue with "connection reset" is being tracked as part of another bug, as mentioned below.

Comment by Immanuel N.