tracker issue : CF-3318104

select a category, or use search below
(searches all categories and all time range)
Title:

Service Temporarily Unavailable after arbitrary amount of time + heavy usage

| View in Tracker

Status/Resolution/Reason: Closed/Fixed/

Reporter/Name(from Bugbase): Samuel Knowlton / Samuel Knowlton (Samuel Knowlton)

Created: 08/26/2012

Components: Web Container (Tomcat)

Versions: 10.0

Failure Type: Crash

Found In Build/Fixed In Build: Final / Latest

Priority/Frequency: Critical / Some users will encounter

Locale/System: English / Win 2008 Server R2 64 bit

Vote Count: 41

Problem Description: 

The problem is described in detail in this thread:

http://forums.adobe.com/message/4649986#4649986

CF 10 under Windows 2008 x64 and IIS7 will periodically start serving up the following messages:

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

Jakarta/ISAPI/isapi_redirector/1.2.32 ()

Restarting the web publishing service causes the problem to go away. 

Steps to Reproduce: Cannot reproduce on demand. Happens every 1-4 weeks on a server with 14 or so CF applications and several thousand active users.

Actual Result:

Expected Result:

Any Workarounds:

----------------------------- Additional Watson Details -----------------------------

Watson Bug ID:	3318104

External Customer Info:
External Company:  
External Customer Name: Aquitaine
External Customer Email:  
External Test Config: My Hardware and Environment details: Win 2008 Server x64 (Web or Enterprise edition), 8 GB RAM, CF10 Standard

Attachments:

  1. September 05, 2012 00:00:00: 1_CF10Error.jpg

Comments:

We have 5 server with CF10 Enterprise that exhibit this same behavior. Within 2 hours of heaver traffic, a buffer overflow error is registered in the isapi_redirect.dll
Vote by External U.
18324 | August 29, 2012 12:29:52 PM GMT
This is happening to me as well, under a small testing load. It's holding up a large state project from going into production.
Vote by External U.
18325 | August 29, 2012 12:49:44 PM GMT
This happens to me when I have a http request with a large paylod in the content, for instance doing a HTML5 XMLHTTPRequest with a file in the request.
Comment by External U.
18289 | August 29, 2012 12:53:04 PM GMT
We were live on Coldfusion 10 Standard Edition on 2 Windows Server 2008 VM (32 bit) for almost a week. Last night, both servers stopped responding at the same time. When a server was restarted, it would run ok initially on restart but then would lock up within the next 10 minutes. IIS logs show increased traffic from bots during that time. We had to revert back to Coldfusion 9.
Vote by External U.
18326 | August 29, 2012 12:55:45 PM GMT
coldfusion-error.log fills up with ~5mb of these entries on each occurance: Aug 29, 2012 11:24:58 AM org.apache.coyote.ajp.AjpProcessor getRealPathFromServer SEVERE: Error in getRealPathFromConn java.net.SocketException: Connection reset by peer: socket write error at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:124) at org.apache.coyote.ajp.AjpProcessor.getRealPathFromServer(AjpProcessor.java:391) at org.apache.coyote.ajp.AbstractAjpProcessor.getRealPath(AbstractAjpProcessor.java:1112) at org.apache.naming.resources.FileDirContext.doGetRealPath(FileDirContext.java:198) at org.apache.naming.resources.BaseDirContext.getRealPath(BaseDirContext.java:412)
Comment by External U.
18290 | August 29, 2012 01:13:40 PM GMT
breaks when calling flash forms which perform data loads from components on init
Vote by External U.
18327 | August 29, 2012 01:15:15 PM GMT
We are seeing server hangs on CF10 that may be related to this.
Vote by External U.
18328 | August 29, 2012 02:08:33 PM GMT
I'm not having this problem (I don't run Win2k8), but I've heard enough of it happening that it seriously needs looking at. -- Adam
Vote by External U.
18329 | August 29, 2012 02:54:35 PM GMT
I, like Adam, am not personally having this issue because I have not upgraded to CF10 yet (still running CF9) but I am hearing lots of other people with similar issues. For what it's worth here is another forum discussion regarding this: http://forums.adobe.com/message/4657976#4657976
Vote by External U.
18330 | August 29, 2012 03:08:43 PM GMT
We are trying to reproduce this issue. Any Sample code that reproduces the issue will be really helpful. We can simulate continuous http request firing to replicate the issue. (Comment added from ex-user id:evelinv)
Comment by Adobe D.
18291 | August 30, 2012 12:06:31 AM GMT
Same problem at Hostek with CF10.
Vote by External U.
18331 | August 30, 2012 07:59:18 PM GMT
We also need some information on server setup. 1. How many sites in the IIS are configured with CF 2. Have you tried giving the timeout value in wsconfig->advanced option. 3. Enable the cachewebserverpath option in the server admin settings page and verify if you can repro the issue. 4. Give us server load information about number of request at normal and peak time.
Comment by Kiran S.
18292 | August 31, 2012 06:04:51 AM GMT
Had this and other problems. I had to revert to CF9
Vote by External U.
18332 | September 04, 2012 08:18:28 AM GMT
Happening on our Production Server.
Vote by External U.
18333 | September 05, 2012 09:38:00 AM GMT
I, like Adam and Miguel, am not personally having this issue (yet) because I am still in the process of configuring our new CF10 server (Win2K8 R2) and don't have any users hitting it in production yet, But I'm very concerned that I may encounter the same issues.
Vote by External U.
18334 | September 07, 2012 03:26:58 PM GMT
After upgrading form CF 9.1 Standard to CF 10 we have seen the same issues.
Vote by External U.
18335 | September 07, 2012 03:48:39 PM GMT
My low traffic server has this issue as well. Maybe once a month or more? I have no known trigger as of yet.
Vote by External U.
18336 | September 14, 2012 11:04:10 AM GMT
We have this problem also. This is terrible. CF10 has been so buggy. We've had to make a program that continuously checks if the server's up and restarts the CF10 Application Service.
Vote by External U.
18337 | September 18, 2012 10:36:18 AM GMT
Still having this problem in CF10 Update 2
Comment by External U.
18293 | September 18, 2012 10:36:50 AM GMT
Also, I would like to say, this happens several times a week for us.
Comment by External U.
18294 | September 18, 2012 10:38:00 AM GMT
Also receiving this error. Look at the number of votes on this issue! CF10 has been an utter disaster for us so far.
Comment by External U.
18295 | September 18, 2012 10:57:47 AM GMT
Also experiencing the issue. Where is Adobe's commentary on this?
Vote by External U.
18338 | September 18, 2012 10:58:11 AM GMT
This is happening on our CF10 production server as well.
Vote by External U.
18339 | September 20, 2012 07:27:50 AM GMT
I am having the same thing with CF10 Update 2. Installed it yesterday and failing already today! The error messages I see are identical to those shown in this bug fix. Can't discern exactly what users were doing at the time, but only resolution was to restart CF service. PLEASE fix this, as it is causing real issues on our production server. Our customers are VERY angry. Perhaps these links will help: http://forums.adobe.com/message/4465125 http://forums.adobe.com/message/4619002#4619002
Vote by External U.
18340 | October 01, 2012 04:01:28 PM GMT
I am having the same thing with CF10 Update 2. Installed it yesterday and failing already today! The error messages I see are identical to those shown in this bug fix. Can't discern exactly what users were doing at the time, but only resolution was to restart CF service. PLEASE fix this, as it is causing real issues on our production server. Our customers are VERY angry. Perhaps these links will help: http://forums.adobe.com/message/4465125 http://forums.adobe.com/message/4619002#4619002
Comment by External U.
18296 | October 01, 2012 04:02:12 PM GMT
I believe we are experiencing this issue as well which was causing a production server to crash with the symptoms reported by others every hour or so at times.
Vote by External U.
18341 | October 05, 2012 05:38:44 AM GMT
This is a very common platform Adobe must fix this ASAP or offer a refund - since CF10 is useless if this unstable
Comment by External U.
18297 | October 06, 2012 10:06:47 AM GMT
Having the same issues here - I have update 2 installed and get this error 3 or 4 times per day. Th server is a dev server and barely used. We cannot move to production with this error occurring
Vote by External U.
18342 | October 08, 2012 01:07:26 PM GMT
Thi fix for this issue will be available in the next update which we plan to release very soon.
Comment by Vamseekrishna N.
18298 | October 08, 2012 10:34:05 PM GMT
We're unable to use web service and it brings the website down.
Vote by External U.
18343 | October 26, 2012 12:25:19 PM GMT
I have been working on this since July. I purchase 3 CF10 licenses (1)enterprise and (2)standards and invested in a new server, all to no avail. I have already spent $1500 in support and am receiving heat about the cost and have had to revert back to CF9. This should be a high priority for Adobe! Updates 1 & 2 fixes some others issues with standard but when will this update be available?
Vote by External U.
18344 | October 29, 2012 05:08:52 PM GMT
seeing this after migrating prod servers to cf10
Vote by External U.
18345 | October 30, 2012 09:07:35 AM GMT
I don't understand -WHY- this issue hasn't been fixed yet. It's bringing down my Fortune 200 client's server multiple times per day. Can Adobe AT LEAST provide an ETA on a fix for these connector issues!?
Comment by External U.
18299 | October 31, 2012 08:02:02 AM GMT
Fix this bug immediately! It is absolutely KILLING any credibility in ColdFusion 10 as a stable enterprise platform. My Fortune 200 client's server is crashing multiple times per day as a result of this issue.
Vote by External U.
18346 | October 31, 2012 08:11:08 AM GMT
Same exact issue for me with one of my biggest clients. Constant crashes on our ColdFusion 10 server multiple times per day. Event viewer is riddled with constant errors (thousands per day) from the w3wp.exe service and the ISAPI connector. I was able to seemingly relieve the situation somewhat by disabling IIS 7.5's "Rapid-Fail Protection" on the application pools (which I can't imagine is advisable for a production environment), but that only seems to have decreased the frequency of the crashes slightly - from a dozen times each day, down to only a few times per day. Completely unacceptable. It absolutely astounds me that Adobe has not halted in their tracks and prioritized every single reputatble development resource with developing and releasing an effective resolution to this problem. Honestly, it gives me cause (for the first time in 17 years of ColdFusion development and support) to consider migrating away from ColdFusion to a different platform. Updater 3 was apparently a disaster (thank God I checked for community feedback before installing it), but I've not heard a single follow up about the fix to this problem since the Updater 3 was recalled by Adobe - and that was a couple weeks ago. How on earth can an absolutely critical bug like this remain open and unaddressed for so long without any resolution!? Adobe, you've -completely- dropped the ball on this and disappointed droves upon droves of loyal customers, developers, and community members. Our clients are angry. Not only are they losing money and credibility every single day that this bug remains unfixed, but their confidence and support of ColdFusion as a platform is dwindling. What in the world is holding up the completion and release of a prioritized hot fix for this issue!? Forget about bundling it up with a bunch of other fixes at this point. This bug is critical enough to stand on its own and be addressed immediately! Unbelievable.
Comment by External U.
18300 | October 31, 2012 08:36:07 AM GMT
We are most likely releasing the fix by end of this week. We needed time to do additional testing plus involve community via private release to test the fix. Thanks for your support and feedback. (Comment added from ex-user id:hkhandel)
Comment by Adobe D.
18301 | October 31, 2012 12:01:38 PM GMT
If this isn't released by tomorrow, I will need someone from Adobe to send me the fix that is ebing used by testers. I have numerous CF 10 servers that I'm on the verge of migrating to and have one if Production that has this issue that is highly visible and need immediate resolve.
Vote by External U.
18347 | November 01, 2012 05:37:28 PM GMT
I have the same issue please help me to solve this because i need the server online for tomorrow and i have 2 weeks looking for an answer.... Thanks
Comment by External U.
18302 | November 15, 2012 08:26:31 AM GMT
having the exact same problem.. cf9 and cf10.. with cf10 happens less often.. problem solves afer restaring the cf service The service is unavailable. Service Temporary Unavailable! The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later. Jakarta/ISAPI/isapi_redirector/1.2.32 () in the logs I am getting Dec 2, 2012 3:33:04 AM org.apache.catalina.core.StandardWrapperValve invoke SEVERE: Servlet.service() for servlet [CfmServlet] in context with path [/] threw exception [ROOT CAUSE: java.lang.OutOfMemoryError: Java heap space ] with root cause javax.servlet.ServletException: ROOT CAUSE: java.lang.OutOfMemoryError: Java heap space for that I am increasing the setting Maximum JVM Heap Size WITH OUT ANY solution
Comment by External U.
18303 | December 03, 2012 09:26:29 AM GMT
Still experiencing this issue. Service crashes 5-10 times per day, primarily during business hours when traffic and usage is higher.
Vote by External U.
18348 | January 11, 2013 09:45:07 AM GMT
Exact server here. Only have this with CF10 - have installed all updates. What I notice is this only happens on sites that have several bad page requests. For example if the client is new to us and has some links out there to non-existent pages, then that site will crash several times a day after a few of these bad GET page requests are made. Here is a recent log entry that killed the site: 2013-02-19 16:06:41 W3SVC26 FULLPARTNER5 76.12.79.46 GET /wp-login.php - 91.207.7.238 Mozilla/5.0+(Windows;+U;+Windows+NT+5.1;+en-US;+rv:1.9.1.3)+Gecko/20090824+Firefox/3.5.3+GTB5 - - 200 0 235 93 Notice the page is /wp-login.php,obviously a page from the previous site that does not exist on the new site. We have only used CF10 for about 3 weeks. No such problem in CF9 or CF8 on these exact same sites. Clearly a CF10 issue, clearly a jakarta issue. I shudder to think what will happen when the PCI guys hit a site for a vulnerability scan - the always try to hit a hundred bad pages to see what the server does. Probably the server will crash! So far we have not had a PCI scan on this server....
Comment by External U.
18304 | February 19, 2013 11:22:52 AM GMT
Regret the upgrade to version 10 - please get a fix for this ASAP!!
Vote by External U.
18349 | February 25, 2013 11:54:56 AM GMT
had same problem, reverted to cf9. please fix asap.
Vote by External U.
18350 | March 20, 2013 11:37:14 AM GMT
This was happening to us apparently with large number of website hits, but not anymore. NOW it happens when PCI scans our servers for compliance. It was happening every month, now about every other month. It just happened this morning on 4/23/13, so the bug hasn't been entirely fixed.
Comment by External U.
18305 | April 23, 2013 07:46:14 AM GMT
It happens to me twice a week, completely unpredictable. Does anyone know if this bug also affects CF10 running on Apache? I'm considering disabling IIS and setting Apache instead. Thx.
Comment by External U.
18306 | April 25, 2013 10:59:47 AM GMT
Can you please let us know a bit more information about your setup - like connector pool configuration along with Timeout values,OS resource usage, handle usage, CF admin details etc so that we can try and repro the issue on our end .Thanks.
Comment by Asha K.
18307 | April 26, 2013 04:04:00 AM GMT
Thanks. Server details: VM Ware cloud. Windows server 2008 R2 web incl. all updates. IIS – one active site. Xeon CPU 2650 @1.00Ghz (2 processors), 6 GB Ram. Coldfusion 10 Standard - 10,0,9,284568 all latest updates. Max simultaneous template requests set to 10 (changed a few times – no affect). Request timeout: 60 Server load – very low, 30 users, 0-50% CPU, 1.3 – 2 GB RAM. Web server connector buffer was disabled (because using cfflush) in an attempt to solve problem – no change. 2 weeks ago completely reinstalled IIS role + CF10 + updates, still throwing the error. Any idea what to do next?
Comment by External U.
18308 | April 29, 2013 10:29:13 PM GMT
Running into this problem with CF 10u9 on Windows Server 2008 R2 64-bit. I had to run this to get it working (DOS Prompt elevated to Administrator security level): {cfusion-install}\cfusion\bin\connectors\Upgrade_all_connectors.bat
Vote by External U.
18351 | May 11, 2013 12:54:30 AM GMT
Can you please try changing your settings as suggested in http://www.webtrenches.com/post.cfm/resolve-stability-problems-and-speed-up-coldfusion-10
Comment by Asha K.
18309 | May 27, 2013 05:38:21 AM GMT
We are hitting this problem as well.
Vote by External U.
18352 | May 28, 2013 09:30:47 AM GMT
Using the method described by Asha K S did not work for us. A more detailed explanation of our problem can be found here: https://bugbase.adobe.com/index.cfm?event=bug&id=CF-3530880 It may be related to the problems mentioned here by others as well.
Comment by External U.
18310 | May 29, 2013 08:47:24 AM GMT
We were down for thirty minutes today because of this. Lost lots of money.
Vote by External U.
18353 | June 12, 2013 11:54:46 AM GMT
HI ParaEl Can you please email your contact mail id to asha@adobe.com so that we can contact you to resolve this issue. Thanks.
Comment by Asha K.
18311 | June 13, 2013 09:41:49 AM GMT
What is the status of this bug? It appears to still exist as of Update 11.
Comment by External U.
18312 | July 09, 2013 12:00:38 PM GMT
Please fix this bug. It is costing us much money.
Vote by External U.
18354 | July 12, 2013 07:32:59 AM GMT
Please fix this bug! What is the status?
Comment by External U.
18313 | July 12, 2013 07:33:40 AM GMT
We just launched a few CF10 production servers. We hope we won't run into this. This is marked Fixed, but when will this fix be available?
Vote by External U.
18355 | July 12, 2013 04:32:27 PM GMT
Agree that Adobe should fix this but did anyone try the BonCode connector instead of the ISAPI to see whether that resolves the problem?
Comment by External U.
18314 | July 12, 2013 08:31:24 PM GMT
Made Changes in the following and the server is stable. worker.properties worker.cfusion.connection_pool_size = 500 worker.cfusion.connection_pool_timeout = 60 server.xml maxThreads=500 and connectionTimeout="60000" Followed blog : http://blogs.coldfusion.com/post.cfm/tuning-coldfusion-10-iis-connector-configuration (Comment added from ex-user id:vijaiswa)
Comment by Adobe D.
18315 | July 17, 2013 08:12:56 AM GMT
We have also experienced this random outage with the same isapi redirect error.
Vote by External U.
18356 | July 18, 2013 07:03:42 AM GMT
We get this all the time. Some times if we leave the server long enough it will come back on its own. Other times, the site with the heaviest usage will stay running but all other sites will go down. Weird.
Vote by External U.
18357 | July 20, 2013 06:30:35 PM GMT
We have been having this problem too. Please fix this issue. Thanks.
Comment by External U.
18316 | August 12, 2013 01:44:07 PM GMT
We have this problem too. We are going to go out of business if we do not get this resolved!
Comment by External U.
18317 | August 13, 2013 08:54:06 PM GMT
Our server goes down multiple times per day - with unreliable servers, we are losing clients on a daily basis. We are going to go out of business!
Vote by External U.
18358 | August 13, 2013 08:55:05 PM GMT
**** Please check the comment below by Vishal Jaswal ***** You need to tune the server to address this. We have worked with quite a few customers and confirmed that tuning the server as per the following blog entry actually resolves the problem. http://blogs.coldfusion.com/post.cfm/tuning-coldfusion-10-iis-connector-configuration The same blog entry should be applicable for Apache as well. Does this not resolve your problem?
Comment by Rupesh K.
18318 | August 14, 2013 12:17:16 AM GMT
I'm seeing this about 3 or four times a week on Windows 2008 server running the latest patched version of CF10 (update 11 I believe). It's driving me crazy and proving to be a very frustrating time as I try and work out what is actually going on.
Vote by External U.
18359 | August 20, 2013 05:10:38 AM GMT
This is happening with us also. We just upgraded to cf10 a couple of months ago. It is costing us $$$$. I have followed the advice for tuning the connector but problem continues.
Vote by External U.
18360 | September 30, 2013 09:21:15 AM GMT
Tried tuning like in the blog Windows server 2008, IIS 7.5 Coldfusion 10 patched to patch level 11 http://blogs.coldfusion.com/post.cfm/tuning-coldfusion-10-iis-connector-configuration did not resolve our problem
Comment by External U.
18319 | October 01, 2013 01:09:33 PM GMT
We are running the same exact configuration as noted above except we have 16 GB of RAM on a virtual server. We upgraded to ColdFusion 10 because our our last web server (ColdFusion 9) was compromised. I installed all Windows updates and have CF 10 up to Update 11. The problem is frustrating because the only way I know how to resolve it it to restart the IIS Service. Hopefully this will be addressed soon as I am losing too much sleep over this issue.
Comment by External U.
18320 | October 08, 2013 09:17:02 AM GMT
************ Please read this ******************* If configuring your server as per http://blogs.coldfusion.com/post.cfm/tuning-coldfusion-10-iis-connector-configuration does not solve your problem, could you please get in touch with us at "cf.install AT adobe DOT com" and we would work with you to resolve this issue.
Comment by Rupesh K.
18321 | November 12, 2013 01:38:43 PM GMT
I am having the exact same problems despite performing all the suggested connector tuning. I have the latest CF10 Update 11 running, clean install, medium sized site, and yet continue to see 503 error messages a few times a week. Windows 2008 R2 (VMware), IIS 7.5. Never, ever had these problems with CF9 on 32-bit Windows 2003/IIS6. Thinking about falling back to CF9 but would really prefer for CF10 to work given all the time and $$$ spent. Will try to get in touch with Adobe privately, something I've never had to do in 16 years of working with CF. Bummer.
Vote by External U.
18361 | November 13, 2013 01:10:57 PM GMT
I just got this for the first time today. About 1500 hours of cpu time without reboot/restart. Our configuration is CF10 with Hotfix level 10. This effected our scheduled jobs which are critical. CF10 64-bit on Win2008with 24 gb ram. The reputation of ColdFusion is on the line in this 6+ enterprise server licensed organization.
Vote by External U.
18362 | November 18, 2013 09:14:53 AM GMT
We have to restart the service almost daily due to this error. Running CF10 with latest updates (10,0,13,287689). WinServer 2008 R2, IIS 7.5.
Vote by External U.
18363 | May 05, 2014 10:47:26 AM GMT
We are having similar issues, though the problem doesn't seem to have any relation to load. I'm implementing the changes outlined here: http://blogs.coldfusion.com/post.cfm/tuning-coldfusion-10-iis-connector-configuration Will report back if this seems to help. We are averaging at least one crash per day. Sometimes it will be fine for several days, then it will go down multiple times in a 24 hour period.
Comment by External U.
18322 | July 09, 2014 02:13:44 PM GMT
Our site is going down at least once daily on average. Doesn't seem to be tied to load, but otherwise fits the descriptions on this page.
Vote by External U.
18364 | July 09, 2014 02:27:41 PM GMT
This still happens on CF10 update 15 (1.2.40 connector). For us, someone posts a massive POST command to the server, then there is an error 500 in the logs. Tuning does nothing to solve this issue. The only consolation is that the server does not crash, but there is no way to stop the error from occurring still. Even if we detect POST data and stop it in in CFML code, the error still persists. I reckon that a user could take a site offline with repeated POSTs of this nature.
Comment by External U.
18323 | December 16, 2014 04:15:34 AM GMT