tracker issue : CF-3980425

select a category, or use search below
(searches all categories and all time range)
Title:

CF11 Hotfix 5 Causing 503 Errors

| View in Tracker

Status/Resolution/Reason: Closed/Withdrawn/NotABug

Reporter/Name(from Bugbase): John Lang / John Lang (John Lang)

Created: 05/02/2015

Components: Installation/Config, Connector

Versions: 11.0

Failure Type: Crash

Found In Build/Fixed In Build: CF11_Final /

Priority/Frequency: Critical / All users will encounter

Locale/System: English / Win 2008 Server x64

Vote Count: 0

Problem Description: Since installing Hotfix 5 on our CF11 servers, after a period of time we get 503 errors from IIS. This did not happen with our prior install of Hotfix 3 (we never installed hotfix 4). The problem seems to happen on our high traffic applications, at least once per day, sometimes more. The CF Administrator is working fine the entire time, so I believe it may be a problem with the IIS connector. I did note that the description of the update includes a note about fixing a bug in the connector for IIS. This is happening on both our Windows 2008 and Windows 2012 servers.

Steps to Reproduce: Unable to reproduce on demand. This has been happening for several weeks since the install and I believe it to be traffic related since our lower traffic sites haven't had a problem.

Actual Result: IIS returns 503 errors

Expected Result: Working website!

Any Workarounds: Sometimes restarting IIS alone fixes, other time restarting the CF service fixes, other times restarting both is required. Occasionally coworkers have had to restart entire machines.

----------------------------- Additional Watson Details -----------------------------

Watson Bug ID:	3980425

External Customer Info:
External Company:  
External Customer Name: John
External Customer Email:  
External Test Config: My Hardware and Environment details: Windows 2008/2012. 64 bit. Enterprise Edition CF. 2 GB RAM allocated to ColdFusion. Running 4 instances per machine. All instances have experienced the issue, but it is most frequent with our high traffic instances.

Attachments:

Comments:

Have rolled back to Update 3 since this is making our services unstable.
Comment by External U.
7661 | May 02, 2015 07:30:26 AM GMT
I forgot to mention that after installing Hotfix 5 I did remove the connectors and then re-add them as instructed by the update. I've also tried doing that again since we experienced issues, but the issues don't stop.
Comment by External U.
7662 | May 02, 2015 07:37:36 AM GMT
Hi John, It seems like a configuration mismatch issue. Can you verify that, the content of workers.properties at \ColdFusion11\config\wsconfig\backup\[Magic Folder].[Backup Folder]\ are similar as workers.properties at \ColdFusion11\config\wsconfig\[Magic Folder]\ ?
Comment by Anit K.
7663 | May 04, 2015 08:58:27 AM GMT
Hi Anit, We've never made any tweaks when running wsconfig and have always just picked the appropriate IIS site and selected submit. I just double checked with some backups we had, and both pre & post update look as follows.. worker.list=cfusion worker.cfusion.type=ajp13 worker.cfusion.host=localhost worker.cfusion.port=8014 worker.cfusion.max_reuse_connections=250 All of our properties files are identical to this value wise, other than the instance/port names changing. In trying to figure out what was going on, I did run into an old blog post about tweaking the max_reuse_connections & connection_pool_timeout from their defaults. However, we've never done this before so if that is the issue, it's just odd that it would only effect us with Hotfix 5 and not Hotfix 3. :( Open to any other ideas you may have.
Comment by External U.
7664 | May 04, 2015 09:27:56 AM GMT
Thanks for confirming John. Here is what you need to do, then. Refer to my blog post http://blogs.coldfusion.com/post.cfm/coldfusion-11-iis-connector-tuning. Add the below entry in workers.properties at \ColdFusion11\config\wsconfig\[Magic Folder]\ worker.cfusion.connection_pool_timeout=60 And in the server.xml at {cf-home/cfusion/runtime/conf} add, <Connector port="8012" protocol="AJP/1.3" redirectPort="8445" tomcatAuthentication="false" maxThreads="500" connectionTimeout ="60000"> </Connector> Note:- Any changes made to {CF-Home}/config/wsconfig/{some no}/, including isapi_redirect.dll or worker.properties, would require an IIS restart.
Comment by Anit K.
7665 | May 04, 2015 09:36:36 AM GMT
Thanks Anit. I'll reinstall the update on one of our servers today & try this. Will give it a few days of monitoring to see if the issue pops up again before trying the others. Is there any reason the issue didn't pop up until this particular patch?
Comment by External U.
7666 | May 04, 2015 10:15:38 AM GMT
Not sure, what might have caused. We will investigate further. IMO, connection_pool_timeout followed by other attributes are important from CF connector tuning perspective. It is a must, to tune the connector for production sites running on CF10/11.
Comment by Anit K.
7667 | May 04, 2015 10:51:49 AM GMT
John, Just checking if the tweak suggested by Anit worked for you?
Comment by Immanuel N.
7668 | June 09, 2015 01:32:50 AM GMT
Immanuel, Yes, that tweak has helped quite a bit along with us finding some code that was stalling and just causing hung threads on the server to build up. We still get the problem occasionally but it's a lot less frequent than before. I appreciate the follow up. - John
Comment by External U.
7669 | June 09, 2015 07:10:41 AM GMT
Thanks for the update John. I believe this problem would be completely eliminated once the connectors are aptly tuned for your environment. Would it be OK to close this bug? We could always reopen it / raise a new bug if the issue persists.
Comment by Immanuel N.
7670 | June 09, 2015 09:32:40 PM GMT
Yup, you can go ahead and close the bug report. Thanks again for the quick support/assistance in working through this issue!
Comment by External U.
7671 | June 10, 2015 06:32:16 AM GMT
Thanks John. Glad it helped! Will be closing the bug as withdrawn.
Comment by Immanuel N.
7672 | June 10, 2015 09:47:40 PM GMT