tracker issue : CF-3530880

select a category, or use search below
(searches all categories and all time range)
Title:

App Pool Crashing "Randomly". "Load balancing workers will not function properly." in isapi_redirct.log at crash time.

| View in Tracker

Status/Resolution/Reason: Closed/Fixed/

Reporter/Name(from Bugbase): Lee Bartelme / Lee Bartelme (Lee Bartelme)

Created: 03/27/2013

Components: Installation/Config, Connector

Versions: 10.0

Failure Type: Crash

Found In Build/Fixed In Build: Final /

Priority/Frequency: Critical / All users will encounter

Locale/System: English / Win 2008 Server R2 64 bit

Vote Count: 7

Problem Description: IIS terminates Application pool due to a series of failures in the process serving that pool. Coldfusion logs show no issues but the isapi_redirect.log has:
Initializing shm:(null) errno=-1. Load balancing workers will not function properly.
found a lot around crash time.

See attached isapi log. Latest Crash started on March 27 at 5:47 and finally killed the pool at 6:06

Steps to Reproduce: Can't reproduce at will.

Actual Result: Application Pool Stops

Expected Result: Application Pool Doesn't stop

Any Workarounds: Attempted manually changing items in workers.properties, server.xml and jvm.config

workers.properties Added:
worker.cfusion.max_reuse_connections=250
worker.cfusion.connection_pool_size = 500
worker.cfusion.connection_pool_timeout = 60
worker.cfusion.connection_pool_minsize = 251

server.xml Changed: (To Match Workers.Properties)
<Connector port="8012" protocol="AJP/1.3" redirectPort="8445" tomcatAuthentication="false" maxThreads="500"  minSpareThreads="251" connectionTimeout ="60000" />

jvm.config Changed: (Increased Memory, and Max Perm Size
java.args=-server  -Xms512m -Xmx2048m -XX:PermSize=192m -XX:MaxPermSize=324m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home={application.home} -D

----------------------------- Additional Watson Details -----------------------------

Watson Bug ID:	3530880

External Customer Info:
External Company:  
External Customer Name: Lee Bartelme
External Customer Email:  
External Test Config: My Hardware and Environment details:

CF10

IIS 7.5

Win Server 2008

Multiple Websites Connector set to ALL

Attachments:

  1. March 27, 2013 00:00:00: 1_isapi_redirect.zip
  2. May 29, 2013 00:00:00: 2_ParaEl-isapi_redirect-2013-05-29.txt
  3. August 08, 2014 00:00:00: 3_isapi_redirect.zip

Comments:

Repeatably affects two of our test servers. We haven't deployed CF10 to production yet, and until we find out what causes this I'm reluctant to do so.
Vote by External U.
15893 | May 28, 2013 09:08:43 AM GMT
We are hitting this problem as well.
Vote by External U.
15894 | May 28, 2013 09:22:34 AM GMT
We are hitting this problem as well, my log is attached.
Comment by External U.
15883 | May 29, 2013 08:43:55 AM GMT
We are experiencing the same issue. The site works fine running under one app pool, but as soon as a second app pool is created, this issue occurs.
Vote by External U.
15895 | June 19, 2013 10:19:32 AM GMT
While we have been fortunate with CF 10 in production, the description of the scenario makes me quite nervous.
Vote by External U.
15896 | June 19, 2013 07:14:31 PM GMT
This should have been fixed with tomcat upgrade . Tomcat has fixed this issue.
Comment by Asha K.
15884 | December 05, 2013 03:32:56 AM GMT
@Asha K S Is the Tomcat upgrade part of a CF10 patch? If so which patch?
Comment by External U.
15885 | December 05, 2013 10:51:08 AM GMT
@TheCycoONE it is not part of any CF 10 released patch yet.
Comment by Asha K.
15886 | January 25, 2014 10:07:04 AM GMT
This issue is impacting our new CF10 patch 13 development servers. I have been unable to keep the development systems stable even with just 1 user accessing it.
Vote by External U.
15897 | January 27, 2014 01:46:06 PM GMT
I just built a fresh Hyper-V Windows 2008 R2 IIS 7.5 with ColdFusion 10 patch 13 and the isapi connectors are all freshly built as the newest version. I have it running on the latest Oracle Java JRE 1.7.0_51 as well. I currently have 2 users accessing this system and the IIS app pool associated with the ColdFusion portion keeps shutting down randomly. It did it 3 times today but yesterday not at all. This is a development system so the load is very small. I've also tried the connector tuning values above to no avail. I've also run through the CF10 hardening guide on this system.
Comment by External U.
15887 | January 27, 2014 02:45:45 PM GMT
Our issue got resolved a while back. 2 things, one thing that seemed to be in common among people with errors include our case was on the same server we had a asp.net application running under the same website, albeit a different pool. This is extensively reported here: http://forums.adobe.com/message/5416168 Our fix was a combination of installing Java separate from CF and explicitly doing all bindings on the server in IIS.
Comment by External U.
15888 | January 27, 2014 03:19:15 PM GMT
@Lee hmmm we have a mixed environment too. The main portion of the website runs DotNetNuke in .NET4 and then CF10 is running as a virtual folder within the website on its own app pool. We have the opposite as well where the main website is CF and the virtual folders run .net code in their own app pools. There is another website on this server that runs CF and is not crashing, it does not have multiple app pools or .NET running in it. We already have Java as its own installation outside of the CF10 installed one as the environment we run within requires any instance of Java on a machine to be kept up to date. However that same environment prevents us from configuring CF on all websites and can only be on the specific websites it needs to run on.
Comment by External U.
15889 | January 28, 2014 09:06:04 AM GMT
Let us know if this issue still exists with the latest Splendor build.
Comment by Immanuel N.
15890 | February 03, 2014 05:19:01 AM GMT
We're experiencing this as well. Random shutdowns. In production. It's really frustrating that Adobe hasn't addressed this yet. Please fix. Please.
Vote by External U.
15898 | July 21, 2014 09:53:31 AM GMT
Anyone get an official response from Adobe? Also experiencing the same problem on a production 2008 R2 IIS server CF10 update 13.
Vote by External U.
15899 | August 07, 2014 02:41:18 PM GMT
@brentil Can you please confirm if the issue still exists with CF 10 HF 14? This update had Tomcat & Tomcat connector upgrades, and the issue must have been addressed.
Comment by Immanuel N.
15891 | December 03, 2014 04:05:10 AM GMT
Will be closing the bug since all open IIS application pool crash related issues were fixed in HF14. Will reopen the bug if this issue still persists for the customer.
Comment by Immanuel N.
15892 | December 09, 2014 03:57:41 AM GMT