tracker issue : CF-4139772

select a category, or use search below
(searches all categories and all time range)
Title:

IIS App Pool Crashes

| View in Tracker

Status/Resolution/Reason: Closed/Withdrawn/UserError

Reporter/Name(from Bugbase): Ashok Ramkumar / Ashok Ramkumar (Ashok kumar Ramkumar)

Created: 04/13/2016

Components: Installation/Config, Connector

Versions: 11.0

Failure Type: Crash

Found In Build/Fixed In Build: CF11_Final /

Priority/Frequency: Critical / All users will encounter

Locale/System: English / Win 2012 Server x64

Vote Count: 0

Problem Description: IIS 8.5 Pool Crash after migrating all CF8 applications to CF11.

Error in the windows logs - application:
Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x5215df96
Faulting module name: iisutil.dll, version: 8.5.9600.17415, time stamp: 0x54504119
Exception code: 0xc0000005
Fault offset: 0x0000000000003a83
Faulting process id: 0x12a0
Faulting application start time: 0x01d195b7c9c993c5
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: c:\windows\system32\inetsrv\iisutil.dll
Report Id: 07aa94a8-01ab-11e6-8107-005056b02d54
Faulting package full name: 
Faulting package-relative application ID: 


Steps to Reproduce: application is coded with CFM, CFCs and seperate content server folder mapped into IIS.

Actual Result: Http error 503 Service unavailable

Expected Result: run smooth

Any Workarounds:

----------------------------- Additional Watson Details -----------------------------

Watson Bug ID:	4139772

External Customer Info:
External Company:  
External Customer Name: Ashok kumar Ramkumar
External Customer Email:  
External Test Config: My Hardware and Environment details:

Win 2012 R2, 12 GB Ram and 2.2X2 Intel processor

Attachments:

  1. April 14, 2016 00:00:00: 1_CrashHang_Report__Date_04_13_2016__Time_03_25_48PM__19.txt
  2. April 14, 2016 00:00:00: 2_CrashHang_Report__Date_04_13_2016__Time_03_53_13PM__871.txt
  3. April 14, 2016 00:00:00: 3_CrashHang_Report__Date_04_13_2016__Time_04_27_19PM__349.txt
  4. April 14, 2016 00:00:00: 4_CrashHang_Report__Date_04_14_2016__Time_09_44_01AM__531.txt
  5. April 15, 2016 00:00:00: 5_app_pool_settings.jpg
  6. April 15, 2016 00:00:00: 6_serviceaccnts.jpg
  7. April 15, 2016 00:00:00: 7_iislogs.jpg

Comments:

Error in the windows logs - application: Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x5215df96 Faulting module name: iisutil.dll, version: 8.5.9600.17415, time stamp: 0x54504119 Exception code: 0xc0000005 Fault offset: 0x0000000000003a83 Faulting process id: 0x12a0 Faulting application start time: 0x01d195b7c9c993c5 Faulting application path: c:\windows\system32\inetsrv\w3wp.exe Faulting module path: c:\windows\system32\inetsrv\iisutil.dll Report Id: 07aa94a8-01ab-11e6-8107-005056b02d54 Faulting package full name: Faulting package-relative application ID:
Comment by External U.
3082 | April 13, 2016 01:56:08 PM GMT
Ashok, Does it crash even if you call a simple test CFML page, or does it happen only when a certain page in you CFML application is accessed? If it is the second case, does it happen consistently if you hit the page, or only under certain conditions such as moderate or high load. Can you perhaps share the advanced settings for the application pool servicing your CF site and collect IIS crash dump. You can take reference the following articles that I found after a cursory look up: https://blogs.msdn.microsoft.com/webtopics/2009/11/25/how-to-collect-a-crash-dump-of-an-iis-worker-process-on-iis-7-0-and-above/ https://blogs.msdn.microsoft.com/chaun/2013/11/12/steps-to-catch-a-simple-crash-dump-of-a-crashing-process/ Kindly share the worker.properties file from the connector folder as well: <cf_root>\config\wsconfig\<connector_folder_no> and the update level of your CF server. You can also get in touch with the CF support team at cf.install@adobe.com for help with narrowing down the issue.
Comment by Piyush K.
3083 | April 14, 2016 08:57:27 AM GMT
This is a intranet server, we have around 3000 employees. so load is not a problem. I am unable to identify which CFM is causing the Crash, there are no clear information in the logs. Worker.properties dumped below: worker.list=cfusion worker.cfusion.type=ajp13 worker.cfusion.host=localhost worker.cfusion.port=8014 worker.cfusion.max_reuse_connections=250 Attached the crash dumps
Comment by External U.
3084 | April 14, 2016 11:16:09 AM GMT
Attached files are of .TXT extension, please rename them to .MHT and view the files, they were generated via Debugdiag. Most of the applications are CFM and some use CFC. Content server is a separate folder to hold all the docs which are not code and added as a virtual folder in IIS. I tried a lot of thing like changing the pool identity, updating the throttle CPU limits, etc, but nothing worked.
Comment by External U.
3085 | April 14, 2016 11:22:23 AM GMT
The logs suggest a access violation exception when reading from a memory location. Can you pls. ensure that the user account that the IIS worker process is running under has R/W/X privileges to the files and folders in your application site and those "mapped separate content folders".
Comment by Piyush K.
3086 | April 14, 2016 12:33:31 PM GMT
Already tried that, All the services and app pools are now running in the service accounts and they have Admin privileges to both. If there was this access problem then should it not have triggered earlier too ?
Comment by External U.
3087 | April 14, 2016 12:39:56 PM GMT
Can you please clarify what you mean by "service accounts" and "both". Also, pls. share the advanced settings for the application pool and the update level of your CF server. You've mentioned in the report that the problem started after you migrated you application from CF8 to CF11 and that the platform you are using is Win server 2012/IIS 8.5. I'm confused when you say the access privileges hasn't been a problem. Wouldn't that be in the older platform? What was the platform when you were using CF8?
Comment by Piyush K.
3088 | April 14, 2016 01:27:44 PM GMT
We operate in a network environment. So all users have unique IDs and passwords, these are unique and no shared to anyone. We create a "service account" which has a unique id and pwd which is shared by a lot people to be used to login and control servers and applications. Attached the Service account data as JPG. Older platform or newer platform the setup was the same. CF 8 We had on Win 2003 server with again the same service accounts running on the server. My point was, if this was an access problem, then every hit to the server would have errored out, correct ? New server and setup has been active since September 2015 but I got the error only yesterday.
Comment by External U.
3089 | April 14, 2016 01:43:49 PM GMT
Hi Ashok, Can you share the update level of your CF server? Also can you share the isapi_redirect.log from your connector folder.
Comment by Chinoy G.
3090 | April 17, 2016 10:17:18 PM GMT
Chinoy, Piyush, The problem has been solved. Problem has occurred just because of a corrupt folder which had a cfm and jpgs. The files did not have date timestamp but a valid file size. IIS Logs indicate no error, it shows 200 for response and shows up all the files which are being loaded for that http request. however, the moment the corrupt files were loaded Worker process w3wp.exe were killed and thereby the app pool was shut down. The file was opened in the IDE and saved, the JPGs were opened with MSPAINT and saved. All the files had valid timestamps and meta data and there were no problems with the site. But you can clearly see that there are no indications in logs what so ever, this was a monster of an error.
Comment by External U.
3091 | April 18, 2016 12:08:31 AM GMT
The bug report says that the pool crash started after migrating application from CF8 to CF11. But, you've mentioned later, that the new set-up has been operational since 2015. Hence, the confusion. Thanks for the update and the details. Happy to hear that it is resolved. I'll close this. wonder how you isolated the cfm page using the corrupt files, though.
Comment by Piyush K.
3092 | April 18, 2016 04:03:05 AM GMT
We did migrate from Cf8 to Cf11, migration was completed and new setup has been active since 2015. I isolated the CFM by tracing down the IIS logs for the very minute the IIS crashed - I ran through every JPGs, CSS, JS, html and CFMs and finally found out the problematic one.
Comment by External U.
3093 | April 19, 2016 12:37:49 AM GMT
Thanks again for sharing that, Ashok. Apparently restoring the files to a previous version can remove the timestamp. Haven't verified that though.
Comment by Piyush K.
3094 | April 19, 2016 01:27:16 PM GMT
Additional info/Note - The IIS would show that the fetch is successful - code 200 will be in the logs for the corresponding hits, but W3WP gets killed when the file is corrupt after fetching the file. In my case, the files were generated using Microsoft Expressions web 4 which were messed up. Now such file problems are very common if you use a separate "content server" to hold your documents other than the actual CF code. You usually add the content folders//drives as a virtual directory and map it to IIS. To fix such file problems the best way to fix is to create the shared folder as an application and not Virtual directory and assign it a separate application pool Pool needs to be specified as 'No managed code' with type as 'Classic'. By doing this, even corrupt file will not disturb the worker processes there by IIS remains secure and the corrupt files can be caught by just browsing through it.
Comment by External U.
3095 | April 19, 2016 05:25:23 PM GMT