tracker issue : CF-4028272

select a category, or use search below
(searches all categories and all time range)
Title:

[ANeff] Bug for: CF duplicates IIS requests

| View in Tracker

Status/Resolution/Reason: Closed/Won't Fix/ThirdParty

Reporter/Name(from Bugbase): Aaron Neff / Aaron Neff (Aaron Neff)

Created: 07/30/2015

Components: Web Container (Tomcat)

Versions: 11.0

Failure Type: Performance Issue

Found In Build/Fixed In Build: CF11_Final /

Priority/Frequency: Normal / Few users will encounter

Locale/System: ALL / Platforms All

Vote Count: 1

CF duplicates IIS requests. Steps to reproduce:

Application.cfc:
----------------
component {
  THIS.name = "ticket_CFDuplicatesIISRequests";
  void function onRequest() {
    cfsetting(requesttimeout=120);
    for(var i=1; i<120; i++) {
      sleep(1000);
    }
  }
}

index.cfm
---------
empty file

1) While above code is running, do the following:
2) In CF's Server Monitor, go to "Active Requests" and see the running request
3) In the Windows Task Manager, end the w3wp.exe process ID* for the associated application pool
4) In CF's Server Monitor, go to "Active Requests" and see the running request was duplicated (bad)
5) Continually repeat #3 and see request is continually duplicated (bad)

To find the w3wp.exe process ID, click the server node at the left side of IIS, then click Worker Processes at the right, then see the process ID associated w/ the application pool.

By default, IIS has health monitoring pings enabled (a per application pool setting). If it is unable to ping an application pool's w3wp.exe process w/in 90 seconds (default value), it deems the process is unhealthy and moves its request queue to a replacement w3wp.exe process and terminates the 'unhealthy' one. Basically, if there is a performance issue on the CF side, and IIS's pings timeout, then IIS follows the described procedure. IIS does this so that users are unaware of the crash.

The issue is that CF's IIS connector is apparently unable to detect that the already-running requests have simply been moved to a replacement process. So, CF treats these moved requests as new requests.

This can become a spiraling performance issue, where an already-burdened CF becomes even more burdened as it is flooded w/ these duplicate requests.

Of course I hope the above issue can be resolved. Additionally, here a suggestion for the future: An onRequestStart() in Server.cfc would be helpful when CF/IIS issues like this arise. We can use IIS's appcmd.exe to get each request's underlying process ID as well as determine if a previous process ID is still active. Using that info, we can filter out these duplicate requests. It'd just be cleaner to deal w/ this in Server.cfc instead of Application.cfc. I'm not asking for a full-blown firewall baked-in, but a Server-level hook would be nice.

----------------------------- Additional Watson Details -----------------------------

Watson Bug ID:	4028272

External Customer Info:
External Company:  
External Customer Name: Aaron
External Customer Email:  
External Test Config: Tested on Windows 2012 R2 w/ CF11 Update 5.

Attachments:

Comments:

vote. Vote must be between 25 and 4000 charactersVote must be between 25 and 4000 charactersVote must be between 25 and 4000 charactersVote must be between 25 and 4000 charactersVote must be between 25 and 4000 charactersVote must be between 25 and 4000 charactersVote must be between 25 and 4000 charactersVote must be between 25 and 4000 charactersVote must be between 25 and 4000 charactersVote must be between 25 and 4000 charactersVote must be between 25 and 4000 characters
Vote by External U.
6416 | July 31, 2015 12:39:42 AM GMT
Adobe, You might want to have a look at this one. Thanks!, -Aaron
Comment by External U.
6409 | August 12, 2015 01:55:14 PM GMT
Aaron, We are evaluating how this issue can be dealt with. The connector does not have the necessary information to 'migrate' hung connections into new ones. Invoking appcmd.exe at runtime would not be the ideal solution.
Comment by Immanuel N.
6410 | August 13, 2015 03:17:37 AM GMT
As mentioned, IIS does duplicate the request. But there is no way to handle it properly. Connector does not store any request related information, so nothing can be done there. On CF side, as Aaron mentioned, apppcmd can be used to check for duplicate request, but that is not doable as it causes an overhead for all the requests. Also it cannot be used in case of remote IIS. So there is no way CF can resolve this. Only feasible solution is to tell IIS to not to replicate the request, so that CF does not get duplicate request, and the earlier one will get terminated after timeout happens.
Comment by Chinoy G.
6411 | September 24, 2015 12:57:33 AM GMT
Hi Chinoy, Regarding "Only feasible solution is to tell IIS to not to replicate the request", how do I do this? This is a real problem. Thanks!, -Aaron
Comment by External U.
6412 | September 24, 2015 01:26:43 AM GMT
Aaron, I was also not able to find any way to do it.
Comment by Chinoy G.
6413 | September 24, 2015 01:33:56 AM GMT
Hi Chinoy, Could onRequestStart() please be added to Server.cfc so that developers can choose to handle this using appcmd.exe for local IIS? It's a mess to handle this in Application.cfc for every site on the server. Thanks!, -Aaron
Comment by External U.
6414 | September 24, 2015 02:02:30 AM GMT
Hi Chinoy, This is a real spiraling problem. I'm sure Adobe will blame Microsoft. I'm sure Microsoft will blame Adobe. So how do we resolve this? I can workaround it by using appcmd.exe. Issue: Server.cfc doesn't have an onRequestStart(), so I have to put code in every Application.cfc's onRequestStart(). That makes it more complicated to maintain. Request: Can this ticket be fixed by adding onRequestStart() to Server.cfc? Thanks!, -Aaron
Comment by External U.
6415 | September 29, 2015 08:44:45 PM GMT