tracker issue : CF-3226380

Problem Description:

It appears that ColdFusion is caching Amazon S3 Metadata on the Server / Instance.  Once you retrieve an object's Metadata on a Server / Instance, changes or additions to that Metadata from other Servers, Instances, 3rd Party Apps, or Amazon S3 Console can't be retrieved on that same Server  / Instance until the CF Service is restarted.

This means that if you get the Metadata from ObjectX using StoreGetMetadata, and then you add an additional key from any other source (server, instance, S3 console, etc), a subsequent call to StoreGetMetadata for ObjectX will not reflect the additions or changes.

In addition, StoreGetMetadata doesn't return the exact S3 Metadata key names even on the same server that used StoreSetMetadata to add/update the Metadata in the first place.

It appears that whatever cache mechanism being used does not refresh or even have an option to refresh the cache information on a given instance.  We have confirmed that the Metadata remains static overnight (at least 18-19 hours) and the only way we've found to get the correct values from Amazon is to restart the CF service.  This is obviously not practical or possible every time Metadata is updated.

It appears that even if you update Metadata from an instance of CF, there is still some "cache" version of the object being referenced locally because the cfdump of the Metadata struct would reflect keys with different upper/lower cases if the struct key was created in all caps.

This would affect anyone that was accessing Amazon S3 metadata from multiple sources include multiple instances of CF, multiple CF Servers, the Amazon S3 Console, or 3rd Party Apps such as BucketExplorer, etc.

Steps to Reproduce:

STEP 1:

CF_INSTANCE_A:
<cfset structContent = StoreGetMetadata("s3://BUCKET/OBJECT")>
<cfdump var="#str#">

CF_INSTANCE_B:
<cfset structContent = StoreGetMetadata("s3://BUCKET/OBJECT")>
<cfdump var="#str#">

CF_INSTANCE_C:
<cfset structContent = StoreGetMetadata("s3://BUCKET/OBJECT")>
<cfdump var="#str#">

RESULT: All 3 instances should return the same values

STEP 2:

CF_INSTANCE_A:
<cfset structNewMeta = StructNew()>
<cfset structNewMeta ["varTest"] = "value">
<cfset StoreSetMetadata("s3://BUCKET/OBJECT", structMeta1)>

<cfset structContent = StoreGetMetadata("s3://BUCKET/OBJECT")>
<cfdump var="#str#">

CF_INSTANCE_B:
<cfset structContent = StoreGetMetadata("s3://BUCKET/OBJECT")>
<cfdump var="#str#">

CF_INSTANCE_C:
<cfset structContent = StoreGetMetadata("s3://BUCKET/OBJECT")>
<cfdump var="#str#">

RESULT: CF_INSTANCE_A will see the original Metadata AND a new key called "VARTEST" (all uppercase). CF_INSTANCE_B and CF_INSTANCE_C do not see the new key at all.  You can look at the object directly in the Amazon S3 Console to confirm the key "x-amz-meta-vartest" was added successfully.  But note that the key name is actually lowercase and not uppercase as CF_INSTANCE_A is reporting.

STEP 3:

CF_INSTANCE_B:
[[Restart CF Instance]]
<cfset structContent = StoreGetMetadata("s3://BUCKET/OBJECT")>
<cfdump var="#str#">

RESULT: CF_INSTANCE_A will see the original Metadata AND the key called "VARTEST" (all uppercase). CF_INSTANCE_B will see the original Metadata AND the key called "vartest" (all lowercase). CF_INSTANCE_C does not see the new key at all.

STEP 4:

AMAZON S3 CONSOLE:
Change value of "vartest" to "new value"

RESULT: CF_INSTANCE_A and CF_INSTANCE_B will still see their respective VARTEST/vartest keys but the value will contain "value" and will not reflect the "new value" that is the actual value at Amazon.

Workaround:

Only workaround we have found so far is to restart the CF service which is not practical in any environment.

----------------------------- Additional Watson Details -----------------------------

Watson Bug ID:	3226380

External Customer Info:
External Company:  
External Customer Name: jjblodg
External Customer Email:  
External Test Config: Initially found on CF 9.0.1 Enterprise running as Multiple Instances on the same machine.  Confirmed the behavior on a CF10 single server install at a completely different company/server accessing the exact same object we referenced on the CF 9.0.1 instances.

Attachments:

Comments: