tracker issue | What iT iS dESign studios

Title:

AWS S3 interaction - File Append error with CFFILE tag and function

| View in Tracker

Status/Resolution/Reason: Closed/Fixed/Fixed

Reporter/Name(from Bugbase): JJ BLODGETT / ()

Created: 11/27/2018

Components: File Management, VFS-S3

Versions: 2016,2018

Failure Type: Data Loss

Found In Build/Fixed In Build: 2016,0,06,308055 / CF2016U12,CF2018U5

Priority/Frequency: Normal / All users will encounter

Locale/System: English / Win All

Vote Count: 5

Problem Description:
Amazon Web Services (AWS); Simple Storage Service (S3). Beginning in Update 6 (and still in Update 7), Calls to append to a text file with CFFILE action="APPEND" (and WRITE if the object already exists) fail with one of the following messages:

Service Error Message. Your metadata headers exceed the maximum allowed metadata sizeCould not close the output stream for file "s3://[bucket]/test.txt"

- OR -

The Content-MD5 you specified did not match what we received.Could not close the output stream for file "s3://[bucket]/test.txt"

This results in the inability to append to files in AWS (or potentially to overwrite existing files). It also could result in data loss when the result is unable to be written (for example in our case where user actions aren't being logged).

Steps to Reproduce:
Execute the following (can be run on separate pages or at different times). But will happen if you run during the same request as well. Also, it does the same with the cffile tag.

cffile(action="write", file="s3://[bucket]/test.txt", output="test content");
cffile(action="append", file="s3://[bucket]/test.txt", output="test content");

Actual Result (Not sure when/why one or the other is returned):
Service Error Message. Your metadata headers exceed the maximum allowed metadata sizeCould not close the output stream for file "s3://[bucket]/test.txt"

- OR -

The Content-MD5 you specified did not match what we received.Could not close the output stream for file "s3://[bucket]/test.txt"

Expected Result:
File content should be appended to file in AWS / S3.

Any Workarounds:
Not that we have found by using cffile

Attachments:

Comments:

Forgot to add that this is happening under Java version 8 U191 (1.8.0_191-b12)

Comment by JJ B.

29975 | November 27, 2018 09:25:40 PM GMT

I am seeing this same behavior in build 2018.0.01.311402.  Any idea when this might get resolved?

Vote by Bobby H.

30133 | January 19, 2019 05:12:56 AM GMT

Hi all,

I see one issue is that CF2018 doesn't garbage collect its superfluous S3 metadata when overwriting files, seemingly allowing some headers to grow past S3's 8KB limit for some file types. Example (just uncomment one section at a time, and start hitting F5):

<cfscript>
  myS3Directory = "s3://mybucket.mydomain.com";//replace w/ your own bucket name
  if(!directoryExists(myS3Directory)) {
	  directoryCreate(myS3Directory);
  }
  
  //F5'ing never breaks for .jpg
  /*myImage = imageNew("", 200, 200, "rgb", "blue");
  imageWrite(myImage, myS3Directory & '/myImage.jpg', 1, true);
  writeDump(storeGetMetadata(myS3Directory & '/myImage.jpg'));*/
  
  //3rd F5 breaks for .png
  /*myImage = imageNew("", 200, 200, "rgb", "blue");
  imageWrite(myImage, myS3Directory & '/myImage.png', 1, true);
  writeDump(storeGetMetadata(myS3Directory & '/myImage.png'));*/
  
  //3rd F5 breaks for .txt
  /*fileWrite(myS3Directory & '/myFile.txt', "a");
  writeDump(storeGetMetadata(myS3Directory & '/myFile.txt'));*/
</cfscript>

Error: "org.apache.commons.vfs2.FileSystemException - Service Error Message. Your metadata headers exceed the maximum allowed metadata size"

Adobe, could you please fix this metadata issue? I've confirmed the issue didn't exist in CF11 Update 7 or CF2016 Update 1, but exists in 2018.0.0.310739.

Thanks!,
-Aaron

Comment by Aaron N.

30283 | February 16, 2019 12:19:03 PM GMT

Hi all,

Regarding the Content-MD5 vs the metadata size errors, just run this twice and see both errors:

<cfscript>
  myS3Directory = "s3://mybucket.mydomain.com";//replace w/ your own bucket name
  if(!directoryExists(myS3Directory)) {directoryCreate(myS3Directory);}
  //On 1st run, throws org.apache.commons.vfs2.FileSystemException "Service Error Message. The Content-MD5 you specified did not match what we received."
  //On 2nd run, throws org.apache.commons.vfs2.FileSystemException "Service Error Message. Your metadata headers exceed the maximum allowed metadata size"
  fileWrite(myS3Directory & '/myFile.txt', "a");
  cffile(action="append", file=myS3Directory & '/myFile.txt', output="b");//both errors are thrown on this line
  writeDump(storeGetMetadata(myS3Directory & '/myFile.txt'));
</cfscript>

Thanks!,
-Aaron

Comment by Aaron N.

30284 | February 16, 2019 09:59:50 PM GMT

+1 - CF's metadata breaks S3

Vote by Aaron N.

30285 | February 16, 2019 10:02:17 PM GMT

Getting the same issue under 2018 when doing "FileWrite" to existing file at S3 bucket:

Service Error Message. The Content-MD5 you specified did not match what we received.Could not close the output stream for file

RootCause - org.apache.commons.vfs2.FileSystemException: Could not close the output stream for file

Vote by Maxim B.

30623 | April 10, 2019 08:29:04 PM GMT

Also getting the The Content-MD5 error on FileWrite

Vote by Keith L.

30624 | April 10, 2019 09:38:11 PM GMT

tracker issue : CF-4203583

AWS S3 interaction - File Append error with CFFILE tag and function