tracker issue | What iT iS dESign studios

Title:

Amazon S3 Support with DirectoryList

| View in Tracker

Status/Resolution/Reason: Closed/Fixed/

Reporter/Name(from Bugbase): Cody Kratzer / Cody Kratzer (cody_lny)

Created: 05/03/2013

Components: ColdFusion Services

Versions: 10.0

Failure Type:

Found In Build/Fixed In Build: Final / CF11 Update5,CF10 Update16

Priority/Frequency: Major / All users will encounter

Locale/System: English / Win 2008 Server R2 64 bit

Vote Count: 3

Listed in the version 11.0.05.293506 Issues Fixed doc

Problem Description:
Using the directoryList function for Amazon S3 (s3://accessKeyId:secretKey@bucket) is pulling back an empty array on my CF10 server. The function is working properly on my other two servers which run CF9 (more details in Test Configuration). 

Steps to Reproduce:
I cannot give you my credentials for the server or my S3 bucket.

Actual Result:
[empty array]

Expected Result:
array with thousands of filenames

Any Workarounds:
I have yet to find a workaround.

----------------------------- Additional Watson Details -----------------------------

Watson Bug ID:	3554224

External Customer Info:
External Company:  
External Customer Name: cody_lny
External Customer Email:  
External Test Config: My Hardware and Environment details:

Windows Server 2008 R2 S1 64-bit with ColdFusion 10 and IIS7 on AWS EC2 m1.small instance



My Hardware on Servers with correct results:

Windows Server 2003 Standard 64-bit with ColdFusion 9 and IIS6 (local)

Windows Server 2003 Standard 64-bit with ColdFusion 9 and IIS6 at RackSpace

Attachments:

May 04, 2013 00:00:00: 1_cf10_s3_screenshot.png

Comments:

Using the application variables method (this.s3.accessKeyId, this.s3.secretKey) does NOT work on ANY of my servers.

Comment by External U.

15507 | May 03, 2013 12:12:54 PM GMT

I've got it to work when I embedded the S3 parameters (using CF 10,0,9,284568, 64bit w/Java 1.7.0_21):

<CFSET S3accessKeyId = "ABC123"> 
<CFSET S3secretAccessKey = "33Ekb4P6">
<CFSET S3Bucket = "s3.mywebsite.com">

<CFSET S3Dir = "s3://#S3accessKeyId#:#S3secretAccessKey#@#S3Bucket#/">
<CFDIRECTORY ACTION="LIST" DIRECTORY="#S3Dir#" NAME="GetResults">
<CFDUMP VAR="#GetResults#">

Comment by External U.

15508 | May 03, 2013 03:36:34 PM GMT

Thanks for the response, Jamo. We saw that we didn't not have our versions up to date (CF10,0,7 and Java 1.6). Our versions are now up-to-date but we are still seeing the same issues.

Comment by External U.

15509 | May 08, 2013 12:58:24 PM GMT

UPDATE! We have found that this issue is only related to certain directories. Currently our S3 bucket has four subdirectories within our main set of subdirectories: thumb, gallery, zoom, and catalog. We have found that cfdirectory and the directoryList function are working when we try to access the catalog subdirectory. However, we still yield zero results when attempting to access thumb, gallery, and zoom. We checked our permissions in S3 and all subdirectories are identical.

Please keep in mind that this error is only occurring on our Windows 2008, CF10, IIS7 64-bit server with fully updated CF and Java versions. We do not see this error when accessing the bucket from our other servers.

We have checked our security settings to ensure that this server has full read capabilities.

Comment by External U.

15510 | May 08, 2013 01:02:53 PM GMT

Another Update! We have found that if we set the directory one level above, we can see results for all four subdirectories! So we can access all four subdirectories at once and we can access the "catalog" subdirectory alone, but we can't access thumb, gallery, or zoom subdirectories alone.

Comment by External U.

15511 | May 08, 2013 02:38:21 PM GMT

unable to repro it can you please provide some more details of those specific folders.

Comment by Asha K.

15512 | May 09, 2013 01:34:29 AM GMT

Contacted customer for more info on the bug with a connect session, hence marking this as Track-> PrNeedInfo.

Comment by HariKrishna K.

15513 | June 05, 2013 11:40:32 PM GMT

We can't use the S3 integration with this issue. It's basically made it useless.

Vote by External U.

15529 | December 19, 2013 05:51:07 PM GMT

I encountered this issue, but it's inconsistent. If I try to list six directories, some will get listed while others will not. If I restart the CF server, then  some other directories will return files, but the previous ones will no longer return any files.

Comment by External U.

15514 | December 19, 2013 05:53:13 PM GMT

I should note that I am using Ubuntu on Amazon EC2 servers.

Comment by External U.

15515 | December 19, 2013 05:54:06 PM GMT

I think the issue is having a trailing / at the end of the directory listing. It should be s3://<accessid>:<secretkey>@bucket/x/y as the path. 

Note that after making the change, you have to restart CF. Otherwise, it will still look broken.

Comment by External U.

15516 | December 19, 2013 08:26:50 PM GMT

I had issues with the cfdirectory with S3 as well.  Here is what I found.

if you do

<CFDIRECTORY ACTION="LIST" DIRECTORY="s3://<accessid>:<secretkey>@bucket/x/y" NAME="GetResults">
  it works just fine.

if do

<CFDIRECTORY ACTION="LIST" DIRECTORY="s3://<accessid>:<secretkey>@bucket/x/y/" NAME="GetResults">

which, has a trailing / on it it doesn't return anything..

But, that's not all..  after you have used it with a trailing forward slash, and then you try it WITHOUT the trailing forward slash it STILL doesn't work.. UNTIL you restart CF, at which point it starts work.  and it ONLY effects the single directory.

So..  If you happen to include a trailing forward slash in your code by mistake.. It blows up the internal CF structure that stores the S3 data until you restart CF.

Thanks, Connor

Comment by External U.

15517 | December 19, 2013 08:56:40 PM GMT

I'm see the same issue.

We have a test server running coldfusion 10 and it doesn't work.

But on our production servers running version 9 it does work listing folders and subfolders (the virtual ones, yes i know that s3 does not have true subfolders).

Works on coldfusion 9 but not 10:

- directoryList("s3/myBucket/")

- directoryList("s3/myBucket/mySubfolder/")

Thanks!

Vote by External U.

15530 | January 22, 2014 09:42:06 AM GMT

The directoryListing is very random. It lists sometimes and sometimes does not. Also Recursive creation of directory is not working . I am logging another bug for that.

Comment by Suchika S.

15518 | December 04, 2014 05:27:46 AM GMT

The fix for this bug is available in the pre-release build of ColdFusion 11 Update 5 and ColdFusion 10 Update 16

Comment by CFwatson U.

15519 | February 20, 2015 09:23:46 AM GMT

I just installed the U5 pre-release and tested this, and I get a NullPointerException.

My path is: s3://[key]:[secret]@[bucket-name]

Note that I'm NOT using a trailing slash. Dumped it to double check.

Here is the stack trace: https://gist.githubusercontent.com/atuttle/739b9f7ff3a5711515ff/raw/87c3b014cb4b068253703ea4253ed9852bca078f/gistfile1.txt

Comment by External U.

15520 | February 20, 2015 12:30:54 PM GMT

And this is why you shouldn't CLOSE a bug until the fix has been verified.

Comment by External U.

15521 | February 20, 2015 12:31:53 PM GMT

Voting to subscribe, since I'm not sure commenting does...

Vote by External U.

15531 | February 20, 2015 12:40:17 PM GMT

For the record, using the same credentials in other S3 explorer applications works fine, so it's not the credentials or the bucket name.

If it helps narrow anything down, my AccessKeySecret contains both a + and a / --- these are legal and common characters for them to use.

My bucket names include hyphens, too: -

Comment by External U.

15522 | February 20, 2015 12:48:36 PM GMT

Adam,

The key we have in our testing environment has "+" and I tried creating a bucket that has a hyphen in the name. Directory Listing Works Fine for me. I tried listing the bucket with/without a trailing slash and also tried to do a listing from a subdirectory. Everything works fine.

Are you experiencing NPE for every bucket listing or it is with  a specific bucket? If it is with a specific bucket can you send some details about the bucket like its structure ,size etc. Is NPE consistently happening or it is random?

Also can you check if the HF5 install was proper and it did not have any errors in the logs?

What platform are you using and also what version of Java are you using?

Comment by Suchika S.

15523 | February 26, 2015 07:48:22 AM GMT

OSX Yosemite, Java 1.7.0_55. NPE is *every* time, not random.

I also tried on Windows, with Java 1.8.0_25, CF11u4 (stable release).

In BOTH cases, a simple directoryList("s3://key:secret@bucket") works, but if you request it back as a query, you get an NPE:

directoryList("s3://key:secret@bucket", false, "query", "", "name asc", "file")

Comment by External U.

15524 | February 26, 2015 09:38:57 AM GMT

Thanks Adam for your quick reply. 
directoryList("s3://key:secret@bucket", false, "query", "", "name asc", "file") gives a NPE whereas same with <cfdirectory action="list"> tag works fine.
This bug fixed the issue for trailing slash. I will log a seperate bug for this issue. Thanks for reporting it.

Comment by Suchika S.

15525 | February 26, 2015 12:10:16 PM GMT

With the benefit of hindsight, I agree that this was not the best place to log this issue. I only ran into it when testing this issue (and trying to use the code in question for work), though. That's why I initially brought it here.

Comment by External U.

15526 | February 26, 2015 12:14:55 PM GMT

Logged a bug for NPE thrown while using
directoryList("s3://key:secret@bucket", false, "query", "", "name asc", "file")
Bug no:CF-3945665

Comment by Suchika S.

15527 | February 26, 2015 12:18:54 PM GMT

Verified CF-3554224 is fixed in CF11 Update 5 (build 11,0,05,293506).

Thanks!,
-Aaron

Comment by External U.

15528 | September 03, 2015 02:16:16 AM GMT

tracker issue : CF-3554224

Amazon S3 Support with DirectoryList