I have had a quota overflow at my FlickR account recently. They would only allow me to see the recent 200 photos. That means very less as I have got used to taking frequent snaps on my mobile phone and Shozu -ing it over . Personal Media is one hell of a thing to manage. Call me a storage paranoid, I have duplicates of personal media thrown all over USB storage, FlickR, PicasaWeb, you name it!. After researching for quite some time I have finally signed-up for an Amazon S3 based Grid Storage Service. It works out cheap and reliable. 
New bandwidth price (effective June 1, 2007)
$0.10 per GB – all data uploaded$0.18 per GB – first 10 TB / month data downloaded
$0.16 per GB – next 40 TB / month data downloaded
$0.13 per GB – data downloaded / month over 50 TB
Data transferred between Amazon S3 and Amazon EC2 will remain free of chargeNew request-based price (effective June 1, 2007)
$0.01 per 1,000 PUT or LIST requests
$0.01 per 10,000 GET and all other requests*
* No charge for delete requests
But Amazon S3 is designed NOT to be consumer facing service. You need to talk in terms of APIs with Amazon S3. Many have found this to be a great business opportunity and created many Amazon S3 Storage Tools. Jeremy Zawodny have done a good job of listing them, but I found a few more tools during my research and here they are – categorized.
Command Line Tools
S3Sync - S3 Sync is a cross platform Ruby Program that easily transfers directories between a local directory and an S3 bucket:prefix. It behaves somewhat, but not precisely, like the rsync program. John Eberly has an efficient way of using S3Sync with Jets3t. [OpenSource]
Quillen S3 Backup – Provides a very efficient chunking system which minimizes the amount of data that has to be transferred between successive versions of a file. Great bandwidth-aware solution for your Amazon S3 account. [OpenSource]
S3cmd - is a command line tool to upload, retrieve and manage data in Amazon S3. [OpenSource]
Drive Letter Integration / Virtual File System
S3Fuse - helps mount S3 buckets as local resource on most Unix/Linux systems. Looks complicated to setup. [OpenSource]
JungleDisk - An excellent, rock-solid solution which works well on Linux and Windows platforms. The Virtual File System is realized atop a cache which makes life lot easier on slower bandwidth connections. Too bad that they don’t offer any other purchase option other than Amazon payments , which does not allow non US customers [Commercial]
S3WebDAV - This java based solution Provides a read+Write WebDAV interface on Unix/Linux systems for your S3 account. This WebDAV URL can be mapped to a drive letter for easier file operations. [OpenSource]
S3Drive - S3Drive is a Windows application that lets you access the Amazon web space as a local network drive. Apart from the ease of use, S3Drive is highly bandwidth aware and fault-tolerant. [OpenSource]
Browser Based / Online
S3Fox Firefox browser extension – Works very well just like an FTP client with 2 panes to operate with. Plus, You can setup automatic synchronization between local folder and S3. [Free]
Sync2S3 - is an online service which helps you manage your S3 buckets and resources through the browser. [Commercial]
S3Browse – You can use this website to manage your Amazon S3 account. [Free]
S3 Browser - is a server-side PHP application that lets you manage your S3 storage and data in it using a web browser. [OpenSource]
Cross Platform Tools
Jets3t ( Pronounced ‘jet-set‘ ) - is a Java based Toolkit, which is very actively under development. The toolkit provides Java programmers with a powerful yet simple API for interacting with S3 and managing data stored there, while the applications like Jets3t CockPit and Uploader allow anyone to easily manage and interact with their S3 accounts. [OpenSource]
S3Backup – S3 Backup is a fail-safe, encrypted online backup solution. [Commercial]
BucketExplorer - A very well designed Java based Application with extensive auditing options for your S3 buckets. [Commercial]
Service Related
Amazon S3 plugin for Wordpress - Allows you to easily use Amazon S3 with your WordPress blog. S3 Bucket can be easily accessed from the Wordpress admin screen. [OpenSource]
FlickR to S3 backup Tool - This is a minimal Python script to backup your FlickR photos to your Amazon S3 account. [OpenSource]
Conclusion
One thing to consider before you select your Amazon S3 Tool is the Integrity and Format of your data. S3 allows http-based access and encryption. Choose a tool or setup a mechanism yourself to encrypt your private data before you store them on S3. Your data should be retrievable even without any dependency on the tool. I chose Jets3t over everything else because I found it to be highly customizable and programmable, not to forget the excellent Cockpit.
{ 2 trackbacks }
{ 12 comments… read them below or add one }
Vinod, thanks for mentioning Bucket Explorer in your post. Just wanted to highlight the fact that Bucket Explorer uses Jets3t library for EVERY S3 operation which has an impact on data integrity. So anyone who trusts jets3t, can trust Bucket Explorer. We have added some extra checks, like we do check for md5 & sha-1 of the file and ifs already there, we don’t upload it again (to save bandwidth and upload time).
Saurabh
I should point out that Quillen also uses jets3t for the underlying operations. In addition to the chunking system you pointed out, it will do date and content checks to avoid re-uploading existing content. It goes even beyond that and will not re-upload data chunks that already exist on S3 because they’re in some other, previously backed up file.
FYI on Jungle Disk – Amazon Payments allows International payments, and we’ve got plenty of customers from around the world.
There is one issue with the Amazon Payments signup form for Intl. users if you’ve got a Mechanical Turk account, but it’s easy to work around (just fill in a bogus address – you can put in your real address with the credit card info). Thanks and good luck!
Nice post. However, the thing I’ve noticed about most S3 clients is that they use a proprietary file-system within the buckets. That doesn’t let you share the files over HTTP (though it may help in security).
In Windows, I use S3Webmaster; it allows you to download the files using HTTP GET, so it’s much easier to upload large files for clients, for example. Wonder if there’s something like that for Linux. Just have to go through your list. Unfortunately, most of the sites are direct links to downloads; no info.
A high performance, S3-backed file system, see PersistentFS, http://www.PersistentFS.com
We are launching a new service that will allow you to rsync your local files to your amazon s3 bucket/s.
For more information take a look at out web site.
There is alse the freeware GUI Client for Amazon S3 – S3 Browser (s3browser.com)
We enhanced the solid CrossFTP engine to support S3, which will allow you to schedule/sync/upload/download your amazon s3 buckets and FTP sites.
For more information please refer the web site.
Best product I’ve seen that plugs directly into the S3 is the: http://www.hostedFTP.com solution.
CloudBerry Explorer is another Amazon S3 and CloudFront client for Windows users. It has an FTP like interface and it support PowerShell. you can download it for free at http://cloudberrylab.com/
You also have a S3 explorer applet that runs into your browser: JS3Explorer at
http://www.jfileupload.com/products/js3explorer/index.html
S3 itself is great, and a great value for the money. I was excited when I first encountered it, and initially went with Jungledisk as a tool. Unfortunately, on lauch it causes panic reboots of my XP Pro box. I can find no other cause or problems with this otherwise stable box.
Not exactly a confidence builder. That and the fact they charge a monthly fee seemed odd. Paying for the app is fine, but paying them for a subscription, and then paying Amazon on top of this seems silly.