Azurecopy (Go version) 0.2.2 has now been released. Major benefit is now that when copying to Azure we can use the absolutely AWESOME CopyBlob functionality Azure provides. This allows blobs to be copied from S3 (for example) to Azure without having to go via the machine executing the instructions (and use my bandwidth!)
An example of copying from S3 to Azure is as simple as:
azurecopycommand_windows_amd64.exe -S3DefaultAccessID=”S3 Access ID” -S3DefaultAccessSecret=”S3 Access Secret” -S3DefaultRegion=”us-west-2″ -dest=”https://myaccount.blob.core.windows.net/mycontainer/” -AzureDefaultAccountName=”myaccount” -AzureDefaultAccountKey=”Azure key” -source=https://s3.amazonaws.com/mybucket/ –copyblob
The key thing is the –copyblob flag. This tells AzureCopy to do it’s magic!
By default AzureCopy-Go will copy 5 blobs concurrently, this is to not overload your own bandwidth, but with Azure CopyBlob feature feel free to crank that setting up using the –cc flag (eg add -cc=20)
Being able to copy from one cloud provider to another is useful, but if everything is purely serial (ie one blob at a time) the time taken to copy everything might be less than stellar. I’ve now released a new version of AzureCopy-Go (0.2.1) which allows concurrently copying of blobs. The default is 5, but using the –cc flag (concurrent copying) it can be expanded up to 1000 (arbitrary max limit). So far, so good!
Also, for this release I’ve build the binaries using the AMAZING Gox project. This allows for easy cross compiling for Go. So we now have Linux, FreeBSD, NetBSD, OpenBSD, Darwin (MacOS) and Windows binaries. For the most part we have 3 variations of each platform binary, one each for ARM, AMD64 and 386.
I knew how to get cross compiling with Go on Linux/MacOS but could never get it working on Windows (current main OS). Gox is definitely a time saver and is so damned easy to use.
Please give AzureCopy-Go 0.2.1 a try if you have any S3 <—> Azure migration needs. More features being worked on every few days.
As mentioned in previous posts, I’ve been writing a GO version of AzureCopy so people would have something that works cross platform (Linux, MacOS and Windows). Today I’ve released the first pre-release just to test the waters. It supports Windows only (simply due to not having compiled up the other platforms yet), and only supports local filesystem, S3 and Azure Blob Storage.
The plan is to build for Linux,OSX, then start adding other cloud platforms. Meanwhile the original Azurecopy (Windows only, full dotnet framework) will still be developed (mainly from a Nuget/library point of view). If you just need an executable to perform copying, then I suggest using this newer version.
Some examples of using this newer version:
In this case we’re just listing the contents of my testken123 (super secret) bucket. My AccessID and AccessSecret are passed in via command line options. The output format is in a basic tree structure (will add in a bog-standard list soon). In the above case, the top of the tree is “testken123” which is the bucket name. Under that we have 2 virtual directories (remember Azure/S3 etc do not really have directories but fake it by using / as a delimiter). In this case we see there is a blob called “ken1/test1” which treats the “ken1” part as a directory and “test1” as the blob name. Same applies for all the other results. Simple enough.
Then we have:
In this case we’re copying from my local filesystem (c:\temp\data\s3\) into the S3 bucket testken123. The console output is just to show what is going to be copied. Output will be modified to show progress.
Finally we have:
That’s coping from Azure Blob Storage to S3. Same deal, basic output.
For every command it is possible to pass the “-debug” flag. This makes things VERY verbose but is extremely useful for figuring out issues.
This is just a first step, pre-release, uber new version. Please give it a go and let me know if there are any issues. The plan is to start cranking out changes pretty frequently.
The Go version of AzureCopy is slowly making progress. So far I’ve just been focusing on local filesystem and Azure (since I can do those while offline on the train commute thanks to the Azure Storage Emulator). The next plan is for S3 integration, primarily because S3 –> Azure seems to be the big use case for the original AzureCopy.
I’m planning on frequent releases once the basic S3 code is added (hopefully within the next few days). Not all features from the original AzureCopy will be available, but will simply be focusing on 1) list content and 2) copy content. There will be a few new additions such as a “don’t overwrite” flag so copies can be continued after being stopped (has been requested by a few people).
Ofcourse, the original AzureCopy will still be developed (mainly from a Nuget packaging point of view) but if you just need a command line tool to copy (and maybe need it on multiple platforms) then this new version is probably the way to go.
Hopefully the S3 code will drop in a few days then I’ll have a first binary release for Linux, MacOS and Windows, and see how things proceed from there.