S3

AWS S3 vs On-Premises

AWS S3 vs On-Premises When you own Enterprise Storage or are asked to build enterprise class storage you find yourself more frequently having to cost justify against outside Cloud Storage vendors like AWS S3, GCE or Azure. So, how will you do that since you have to make capital expenditures vs OpEx that these providers discuss so much? When you break down the pricing of AWS S3 and others you find that the actual storage cost is not too bad (depending on amount of stored data).

Continue reading

SLA - Availability or Durability

SLA - Availability or Durability I seem to be getting more questions lately like, “how many 9s is your object store?” or “how do you calculate 9s?”. The questions need to be narrowed down since those are very broad. For example, when dealing with storage like an object store, there are two types of “9s”. One for durability and one for availability. If you look at AWS S3, they put on their primary site 11 9s of durability which is very good.

Continue reading

TCP and Load Balancers

Why do I get a TCP RST return when I try to PUT or GET S3 objects? I have a colleague that has made the following statement: "The data is sacred but, the requests are not!" This holds very true when dealing with HTTP calls such as S3 GETs and PUTs. Recently, we noticed that a client was doing a massive amounts of PUTs and then issuing corresponding GETs on the same S3 objects.

Continue reading

Klaus - Hybrid WebStack

There are very good Web Servers on the market, Apache and NGINX to name a few. However, to interface your backend code you need to use Apache modules or WSGI/FastCGI with NGINX (depending on your language of choice). To actually use a fully integrated environment you need to use something like Django or Tornado for Python. For GO there is a small built-in web server that allows you to build up a more robust environment around it.

Continue reading

S3LSIO - S3 Utility

The initial release of s3lsio has been released for both MacOSX and Linux (RHEL/Fedora/CentOS). Ubuntu will be released soon. In theory Windows should work but OpenSSL can cause initial setup of the Rust build environment for s3lsio. There are constant enhancements being made on a weekly basis. S3lsio is a command line tool that can work within a script, called from an app or ran stand alone to easily manipulate your AWS S3 and Ceph Rados Gateway (S3) environments.

Continue reading

Building Large Ceph Clusters

Ceph is a very complex distributed storage system that provides an Object Store, Block Storage Devices and Distributed File System. It has a built-in installation program called Ceph-Deploy but it’s design is for very simple and small installations. There are two official automated installation and maintenance systems for Ceph, Ceph-Ansible and Ceph-Chef. As the name implies, Ceph-Ansible is built for the Ansible while Ceph-Chef is built for Chef. I will focus on Ceph-Chef for the Chef environment.

Continue reading

Rust AWS S3 SDK

The aws-sdk-rust library is officially released which allows both V2 and V4 API signatures. This is import for those that wish to use the SDK to access storage products that implement the S3 Interface such as Ceph’s Rados Gateway (RGW). Ceph Hammer release only uses V2 while the Jewel and higher releases support V4. Enterprise level proxy support has also been added. So, if http_proxy, https_proxy and no_proxy environment variables are in use, aws-sdk-rust will use them to access the S3 resource.

Continue reading