Posted on Thu, Dec 17, 2015
Swift is OpenStack’s implementation of an object storage. In cloud computing, we often refer to block and object storage. OpenStack provides two different components, each responsibe for one of these parts:
Note: We won’t be covering Cinder in this article.
What is the difference between these two types of storage? Well, the name probably says it all, one is on the block level, it means that you can install an operating system, have a filesystem and use it as you would use any physical drive you may have.
The other one is about objects, it means you can store data (images, music, videos, archives, all kind of data). So you guessed right, the title is misleading with the “VS”, because they don’t provide the same kind of service.
Here are a few cool things about Swift:
Swift, like all the components of the stack uses a REST API to communicate. It’s also true for the user. It’s possible to interact with swift in 3 separate ways:
CLI: using the CLI (Command-Line Interface). To do so, you need to install the python client that provides the command. Usually, the openstack clients packages are named like this: python-*name*client, which in the case of swift is:
To use it, you only need to install the following packages:
$ sudo apt-get install python-swiftclient python-keystoneclient OR $ sudo pip install python-swiftclient python-keystoneclient
Note that we also install the keystone client because by default swift will use keystone for authentification. It’s possible to change this, swift has its own auth methods but that would only add some needless complications so we’ll stick with keystone.
Here are a few examples of what you can do with the cli (you need to source an openrc.sh script first or use the env vars):
$ swift -V 3 upload container-name file $ swift -V 3 download container-name file $ swift -V 3 post container-name
So, in order:
You can also work on your own version of the python client, we don’t do that so this is just for reference. More info can be found here.
Usually, the cli python clients do nothing more than provide you with friendly commands you can use on your terminal. All these commands exist with the REST API as well (the basic rule is, you should be able to do the same things with the CLI and the REST API). As for Swift, we never really had to work with the API, but once again, you can check it here for reference.
A fair question indeed. You can either use it for storage, which would probably be the primary option but there are other things one can do with an object store. This image will give you a good hint:
If you have ever worked with Hadoop or Spark, chances are pretty good that you are familiar with HDFS. If not, HDFS is simply a shared filesystem used by the hadoop workers to share data (read and write). Hadoop needs HDFS and Spark may use it as well but is not entirely dependant. It should be possible (as we read on Spark’s website) to replace HDFS by Swift. Why would we do that though? Well, first reason, we already have a working swift cluster. An other reason is, to use HDFS we need to install it on all the workers, meaning if we want to have a “virtual” cluster, we still need to give our instances a lot of block storage. Also, deleting the instances will delete the HDFS cluster, which won’t be the case with Swift.
We are still working on this but the big part of the work is to adapt our codes to work with swift (login, access, etc). We have high expectations regarding this so we’ll keep working on it until it works :)
We’ve had a few problems while installing Swift. First, swift does not create logs files, it uses syslog for logging. It means that you need to consult this file to see what’s up: > /var/log/syslog
Is this really a problem? No but it’s worth knowing :)
Also, Swift makes a loooooot of logs, because it’s very verbose. Once again, having an ELK Stack is very useful :P (you can check our post about it if you are interested).
The installation in itself is pretty straightforward, once the conf files are downloaded, change them accordingly, create the rings, distribute them accross your swift-storage nodes and check if everything works fine.