At DropMyEmail, we are evaluating open source private cloud storage solutions. We have found OpenStack Swift to be the best choice in the market right now. We are documenting the steps to build a prototype swift cluster. Much of the work is from this amazing blog post by Amar and the multiple server swift installation document. Our setup is largely similar except that we are on a more recent stack.
Overview
It is a simple architecture. We have a proxy server which acts as the interface for the outside world. The proxy server includes the authentication server. The proxy server is connected to 5 storage nodes. Each storage node includes account, container and object servers. Each node is a single EC2 instance.
Setting up the instances
We assume you know how to set up an EC2 instance. Our instance are all Ubuntu Linux 12.04 images from Alestic.
For the security groups, here are the settings we used for each type of node.
Security group
Port 22 from source 0.0.0.0/0
Port 43 from source 0.0.0.0/0
Port 6000-6100 from source 0.0.0.0/0
For the rest of the article, we’ll be using the public and private IP addresses of the EC2 instance at different scenarios. Do note the differences betwen them. We’ll be assuming you are logging into the instance via ssh and running commands.
Preparing the base Swift image
We are going to prepare an AMI image with the core Swift software installed.
First we add the swift repository and install the base swift packages.
123
sudo add-apt-repository ppa:swift-core/release
sudo apt-get update
sudo apt-get install swift
Next we set up the swift configuration files. Create the Swift configuration directory.
1
sudo mkdir -p /etc/swift
We need to create a random string to act as the hash for Swift. Run the following command, you should see a random string as the result.
1
sudo od -t x8 -N 8 -A n </dev/random
Create /etc/swift/swift.conf. Add the following lines to it.
123
[swift-hash]
# random unique string that can never change (DO NOT LOSE)
swift_hash_path_suffix = <RESULTS FROM ABOVE od COMMAND>
Change the owner to swift.
1
sudo chown -R swift:swift /etc/swift
Now save the image as an AMI. This base image can be used for the other instances.
Setting up the Swift Proxy server
Now choose one instance for the proxy server. The proxy server contains both the proxy and authentication server. Let us install the proxy server first.
Run these commands to install the dependencies.
1
sudo apt-get install swift-proxy memcached swauth
You could use OpenStack Keystone for authentication. Cons is you need to set up and maintain another system for authentication. Swauth is a self-contained authentication server on the proxy server. If you just want to run Swift on its own, we’d recommend going with Swauth.
Next we create the SSL cert to encrypt our requests.
We need to replace the IP address in the memcached configuration file and then restart memcached. Check the private IP address for the instance and replace the IP with your own. We are assuming it is 10.0.0.0.
Next we need to create the configuration file for the proxy server. Create /etc/swift/proxy-server.conf with the following contents. Remember to replace the URL and IP address with those from your instance.
1234567891011121314151617181920212223242526
[DEFAULT]
bind_port = 443
cert_file = /etc/swift/cert.crt
key_file = /etc/swift/cert.key
workers = 8
user = swift
[pipeline:main]
pipeline = healthcheck cache swauth proxy-server
[app:proxy-server]
use = egg:swift#proxy
allow_account_management = true
[filter:swauth]
use = egg:swauth#swauth
set log_name = swauth
super_admin_key = swauthkey
default_swift_cluster = cluster_name#https://ec2-1-2-3-4.compute-1.amazonaws.com:443/v1#https://127.0.0.1:443/v1
[filter:healthcheck]
use = egg:swift#healthcheck
[filter:cache]
use = egg:swift#memcache
memcache_servers = 10.0.0.0:11211
This is it for now. We’ll set up the storage nodes first before coming back to the proxy server.
Setting up the Swift Storage nodes
Unlike what Amar proposed, there is no need to keep the instances under the same availability zone. Ideally the storage nodes should be in different regions to prevent multiple failures. For our case, we are setting up everything in the same region for convenience.
We are going to add EBS volumes to mimic hard drives. Create a new EBS volume for each storage server and attach them. You may use the default value of /dev/sdf for Device. They will be renamed to /dev/xvdf in your instance.
The storage node contain the account, container and object servers. Install the dependencies.
sudo mkdir -p /srv/node/xvdf1
sudo mount /srv/node/xvdf1
And change the owner to swift.
1
sudo chown -R swift:swift /srv/node
Rsync
We need to configure rsync as well. Create /etc/rsyncd.conf. Add the following to the file. STORAGE_LOCAL_NET_IP is the IP address of the instance. Change it accordingly.
1234567891011121314151617181920212223
uid = swift
gid = swift
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
address = <STORAGE_LOCAL_NET_IP>
[account]
max connections = 2
path = /srv/node/
read only = false
lock file = /var/lock/account.lock
[container]
max connections = 2
path = /srv/node/
read only = false
lock file = /var/lock/container.lock
[object]
max connections = 2
path = /srv/node/
read only = false
lock file = /var/lock/object.lock
Enable rsync in the rsync configuration at /etc/default/rsync.
1
RSYNC_ENABLE = true
And start rsync
1
service rsync start
Account/Container/Object server configurations
Next we need to set the IP address for the account, container and object server configuration files. Use the instance’s IP address for STORAGE_LOCAL_NET_IP.
The account server’s configuration file is at /etc/swift/account-server.conf.
Repeat these steps for the rest of the storage nodes.
Starting the servers
Let’s go back to the proxy server. We are going to create a script to build the rings. Create a file at /etc/swift/build_rings.sh. Replace the IP address accordingly.
#!/bin/sh
export ZONE=1 # set the zone number for that storage device
export STORAGE_LOCAL_NET_IP=10.0.0.1 # and the IP address
export WEIGHT=100 # relative weight (higher for bigger/faster disks)
export DEVICE=sdf1
sudo swift-ring-builder account.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6002/$DEVICE $WEIGHT
sudo swift-ring-builder container.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6001/$DEVICE $WEIGHT
sudo swift-ring-builder object.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6000/$DEVICE $WEIGHT
export ZONE=2 # set the zone number for that storage device
export STORAGE_LOCAL_NET_IP=10.0.0.2 # and the IP address
export WEIGHT=100 # relative weight (higher for bigger/faster disks
export DEVICE=sdf1
sudo swift-ring-builder account.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6002/$DEVICE $WEIGHT
sudo swift-ring-builder container.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6001/$DEVICE $WEIGHT
sudo swift-ring-builder object.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6000/$DEVICE $WEIGHT
export ZONE=3 # set the zone number for that storage device
export STORAGE_LOCAL_NET_IP=10.0.0.3 # and the IP address
export WEIGHT=100 # relative weight (higher for bigger/faster disks)
export DEVICE=sdf1
sudo swift-ring-builder account.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6002/$DEVICE $WEIGHT
sudo swift-ring-builder container.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6001/$DEVICE $WEIGHT
sudo swift-ring-builder object.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6000/$DEVICE $WEIGHT
export ZONE=4 # set the zone number for that storage device
export STORAGE_LOCAL_NET_IP=10.0.0.4 # and the IP address
export WEIGHT=100 # relative weight (higher for bigger/faster disks
export DEVICE=sdf1
sudo swift-ring-builder account.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6002/$DEVICE $WEIGHT
sudo swift-ring-builder container.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6001/$DEVICE $WEIGHT
sudo swift-ring-builder object.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6000/$DEVICE $WEIGHT
export ZONE=5 # set the zone number for that storage device
export STORAGE_LOCAL_NET_IP=10.0.0.5 # and the IP address
export WEIGHT=100 # relative weight (higher for bigger/faster disks
export DEVICE=sdf1
sudo swift-ring-builder account.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6002/$DEVICE $WEIGHT
sudo swift-ring-builder container.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6001/$DEVICE $WEIGHT
sudo swift-ring-builder object.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6000/$DEVICE $WEIGHT
Copy your instance’s private key(the key file which you use for sshing in to your instance) to the proxy server. I’m assuming it is private_key.pem. Change the permission of private_key.pem to 0600.
Next we need to propagate the ring files to all the storage nodes
