Install and deploy a Pelican blog using Fabric - Part 3: Fabric
Last updated: 2015-03-04 :: Published: 2015-03-02 :: [ history ]You can also subscribe to the RSS or Atom feed, or follow me on Twitter.
In part 2, we covered the installation and configuration of Pelican in our local environment. It is now time to provision our server and publish content using Fabric.
But first, let's version our blog.
Summary
- Part 1: local environment
- Part 2: installation and configuration
- Part 3: Fabric
- Part 4: workflow, extras and conclusion
Versioning
The next stop in our journey is versioning. What we want to do is to be able to push our content to some repository so it can be pulled on the server later on (and it is always good to have a backup that we can clone and start working on locally in no time).
Create a new repo on your favorite host (like GitHub or Bitbucket) and set up your local one:
git init
git remote add origin your-repo-address
Create a ".gitignore" file (".vagrant/" is not required if you don't use Vagrant):
output/*
*.py[cod]
cache/
.vagrant/
We exclude the content of "output/" as the content will be generated directly on the live server, with its own settings.
Make your first commit and push your code:
git add -A
git commit -m 'First commit'
git push
That's it!
Deployment with Fabric
Fabric is a command-line tool written in Python allowing to perform remote operations over SSH. It is often used for server provisioning and deployment.
We installed it earlier with pip (pip install Fabric) and started using it already to build and serve our blog locally, using the fab build and fab serve commands among others. These commands are contained in a file named "fabfile.py", at the root of your blog. We are now going to have a look at it and explain it a little bit.
fabfile.py
Let's open the file in and editor and dissect it section by section:
from fabric.api import *
import fabric.contrib.project as project
import os
import sys
import SimpleHTTPServer
import SocketServer
Basically the imports of all the necessary libraries. SimpleHTTPServer is the one used by fab serve, for example.
# Local path configuration (can be absolute or relative to fabfile)
env.deploy_path = 'output'
DEPLOY_PATH = env.deploy_path
# Remote server configuration
production = 'root@localhost:22'
dest_path = '/var/www'
# Rackspace Cloud Files configuration settings
env.cloudfiles_username = 'my_rackspace_username'
env.cloudfiles_api_key = 'my_rackspace_api_key'
env.cloudfiles_container = 'my_cloudfiles_container'
This section is about configuration. We will update the "Remote server configuration" bit soon. We are not using Rackspace in this tutorial so we can just ignore this part.
The following function definitions (e.g. "def clean()") are all the commands you can run from the terminal (using "fab command_name" as you already know). You are already familiar with a few of them (build, serve, reserve) and I encourage you to take a look at the others and try them out (you might find that you are more comfortable with some of them for your workflow). The code for most of these commands is quite self-explanatory.
Now let's have a look at the last one, "publish":
@hosts(production)
def publish():
local('pelican -s publishconf.py')
project.rsync_project(
remote_dir=dest_path,
exclude=".DS_Store",
local_dir=DEPLOY_PATH.rstrip('/') + '/',
delete=True,
extra_opts='-c',
)
What's happenning here? First, we indicate that we want to ssh the host whose configuration is contained in the "production" variable mentioned above. Then, in the body of the function itself, we first generate the HTML locally using the live config ("publishconf.py"), and we synchronize the output with the remote server's destination directory defined by the "dest_path" variable and the "DEPLOY_PATH" environment variable.
The synchronization is performed using project.rsync_project, which is a wrapper for the rsync command, allowing to upload newly modified files only.
Basically, everything is there already for you to update a remote server with new content with a single command executed locally. And honestly that might be just enough for your needs.
But it implies that the remote server already exists and is properly set up. And we'd rather see the new content being pulled from our Git repository instead.
Now, you could just stop here for today. I won't be mad, promised.
But if you are interested in seeing how to update the fabfile to both provision our server and publish our versionned content, stick with me.
Provision
Still there? Good.
In our context, what's behind the word "provisioning" is the act of installing all the required software, packages, dependencies, etc for a project to work on a server.
But before that, we are going to create the different error pages.
By default, when trying to access a page that doesn't exist for example, the default HTTP server's 404 page will be displayed.
And it's ugly.
And even if we all agree that the true beauty comes from the inside, we don't want it to be ugly.
Under "content/pages/", create a new folder named "errors/" and the files "403.md", "404.md" and "50x.md" in it (adapt the extensions to the format you chose).
Here is the content of my 404 page as an example:
Title: Hmm...
Slug: 404
Status: hidden
Nope. Don't know what you're talking about, pal.
[Go home](/ "Back to home").
Not much new here, except for the "hidden" status, whose effect is to prevent it from being displayed along with the other pages ("about" etc).
Follow the same format for the two other pages.
When you are done, create a new file called "blog.conf" under the ".provision/" directory, with this content:
server {
listen 80; ## listen for ipv4; this line is default and implied
listen [::]:80 default ipv6only=on; ## listen for ipv6
# Make site accessible from http://my-blog.local.com
server_name my-blog.local.com;
root /var/www/blog;
location = / {
# Instead of handling the index, just
# rewrite / to /index.html
rewrite ^ /index.html;
}
location / {
try_files $uri.htm $uri.html $uri =404;
}
access_log /var/log/blog/access.log;
error_log /var/log/blog/error.log;
# Redirect server error pages
error_page 500 502 503 504 /pages/50x.html;
error_page 404 /pages/404.html;
error_page 403 /pages/403.html;
}
You may have recognized the nginx format: it is indeed the HTTP server I am going to use (feel free to adapt this to your favorite one).
This is a rather basic config: the website will be accessible at http://my-blog.local.com, its root will be "/var/www/blog/" on the remote server, the logs will be written under "/var/log/blog/", and the errors will be redirected to the different pages you have just created.
Now, open back "fabfile.py" and, before the "publish" function, add a "provision" one (it could be after as well, but the provisioning is supposed to come before the publication, right? But maybe that is just my OCD speaking):
@hosts(production)
def provision():
if run('nginx -v', warn_only=True).failed:
sudo('apt-get -y install nginx')
sudo('rm /etc/nginx/sites-available/default')
sudo('service nginx start')
put('./.provision/blog.conf', '/etc/nginx/sites-available/blog.conf', use_sudo=True)
sudo('rm -f /etc/nginx/sites-enabled/blog.conf')
sudo('ln -s /etc/nginx/sites-available/blog.conf /etc/nginx/sites-enabled/blog.conf')
if run('test -d %s/%s' % (log_path, sitename), warn_only=True).failed:
sudo('mkdir %s/%s' % (log_path, sitename))
if run('test -d %s' % root_path, warn_only=True).failed:
sudo('mkdir %s' % root_path)
if run('git -v', warn_only=True).failed:
sudo('apt-get install -y git-core')
if run('pip --version', warn_only=True).failed:
run('wget https://raw.github.com/pypa/pip/master/contrib/get-pip.py -P /tmp/')
sudo('python /tmp/get-pip.py')
run('rm /tmp/get-pip.py')
if run('fab --version', warn_only=True).failed:
sudo('pip install Fabric')
if run('virtualenv --version', warn_only=True).failed:
sudo('pip install virtualenv')
sudo('pip install virtualenvwrapper')
run('echo "export WORKON_HOME=$HOME/.virtualenvs" >> /home/vagrant/.bashrc')
run('echo "source /usr/local/bin/virtualenvwrapper.sh" >> /home/vagrant/.bashrc')
with prefix('WORKON_HOME=$HOME/.virtualenvs'):
with prefix('source /usr/local/bin/virtualenvwrapper.sh'):
run('mkvirtualenv %s' % sitename)
sudo('service nginx restart')
Quite a few things here but, if you look closely, you will realize these are basically all the steps you have taken at the beginning of the tutorial.
The pattern is almost always the same:
- test if the package is installed (
if run('package --version', warn_only=True).failed:) - install it if it is not
The "warn_only=True" parameter allows Fabric not to exit in case of a command failure: this is exactly what we want, i.e. knowing if the command fails so we can install the missing package.
"with prefix" allows to execute the subsequent commands in the context of the current one. If your version of Python is prior to 2.6, you will need to add from __future__ import with_statement at the top of the file.
We will ssh the "server" box as the "vagrant" user, meaning the "run" commands will be executed with its permissions, and the "sudo" ones with the root permissions, just like we did throughout this tutorial.
Back to the actual script: we first install nginx if necessary (and remove the default config), start it, copy the server config file we created earlier to the right location, and recreate the symlink.
Then we ensure the destination directories for the logs and the blog's generated HTML exist, that Git is installed, then pip, Fabric, virtualenv and virtualenvwrapper, we create the "blog" virtual environment and, finally, we restart nginx.
Again, if you followed this tutorial from the start, this should feel familiar.
Now as I am using Ubuntu 14.04.1 LTS boxes, I know my "remote" server comes with Python preinstalled. If yours doesn't, well you will have to add the steps to install it :)
Shall we test this now? Sure thing, but we need to do something first. Remember the configuration section of "fabfile.py" mentioned earlier? It is time to set up the details of our remote server. This is where the use of Vagrant comes in handy as, if you used the Vagrant config I gave you at the beginning of this post, then the "remote server" box is already there, reporting for duty.
Open a new terminal on your host machine, go to the blog's root, and type this:
vagrant up server
Now update "fabfile.py", changing the server configuration for this one:
# Remote server configuration
production = 'vagrant@192.168.72.3:22'
env.key_filename = '/home/vagrant/.ssh/insecure_private_key'
root_path = '/var/www'
log_path = '/var/log'
dest_path = '~/dev'
sitename = 'blog'
symlink_folder = 'output'
The IP address is the one that was specified in the Vagrant config, and the private SSH key is the Vagrant one, copied over from the host machine, via this line of the config:
# Copy the default Vagrant ssh private key over
local.vm.provision "file", source: "~/.vagrant.d/insecure_private_key", destination: "~/.ssh/insecure_private_key"
As mentioned earlier, we will ssh the box as the "vagrant" user. The rest of the variables are path/folder names used in both the "provision" and "publish" functions (next section).
Now, from the "local" Vagrant machine:
fab provision
If everything is set up properly, you should see a series of text lines scrolling off the screen: your "server" box is being provisionned :)
When it is done, you can ssh your "server" VM and play around to observe that everything was properly installed:
vagrant ssh server
Publish
We are now able to provision a server with everything required to run our blog, and all it takes is to update a few lines of configuration in "fabfile.py" and running one command.
Pretty cool, eh?
Anyway, we are yet to update the "publish" function to automate the publication of new content, pulling it from our Git repository.
Here is what it looks like:
@hosts(production)
def publish():
if run('cat ~/.ssh/id_rsa.pub', warn_only=True).failed:
run('ssh-keygen -N "" -f ~/.ssh/id_rsa')
key = run('cat ~/.ssh/id_rsa.pub')
prompt("Add this key to your Git repository and then hit return:\n\n%s\n\n" % key)
if run('test -d %s' % dest_path, warn_only=True).failed:
run('mkdir %s' % dest_path)
with cd(dest_path):
if run('test -d %s' % sitename, warn_only=True).failed:
run('mkdir %s' % sitename)
with cd(sitename):
run('git clone %s .' % git_repository)
if run('test -d %s' % symlink_folder, warn_only=True).failed:
run('mkdir %s' % symlink_folder)
sudo('ln -s %s/%s/%s %s/%s' % (dest_path, sitename, symlink_folder, root_path, sitename))
with cd(sitename):
run('git reset --hard HEAD')
run('git pull origin master')
with prefix('WORKON_HOME=$HOME/.virtualenvs'):
with prefix('source /usr/local/bin/virtualenvwrapper.sh'):
run('workon %s' % sitename)
run('pip install -r requirements.txt')
run('fab preview')
First, we check if there is an existing SSH private key for the "vagrant" user (or whatever user you set up): we are going to pull the content from a Git repository over SSH, so we need one. If none is found, the script will generate one for us and display it so we can add it to our repo. Once this is done, just hit return to continue the execution.
The destination path is then created if necessary: in our case, the repository is cloned and updated in "~/dev/blog". From there, a symlink is created between "~/dev/blog/output" and "/var/www/blog", so the generated HTML files alone are in "/var/www/blog".
Finally, the Virtual Environment is activated, the pip dependencies installed, and the content generated with the live config.
Let's test our new function:
fab publish
Once the execution is over, you should be able to see your blog at the private IP address 192.168.72.3.
There is one last little step to take to access it from the server name as defined in the nginx config. Open the "hosts" file of your host machine and add the following line (change the domain for the one you chose, if different):
192.168.72.3 my-blog.local.com
You can now access http://my-blog.local.com.
That's it! You now know how to provision a server and publish your content using Fabric.
In the next part, we will review a complete workflow, implement a few extra things and conclude this tutorial with a few openings on what to do to go further.
You can also subscribe to the RSS or Atom feed, or follow me on Twitter.