ElasticSearch Backup in linux

Elasticsearch backup script with snapshot rotation

Mar 23, 2015 · 2 minute read · 5 Comments
DevopsDevelopment

Edit 2015/10/16: Added the example restore script.

Edit 2015/3/31: It seems there is also a python script called curator that is intended to be a housekeeping tool for elasticsearch. While curator is being a more complete tool, my script below works just as well and doesn’t need python installed. Use whatever tool you prefer.

Elasticsearch 1.4 has an easy way to make backups of an index: snapshot and restore. If you use the filesystem way, you can just make a snapshot, rsync/scp/NFS export the files to another host and restore them from those files.

Setup the snapshot repository

Setup the snapshot repository location:

curl -XPUT 'http://localhost:9200/_snapshot/my_backup' -d '{
  "type": "fs",
  "settings": {
    "location": "/mount/backups/my_backup",
    "compress": true
  }
}'

Take snapshots

A backup script you can run on cron would be as simple as this:

#!/bin/bash
SNAPSHOT=`date +%Y%m%d-%H%M%S`
curl -XPUT "localhost:9200/_snapshot/my_backup/$SNAPSHOT?wait_for_completion=true"

While it’s very easy to set up this backup, there is currently no logrotate included to remove old snapshots. I wrote a small script using the jq program that keeps the last 30 snapshots and deletes anything older:

#!/bin/bash
#
# Clean up script for old elasticsearch snapshots.
# 23/2/2014 [email protected]
#
# You need the jq binary:
# - yum install jq
# - apt-get install jq
# - or download from http://stedolan.github.io/jq/

# The amount of snapshots we want to keep.
LIMIT=30

# Name of our snapshot repository
REPO=my_backup

# Get a list of snapshots that we want to delete
SNAPSHOTS=`curl -s -XGET "localhost:9200/_snapshot/$REPO/_all" \
  | jq -r ".snapshots[:-${LIMIT}][].snapshot"`

# Loop over the results and delete each snapshot
for SNAPSHOT in $SNAPSHOTS
do
 echo "Deleting snapshot: $SNAPSHOT"
 curl -s -XDELETE "localhost:9200/_snapshot/$REPO/$SNAPSHOT?pretty"
done
echo "Done!"

Restore snapshots

Get a list of all the snapshots in the snapshot repository:

curl -s -XGET "localhost:9200/_snapshot/my_backup/_all?pretty"

From that list pick the snapshot id you want to restore and then make a script like this:

#!/bin/bash
#
# Restore a snapshot from our repository
SNAPSHOT=123

# We need to close the index first
curl -XPOST "localhost:9200/my_index/_close"

# Restore the snapshot we want
curl -XPOST "http://localhost:9200/_snapshot/my_backup/$SNAPSHOT/_restore" -d '{
 "indices": "my_index"
}'

# Re-open the index
curl -XPOST 'localhost:9200/my_index/_open'

 

Comments are closed.