Automatically downloading GotRoot / AtomiCorp mod_security WAF rules

Phew, the heading of this post was a mouthful!

This post is a technical one, that probably won't interest many of my regular readers. However I know there are plenty of people looking for how to do this, so I thought I'd post this in case it helps others later.

Background

mod_security is a free or charge module for the Apache webserver. It allows HTTP requests to be analyzed, using a large number of characteristics. You'd be looking for evidence that someone's request to GET or POST is malicious, and you'd block them if so.

However mod_security, by itself, does nothing. It needs a set of rules to tell it what to look for and how to behave if there is a match.

Writing your own rules is not a trivial task. Simple rules are easily written - it's no trouble to block all requets that have a particular user-agent, for example. However, to come up with a comprehensive ruleset that matches most known malware without triggering false positives is much harder.

So, there are various ways to obtain rules to use. cPanel ships with some default rules, although this does not catch much. You can download free rules from OWASP, although the consensus seems to be that these trigger a fair number of false-positives, so some careful tuning is needed to prevent this. The team behind mod_security release some commercial rules. I've not tried those, but I've also read no major criticisms of them. In my view, the best rules currently available are published by AtomiCorp, and are the rules formerly known as the GotRoot ruleset.

Keeping up to date is non-trivial

The challenge is to keep your rules up to date. New rules are released daily, sometimes several times each day.

AtomiCorp don't supply a script that will automatically download new rules and apply them to your Apache installation. There are two reasons for this - one commercial and one technical.

The commercial reason is that as well as selling access to their rules, they also sell another product called "Atomic Secured Linux" (ASL). ASL does much, much more than just the mod_security ruleset. By all accounts, it's a highly efficient security suite. It comes with a mechanism to keep itself up to date. So AtomiCorp take the line that you should only subscribe to their mod_security rules if you are willing to maintain them yourself; if you need this doing for you, you should use their ASL suite instead.

The technical reason is that there is more to keeping rules up to date than just downloading a file. Sometimes new rules depend on a new version of mod_security itself. So sometimes mod_security needs updating before the new rules are loaded. Therefore writing such a script is not actually a trivial task.

According to a post in the AtomiCorp forums, they are now developing such a script that will be made freely available when it's ready.

In the meantime, however, what follows may help some people.

What this script does and does not do

In a minute, I will paste a bash script that will download the latest rules and apply them.

Before I do so, please note a number of things:

1. This script does not install mod_security. You would need to install mod_security (by compiling from source, from a repository, or from a tool like cPanel's EasyApache) first.
2. This script does not configure mod_security to use these rules. AtomiCorp have detailed instructions on how to do this. These instructions must be followed carefully, and the installation must be tested. You should download the rules manually this first time to make sure they are in the right place. Part of what you need to do is to put a configuration file where mod_security will find it, and then use that file to call whichever of the AtomiCorp rules files you wish.
3. This script does not check for compatibility with your version of mod_security. If mod_security needs updating, that would be something you would need to do yourself.

So here's what it does do:

1. Download the VERSION file that AtomiCorp use to identify that latest timestamp for their mod_security rules.
2. Compare the timestamp in the VERSION file with the version of the rules you have installed.
3. If you are not using the latest version, it will download and untar the latest version.
4. Take a backup of your current mod_security rules files.
5. Replace your mod_security rules with the ones just downloaded
6. Check the resulting Apache configuration to make sure Apache will start with these new rules. (This should pick up any conflicts with your current version of mod_security).
7. If everything is OK, Apache is restarted.
8. If there is a problem with the configuration, Apache is not restarted and the backup rules are put back.

There's a key assumption in here: It is assumed that you want the AtomiCorp rules to go into a subdirectory of your Apache configuration folder named modsec_rules. This is almost always arrangeable, as you'll be linking to these files from another configuration file in the modsecurity.d folder (for repository-installed Apache) or in your modsec2.user.conf file (for cPanel-installed Apache).

Disclaimer Time!

This script comes without any warranty, including fitness for any particular purpose. You use this at your own risk. I cannot be held liable if using this damages your server, causes loss of data, overwrites wanted configuration files, or starts a world war.

The Script

Finally, here's my script.

It's possible that some lines may have wrapped round, causing problems, so I've attached it to this post as a .txt file. If you use the download, rename it to .sh, and you may need to run dos2unix on it as well. Don't forget to chmod +x.

#!/bin/bash

# ____________________________________________________________________________
#
#       Copyright 2013 James Oakley, www.oakhosting.net
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
# ____________________________________________________________________________

# ____________________________________________________________________________
#
# This script automates downloading WAF mod_security rules from
# atomicorp.com. It comes with no warranty, actual or implied.
# It does NOT install these rules to start with.
# Follow their documentation to create the necessary directories
#
# This script assumes the rules go in a modsec_rules directory within the
# Apache configuration directory. Link to the ones you want in the main
# file you use to set the mod_security configuration.
# ____________________________________________________________________________

# ____________________________________________________________________________
#
#   -- Variables section --
# You need to set three variables for this script to work.
# 1. The directory where the Apache configuration files are to be found
#   (The default value given is for a cPanel installation.)

CONF_DIR=/usr/local/apache/conf

# 2. Your username for the atomicorp.com website.
#   (This will be the username (not e-mail) you gave, usually lower-case)

ASL_USERNAME={your-username-here}

# 3. Your password for the atomicorp.com website.

ASL_PASSWORD={your-password-here}

#
# ____________________________________________________________________________

# Now the script itself begins

# Step 1: Retrieve the version number of the latest stable rules
cd /usr/local/src
echo "`date +\%Y-\%m-\%d\ \%H:\%M:\%S` -- Retrieving latest version number"
wget --quiet --user=$ASL_USERNAME --password=$ASL_PASSWORD http://updates.atomicorp.com/channels/rules/subscription/VERSION
echo "`date +\%Y-\%m-\%d\ \%H:\%M:\%S` -- Version number retrieved"

# Step 2: Compare this retrieved version to the version of the existing rules installed
# If there are no rules installed, or if there is no version file within them, new rules will be downloaded
if [ -e VERSION ]; then
  source /usr/local/src/VERSION
  OLD_VERSION="n/a"
  if [ -e $CONF_DIR/modsec_rules/version.txt ]; then
    OLD_VERSION=$(cat $CONF_DIR/modsec_rules/version.txt)
  fi
  echo "`date +\%Y-\%m-\%d\ \%H:\%M:\%S` -- Old Version: $OLD_VERSION"
  echo "`date +\%Y-\%m-\%d\ \%H:\%M:\%S` -- New Version: $MODSEC_VERSION"
  if [ "$OLD_VERSION" = "$MODSEC_VERSION" ]; then
    echo "`date +\%Y-\%m-\%d\ \%H:\%M:\%S` -- Version is unchanged - nothing to do"
  else

# Step 3: If we need to download a new version, we download and then extract
    wget --quiet --user=$ASL_USERNAME --password=$ASL_PASSWORD http://updates.atomicorp.com/channels/rules/subscription/modsec-$MODSEC_VERSION.tar.gz
    if [ -e modsec-$MODSEC_VERSION.tar.gz ]; then
      tar -xzf modsec-$MODSEC_VERSION.tar.gz

# Step 4: Back up the old rules
# This is done by creating directories in the Apache configuration folder
# These are named modsec_rules_bakN, and up to 4 old copies are retained
      echo "`date +\%Y-\%m-\%d\ \%H:\%M:\%S` -- Backing up old rules"
      if [ -e $CONF_DIR/modsec_rules_bak4 ]; then
        rm $CONF_DIR/modsec_rules_bak4 -rf
      fi
      if [ -e $CONF_DIR/modsec_rules_bak3 ]; then
        mv $CONF_DIR/modsec_rules_bak3 $CONF_DIR/modsec_rules_bak4 -f
      fi
      if [ -e $CONF_DIR/modsec_rules_bak2 ]; then
        mv $CONF_DIR/modsec_rules_bak2 $CONF_DIR/modsec_rules_bak3 -f
      fi
      if [ -e $CONF_DIR/modsec_rules_bak1 ]; then
        mv $CONF_DIR/modsec_rules_bak1 $CONF_DIR/modsec_rules_bak2 -f
      fi
      mkdir $CONF_DIR/modsec_rules_bak1

# Step 5: Move the old rules into the _bak1 directory
# Then move the new rules into the modsec_rules directory
# As we do so, we create a version file that contains the version number of these rules
# This helps if support is needed from atomicorp.
# This is also what we will use to avoid downloading the same version again later.
      mv $CONF_DIR/modsec_rules/* $CONF_DIR/modsec_rules_bak1 -f
      echo "`date +\%Y-\%m-\%d\ \%H:\%M:\%S` -- Moving new rules into place"
      mv modsec/* $CONF_DIR/modsec_rules
      echo $MODSEC_VERSION > $CONF_DIR/modsec_rules/version.txt

# Step 6: Run Apache's configtest instruction,
# to check that there are no errors in our configuration.
# It's too risky to restart Apache without checking first.
      echo "`date +\%Y-\%m-\%d\ \%H:\%M:\%S` -- Testing Apache Configuration"
      APACHECHK=` /etc/init.d/httpd configtest 2>&1 | grep "Syntax OK"`
      if [[ $APACHECHK = "Syntax OK" ]]; then

# Step 7: If there are no errors, we restart Apache.
        echo "`date +\%Y-\%m-\%d\ \%H:\%M:\%S` -- Restarting Apache"
        /etc/init.d/httpd restart
        echo "`date +\%Y-\%m-\%d\ \%H:\%M:\%S` -- Apache Restarted"
        echo "`date +\%Y-\%m-\%d\ \%H:\%M:\%S` -- Rules update completed"

# Step 8: If there were errors, we use our backup directories to roll back
# Note: If we had previously used all 4 _bakN directories, we will have lost
# _bak4 in the process, but bak1_ to bak3_ will be as they were before.
      else
        echo "`date +\%Y-\%m-\%d\ \%H:\%M:\%S` -- Configuration Test Failed. Error follows:"
        /etc/init.d/httpd configtest 2>&1
        echo "`date +\%Y-\%m-\%d\ \%H:\%M:\%S` -- Rolling Back"
        rm $CONF_DIR/modsec_rules -rf
        mv $CONF_DIR/modsec_rules_bak1 $CONF_DIR/modsec_rules -f
        if [ -e $CONF_DIR/modsec_rules_bak2 ]; then
          mv $CONF_DIR/modsec_rules_bak2 $CONF_DIR/modsec_rules_bak1 -f
        fi
        if [ -e $CONF_DIR/modsec_rules_bak3 ]; then
          mv $CONF_DIR/modsec_rules_bak3 $CONF_DIR/modsec_rules_bak2 -f
        fi
        if [ -e $CONF_DIR/modsec_rules_bak4 ]; then
          mv $CONF_DIR/modsec_rules_bak4 $CONF_DIR/modsec_rules_bak3 -f
        fi

# Step 9: Run Apache's configtest to screen.
# We do not restart Apache, but simply advise the user of the result of the test
# There may be other reasons why Apache's configuration is broken.
# If the rules update did not work, it is safer to leave Apache running while
# the user checks what the issues were.
        echo "`date +\%Y-\%m-\%d\ \%H:\%M:\%S` -- Rerunning Apache Configuration Test"
        /etc/init.d/httpd configtest 2>&1
        echo "`date +\%Y-\%m-\%d\ \%H:\%M:\%S` -- Rules updated failed"
        echo "`date +\%Y-\%m-\%d\ \%H:\%M:\%S` -- APACHE HAS NOT BEEN RESTARTED. CHECK THE CONFIGTEST RESULT ABOVE BEFORE DOING SO"
      fi

# Step 10: We print out the first 5 lines of modsec_rules
# This allows the user to sanity-check the datestamps of the files
# These should match the version number that has been downloaded.
      echo
      echo "`date +\%Y-\%m-\%d\ \%H:\%M:\%S` -- Start of modsec_rules directory follows for reference"
      ls $CONF_DIR/modsec_rules -l | head -n5
    fi
  fi

# Step 11: Remove the modsec download (if it was downloaded)
  rm -rf modsec modsec-$MODSEC_VERSION.tar.gz
fi

# Step 12: Remove the VERSION file and any sensitive environment variables
rm -f /usr/local/src/VERSION
unset ASL_USERNAME
unset ASL_PASSWORD