Automate Deployment with CloudFormation

CloudFormation from Amazon Web Services provides an easy mechanism to create and manage a collection of AWS resources. To use CloudFormation you create and deploy a template which describes the resources in your stack via the AWS Management Console. CloudFormation templates are simple JSON formatted text files that can be placed under your normal source control mechanisms, stored in private or public locations such as Amazon S3.

MongoDB has a series of reference templates that you can use as a starting point to build your own MongoDB deployments using CloudFormation. The sample templates show how to build a single node MongoDB deployment and a MongoDB replica set deployment. Refer to the following sections for information on how these sample templates were developed and instructions on how to customize them for your own deployment onto AWS. The documentation provided below focuses on the sections specific to deploying MongoDB on AWS, for background on standard template sections (e.g. input parameters) refer to the AWS CloudFormation User Guide.

The sample templates are as follows (click to download):

Template Walkthrough

Security

EC2 instances in AWS must have security groups designated during creation that specify firewall rules for each instance. The templates create a simple security group MongoSecurityGroup that opens up port 22 (SSH) to the outside world and a separate rule (AWS::EC2::SecurityGroupIngress) that is used to open up the standard mongod port (27017) to other instances within that group:

"MongoSecurityGroup" : {
    "Type" : "AWS::EC2::SecurityGroup",
    "Properties" : {
        "GroupDescription" : "MongoDB security group",
        "SecurityGroupIngress" : [
            {
                "IpProtocol" : "tcp",
                "FromPort" : "22",
                "ToPort" : "22",
                "CidrIp" : "0.0.0.0/0"
            }
        ]
    }
},
"MongoSecurityGroupIngress" : {
    "Type" : "AWS::EC2::SecurityGroupIngress",
    "Properties" : {
        "GroupName" : { "Ref" : "MongoSecurityGroup" },
        "IpProtocol" : "tcp",
        "FromPort" : "27017",
        "ToPort" : "27017",
        "SourceSecurityGroupName" : { "Ref" : "MongoSecurityGroup" }
    }
},

Depending on the type of deployment you are creating, you may need to add additional security group ingress rules to open up additional ports, available to your instances or to the outside world (e.g. port 27018 for sharding).

Storage Configuration

When using MongoDB on AWS, it is recommended that you use multiple EBS volumes configured as a single RAID10 storage device for your data. Configuring storage via EBS volumes using CloudFormation requires multiple steps. First the EBS volume must be created, using the VolumeSize input value and the same availability zone to be used for the EC2 instance (see Instance Configuration).

"MongoVolume1" : {
    "Type" : "AWS::EC2::Volume",
    "Properties" : {
        "Size" : { "Ref" : "VolumeSize" },
        "AvailabilityZone" : { "Fn::GetAtt" : [ "MongoInstance", "AvailabilityZone" ]}
    }
},

The next step is to attach the volume to an EC2 instance. Referencing the instance name in InstanceId ensures that the EBS volumes will be created after the instance is created.

"MongoVolumeMount1" : {
    "Type" : "AWS::EC2::VolumeAttachment",
    "Properties" : {
        "InstanceId" : { "Ref" : "MongoInstance" },
        "VolumeId" : { "Ref" : "MongoVolume1" },
        "Device" : "/dev/sdh1"
    }
},

The templates use four EBS volumes as the basis for the RAID10 configuration. If you are interested in increasing the number of EBS volumes, you will need to add additional AWS::EC2::Volume and AWS::EC2::VolumeAttachment resources inside of your template. See Customize Templates for more information on the steps required.

Instance Configuration

The centerpiece of the CloudFormation template is the creation of the EC2 instances that will be used as the MongoDB server. There are two main sections to be configured for your instance, the instance metadata and instance properties. In the sample provided, the metadata contains information about the packages to be installed (mdadm and sysstat) and any files to be created within the instance. In the sample, the only file to be created is a yum repository entry to facilitate MongoDB being installed via yum after the instance has booted.

"MongoInstance" : {
    "Type" : "AWS::EC2::Instance",
    "Metadata" : {
        "AWS::CloudFormation::Init" : {
            "config" : {
                "packages" : {
                    "yum" : {
                        "mdadm" : [],
                        "sysstat" : []
                    }
                },
                "files" : {
                    "/etc/yum.repos.d/10gen.repo" : {
                        "content" : { "Fn::Join" : ["", [
                            "[10gen]\n",
                            "name=10gen Repository\n",
                            "baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64\n",
                            "gpgcheck=0\n"
                        ] ] },
                        "mode" : "000644",
                        "owner" : "root",
                        "group" : "root"
                    }
                }
            }
        }
    },

For more information about the possible options for the metadata section, refer to the CloudFormation documentation for the AWS::EC2::Instance resource. The properties section in the template is uses to specify things like the instance type, AMI, security groups and a script to be run after boot (found in the UserData section):

"Properties" : {
    "InstanceType" : { "Ref" : "InstanceType" },
    "ImageId" : { "Fn::FindInMap" : [ "RegionImageZone", { "Ref" : "AWS::Region" },
        { "Fn::FindInMap" : [ "InstanceTypeArch", { "Ref" : "InstanceType" }, "Arch" ] } ] },
    "SecurityGroups" : [ { "Ref" : "MongoSecurityGroup" } ],
    "KeyName" : { "Ref" : "KeyName" },
    "UserData" : { "Fn::Base64" : { "Fn::Join" : ["", [
        "#!/bin/bash\n",
        ...

The instance type is determined by the InstanceType input parameter. The ImageId specifies the AMI to use, which is determined by the chosen instance type (e.g. m1.large) and region in which the instance is launched (e.g. us-east-1). Refer to the Mappings section inside the sample templates for a list of the available AMIs.

The UserData section shown above contains a bash script that is executed once the instance is launched and available. The first step is to install the aws-cfn-bootstrap tools which are used in the script to initialize the instance, signal when errors have occurred, and signal when instance creation is complete:

"yum update -y aws-cfn-bootstrap\n",

"## Error reporting helper function\n",
"function error_exit\n",
"{\n",
"   /opt/aws/bin/cfn-signal -e 1 -r \"$1\" '", { "Ref" : "WaitHandleMongoInstance" }, "'\n",
"   exit 1\n",
"}\n",

"## Initialize CloudFormation bits\n",
"/opt/aws/bin/cfn-init -v -s ", { "Ref" : "AWS::StackName" }, " -r MongoInstance",
"   --access-key ",  { "Ref" : "HostKeys" },
"   --secret-key ", {"Fn::GetAtt": ["HostKeys", "SecretAccessKey"]},
"   --region ", { "Ref" : "AWS::Region" }, " > /tmp/cfn-init.log 2>&1 || error_exit $(</tmp/cfn-init.log)\n",

Next include a series of sleep conditions in case the EBS volumes are not yet available. If you plan to use more than 4 EBS volumes in your CloudFormation template, you should add additional sleep conditions here:

"## Waiting for EBS mounts to become available\n",
"while [ ! -e /dev/sdh1 ]; do echo waiting for /dev/sdh1 to attach; sleep 10; done\n",
"while [ ! -e /dev/sdh2 ]; do echo waiting for /dev/sdh2 to attach; sleep 10; done\n",
"while [ ! -e /dev/sdh3 ]; do echo waiting for /dev/sdh3 to attach; sleep 10; done\n",
"while [ ! -e /dev/sdh4 ]; do echo waiting for /dev/sdh4 to attach; sleep 10; done\n",

Then install MongoDB and create the RAID10 device:

"yum -y install mongo-10gen-server > /tmp/yum-mongo.log 2>&1\n",

"## Create RAID10 and persist configuration\n",
"mdadm --verbose --create /dev/md0 --level=10 --chunk=256 --raid-devices=4 /dev/sdh1 /dev/sdh2 /dev/sdh3 /dev/sdh4 > /tmp/mdadm.log 2>&1\n",
"echo '`mdadm --detail --scan`' | tee -a /etc/mdadm.conf\n",

With the RAID10 created, set some block device attributes (read-ahead) for each storage device:

"## Set read-ahead on each device\n",
"blockdev --setra 128 /dev/md0\n",
"blockdev --setra 128 /dev/sdh1\n",
"blockdev --setra 128 /dev/sdh2\n",
"blockdev --setra 128 /dev/sdh3\n",
"blockdev --setra 128 /dev/sdh4\n",

Now use LVM to create a series of logical volumes for data, journal and logs. The values used below for each volume are specified as percentages, those may need to be changed for your deployment. After creating the volumes, next create the filesystem, mount points and entries in the filesystem table. The last storage-related step is to set the user:group ownership of each mount point to mongod:mongod.

"## Create physical and logical volumes\n",
"dd if=/dev/zero of=/dev/md0 bs=512 count=1\n",
"pvcreate /dev/md0\n",
"vgcreate vg0 /dev/md0\n",
"lvcreate -l 90%vg -n data vg0\n",
"lvcreate -l 5%vg -n log vg0\n",
"lvcreate -l 5%vg -n journal vg0\n",

"## Create filesystems and mount point info\n",
"mke2fs -t ext4 -F /dev/vg0/data > /tmp/mke2fs1.log 2>&1\n",
"mke2fs -t ext4 -F /dev/vg0/log > /tmp/mke2fs2.log 2>&1\n",
"mke2fs -t ext4 -F /dev/vg0/journal > /tmp/mke2fs3.log 2>&1\n",

"mkdir /data\n",
"mkdir /log\n",
"mkdir /journal\n",

"echo '/dev/vg0/data /data ext4 defaults,auto,noatime,noexec 0 0' | tee -a /etc/fstab\n",
"echo '/dev/vg0/log /log ext4 defaults,auto,noatime,noexec 0 0' | tee -a /etc/fstab\n",
"echo '/dev/vg0/journal /journal ext4 defaults,auto,noatime,noexec 0 0' | tee -a /etc/fstab\n",

"mount /data > /tmp/mount1.log 2>&1\n",
"mount /log > /tmp/mount2.log 2>&1\n",
"mount /journal > /tmp/mount3.log 2>&1\n",

"ln -s /journal /data/journal\n",

"chown -R mongod:mongod /data > /tmp/chown1.log 2>&1\n",
"chown -R mongod:mongod /log > /tmp/chown2.log 2>&1\n",
"chown -R mongod:mongod /journal > /tmp/chown3.log 2>&1\n",

Next proceed to creating a MongoDB configuration file, specifying the logpath and data directory (among others), and then start MongoDB:

"## Update mongod configuration\n",
"cat <<EOF > /etc/mongod.conf\n",
"logpath=/data/log/mongod.log\n",
"logappend=true\n",
"fork = true\n",
"dbpath=/data\n",
"rest=true\n",
"EOF\n",

"## Start mongod\n",
"/etc/init.d/mongod start > /tmp/mongod-start.log 2>&1\n",

The final step is to signal the previously created WaitCondition that setup is complete:

"## CloudFormation signal that setup is complete\n",
"/opt/aws/bin/cfn-signal -e 0 -r \"MongoInstance setup complete\" '", { "Ref" : "WaitHandleMongoInstance" }, "'\n"

Once this script has completed executing, the instance and associated resources have been created and the CloudFormation stack is ready to go.

Replica Set Stack

The “ReplicaSetStack” sample template first creates two “ReplicaSetMember” instances (complete with storage configuration) and then creates the overall replica set. The “ReplicaSetMember” template is modeled very closely after the “SingleNode” template except it includes additional input parameters and adds additional commands to the instance setup script in UserData specific to creating replica set members (adding in the replSet parameter to the MongoDB configuration file).

"## Update mongod configuration\n",
"cat <<EOF > /etc/mongod.conf\n",
"logpath=/data/log/mongod.log\n",
"logappend=true\n",
"fork = true\n",
"dbpath=/data\n",
"rest=true\n",
"replSet=", {"Ref" : "ReplicaSetName"}, "\n",
"EOF\n",

The “ReplicaSetStack” template also closely follows the “SingleNode” template but adds the following replica set specific steps:

  1. It creates a replicaSetConfigInit.js file containing the replica set configuration (with hostnames for the additional members).
  2. It initiates the replica set. These steps are executed just prior to signaling that the instance setup has been completed.
"cat <<EOF > /tmp/replicaSetConfigInit.js\n",
"config = {_id: \"", { "Ref" : "ReplicaSetName" } ,"\", members : [",
    "{_id : 0, host:\"$HOSTNAME:27017\"},",
    "{_id : 1, host:\"", { "Fn::GetAtt" : ["ReplicaSetMember1", "Outputs.ReplicaSetMemberName"] },":27017\"},",
    "{_id : 2, host:\"", { "Fn::GetAtt" : ["ReplicaSetMember2", "Outputs.ReplicaSetMemberName"] },":27017\"}",
"]};\n",
"rs.initiate(config);\n",
"EOF\n",

"/usr/bin/mongo /tmp/replicaSetConfigInit.js > /tmp/replica-setup.log 2>&1\n",

The child “ReplicaSetMember” instances are created from within the “ReplicaSetStack” template using the following resource definition. The inputs for the “ReplicaSetMember” instances are taken from the “ReplicaSetStack” template:

"ReplicaSetMember1" : {
    "Type" : "AWS::CloudFormation::Stack",
    "Properties" : {
        "TemplateURL" : "http://S3URL/MongoDB_ReplicaSetMember.template",
        "Parameters" : {
            "KeyName" : { "Ref" : "KeyName" },
            "InstanceType" : { "Ref" : "InstanceType" },
            "VolumeSize" : { "Ref" : "VolumeSize" },
            "AccessKeyId" : { "Ref" : "AccessKey" },
            "SecretAccessKey" : {"Fn::GetAtt": ["AccessKey", "SecretAccessKey"]},
            "ReplicaSetName" : { "Ref" : "ReplicaSetName" },
            "SecurityGroupName" : { "Ref" : "MongoSecurityGroup" },
            "InstanceZone" : { "Fn::FindInMap" : ["RegionZone", { "Ref" : "AWS::Region"}, "AZ1"] }
        }
    }
}

Customize Templates

Operating System

The sample templates provided use Amazon Linux as the base operating system. If you are interested in using a recent release of Ubuntu (9.10 or greater), please use the following steps to customize the templates. The steps refer to the line numbers for the “MongoDB_ReplicaSetStack.template” but changes should also be made in “MongoDB_ReplicaSetMember.template.”

First, update the files content of the EC2 instance Metadata section to use the 10gen apt source:

"files" : {
    " /etc/apt/sources.list.d/10gen.list" : {
        "content" : { "Fn::Join" : ["", [
            "deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen\n"
        ] ] },
        "mode" : "000644",
        "owner" : "root",
        "group" : "root"
    }
}

Then in the UserData section of the EC2 instance configuration, make the following changes:

  • Line 220: change yum -y install mongo-10gen-server... to apt-get install mongodb-10gen
  • Line 221: add service mongodb stop
  • Line 260, 261, 262: update to use mongodb:mongodb as the owner:group
  • Line 265: update to cat <<EOF > /etc/mongodb.conf
  • Line 275: change /etc/init.d/mongod start... to service mongodb start

Storage

The sample templates provided use four EBS volumes as the basis for the RAID10 configuration. If you are interested in using additional volumes you will need to update the following items for each new volume you want to add:

  • Add an “AWS::EC2::Volume” resource
  • Add an “AWS::EC2::VolumeAttachment” resource and mount point
  • Add an additional sleep condition
  • Update the call to mdadm to include your additional volumes

Instances

The sample “ReplicaSetStack” template creates three instances, one from the “stack” template and two additional “ReplicaSetMember” instances, via the “AWS::CloudFormation::Stack” resource in the “ReplicaSetStack” template. If you are interested in adding additional replica set members, you will need to create an additional member instances and edit the replicaSetConfigInit.js found within the “ReplicaSetStack” template. Refer to Replica Set Stack for information about the additional resources and config file to be updated. When creating the replica set the templates spread the created instances across multiple availability zones (e.g. us-east-1a or us-east-1b). When adding additional instances be sure to specify your desired Availability Zone for increase redundancy.

Sample Template Usage

If you are interesting in launching a single MongoDB node in AWS, download the “MongoDB_SingleNode.template” file and edit it for your specific deployment. Once you have a completed template, login to the AWS Management Console and navigate to “AWS CloudFormation” and “Create New Stack.” There you will be prompted to upload your template and input the necessary parameters.

If instead you are interested in launching a multi-node replica set, download the “MongoDB_ReplicaSetStack.template” and “MongoDB_ReplicaSetMember.template.” In order for a parent template (“ReplicaSetStack”) to refer to child templates (“ReplicaSetMember”), the child template must be uploaded to S3 and the S3 URL of the child template must be specified in the parent template. Once you have uploaded the child template to S3, update the “TemplateURL” parameter in each “ReplicaSetMember” resource in the “ReplicaSetStack” template:

"ReplicaSetMember1" : {
    "Type" : "AWS::CloudFormation::Stack",
    "Properties" : {
        "TemplateURL" : "http://S3URL/MongoDB_ReplicaSetMember.template",
        "Parameters" : {
            "KeyName" : { "Ref" : "KeyName" },
            "InstanceType" : { "Ref" : "InstanceType" },
            "VolumeSize" : { "Ref" : "VolumeSize" },
            "AccessKeyId" : { "Ref" : "AccessKey" },
            "SecretAccessKey" : {"Fn::GetAtt": ["AccessKey", "SecretAccessKey"]},
            "ReplicaSetName" : { "Ref" : "ReplicaSetName" },
            "SecurityGroupName" : { "Ref" : "MongoSecurityGroup" },
            "InstanceZone" : { "Fn::FindInMap" : ["RegionZone", { "Ref" : "AWS::Region"}, "AZ1"] }
        }
    }
}

After updating the TemplateURL parameters, login to the AWS Management Console and navigate to the “AWS CloudFormation” and “Create New Stack.” There you will be prompted to upload your template and input the necessary parameters.

For more information on deploying MongoDB on AWS, refer to the Amazon EC2 page and the Install MongoDB on Amazon EC2.