Self host Mongodb on AWS with Pulumi with go

Posted on:January 17, 2023 at 09:42 PM

Introduction

According to the db engine rankings, MongoDB is the most popular NoSQL database. With MongoDB, you get the efficiency of NoSQL combined with the flexibility of being able to query inside your documents.

For most use-cases, I would recommend MongoDB Atlas as the best way to use MongoDB. It includes free tier with 512 MB of storage. Your data is the most important part of your organization. You wouldn’t want to worry about security, backups and scaling issues that come with self-hosting.

Why Pulumi?

Pulumi is a universal infrastructure as code platform similar to Terraform that you can use to provision cloud resources using the popular programming languages (TypeScript, Go, .NET, Python, and Java) and markup languages (YAML, CUE). The benefits of using pulumi is that you don’t have to learn a new and restrictive language like HCL (for Terraform).

If you are new to infrastructure as code, I would recommend you to read this blog post on infrastructure as code.

Prerequisites

Let’s begin this step-by-step tutorial on how to self-host MongoDB on AWS with Pulumi.

Create a new Pulumi project

mkdir aws-go
cd aws-go
pulumi new aws-go

This opens up a wizard in your terminal asking about the details.

This command will walk you through creating a new Pulumi project.

Enter a value or leave blank to accept the (default), and press <ENTER>.
Press ^C at any time to quit.

project name: (aws-go)
project description: (A minimal AWS Go Pulumi program)
Created project 'aws-go'

stack name: (dev) tutorial
Created stack 'tutorial'

aws:region: The AWS region to deploy into: (us-east-1) ap-south-1
Saved config

Installing dependencies...

go: downloading github.com/pulumi/pulumi/sdk/v3 v3.51.0
go: downloading github.com/pulumi/pulumi-aws/sdk/v5 v5.27.0
Finished installing dependencies

Your new project is ready to go!

Create mongodb credentials

We will create 2 users in mongodb - one for the admin and one for an application that you can use later. Using openssl to create 2 random strings for the mongodb passwords.

openssl rand -base64 32
openssl rand -base64 32

This outputs for me:

sCwiPkDoReAW1hYdg0vCwLWp08Yqv6G+dFhTxbdHKt0=
3fNAyiSocILN522DwyWBFo3+53rgkSw1jRoPWP9Wf+0=
pulumi config set mongodb:adminPass --secret sCwiPkDoReAW1hYdg0vCwLWp08Yqv6G+dFhTxbdHKt0=
pulumi config set mongodb:appPass --secret 3fNAyiSocILN522DwyWBFo3+53rgkSw1jRoPWP9Wf+0=

For deploying, we are going with the “r6g.large” instance type which has the following features.

NamevCPURAM
r6g.large216
pulumi config set mongodb:instanceType "r6g.large"

Setup AWS CLI

You need AWS credentials to programatically create resources. This has to be done using the AWS UI for the first time. This is out of scope for this tutorial. Please go through AWS docs if you need any help.

Using the credentials that you just generated configure aws cli.

aws configure
AWS Access Key ID [None]: keyId
AWS Secret Access Key [None]: accessKey
Default region name [None]: yourRegionName
Default output format [None]:

Let’s create a new key pair which will allow us to ssh into the instance if required.

aws ec2 create-key-pair --key-name mongodb-key-pair --output text > mongodb-key-pair.pem

Create mongodb installation script

Mongodb is a nosql database and is usually not limited by the CPU. So I am going to use AWS ARM based processors which are cheaper. If you want to deploy to a different architecture, just make the necessary changes in the script, e.g. change arm64 to amd64.

Create a file named mongodb.sh and add the following content to it.

#!/bin/bash
# you might want to update the versions here
export MONGO_PACKAGE=mongodb-org
export MONGO_REPO=repo.mongodb.org
export MONGO_MAJOR=6.0

# Download all the dependencies
set -eux; \
sudo apt-get update; \
sudo apt-get install -y --no-install-recommends ca-certificates

# Recommended by mongodb production guide
echo "vm.swappiness=1" | sudo tee -a /etc/sysctl.conf
echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

# install mongodb
wget -qO - https://www.mongodb.org/static/pgp/server-$MONGO_MAJOR.asc | sudo apt-key add -

echo "deb [ arch=amd64,arm64 ] https://$MONGO_REPO/apt/ubuntu focal/$MONGO_PACKAGE/$MONGO_MAJOR multiverse" | sudo tee /etc/apt/sources.list.d/$MONGO_PACKAGE-$MONGO_MAJOR.list

set -x; \
	export DEBIAN_FRONTEND=noninteractive; \
	sudo apt-get update; \
	sudo apt-get install -y $MONGO_PACKAGE;
	sudo rm -rf /var/lib/mongodb \

# Download yq, useful for parsing yaml files
wget https://github.com/mikefarah/yq/releases/download/v4.28.2/yq_linux_arm64
sudo mv yq_linux_arm64 /usr/local/bin/yq
sudo chmod +x /usr/local/bin/yq
sudo ln -sf /usr/local/bin/yq /usr/bin/yq

# Create data directory at /data/db and log directory at /var/log/mongodb with the correct permissions
set -eux; \
sudo mkdir -p /data/db; \
sudo chown -R mongodb:mongodb /data/db /var/log/mongodb

# Create the mongod.conf file
sudo mv /etc/mongod.conf /etc/mongod.conf.orig
sudo yq '.storage.dbPath = "/data/db"' /etc/mongod.conf.orig | sudo tee /etc/mongod.conf

# Start MongoDB and add it as a service to be started at boot time:
sudo systemctl start mongod
sudo systemctl enable mongod
sudo systemctl restart mongod

# Create users for admin and app with the passwords provided in the config
# %s values will be provided by the program.
mongosh admin --eval "db.createUser({'user':'admin', 'pwd':'%s', 'roles':['root']})"
mongosh admin --eval "use store"
mongosh store --eval "db.createUser({'user':'app', 'pwd': '%s', 'roles':[{'role':'readWrite', 'db': 'store'}]})"

# enable auth after creating users
sudo yq '.security.authorization = "enabled"
| .net.bindIp = "0.0.0.0"
| .storage.dbPath = "/data/db"' /etc/mongod.conf.orig | sudo tee /etc/mongod.conf

# restart mongod service
sudo systemctl restart mongod

Lets write pulumi code now

Create a file named database.go and add the following code to it.

package main

import (
	"fmt"
	"os"

	"github.com/pulumi/pulumi-aws/sdk/v5/go/aws/ec2"
	"github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func CreateDatabase(ctx *pulumi.Context) error {
	// Create a new security group for our database servers.
	// This security group will allow inbound traffic from the internet on port 27017 from all Ips.
	// This is the default port for mongodb.
	mongodbSg, err := ec2.NewSecurityGroup(ctx, "mongodb-sg", &ec2.SecurityGroupArgs{
		Ingress: ec2.SecurityGroupIngressArray{
			ec2.SecurityGroupIngressArgs{
				Protocol:   pulumi.String("tcp"),
				FromPort:   pulumi.Int(27017),
				ToPort:     pulumi.Int(27017),
				CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")},
			},
		},
		Tags: pulumi.StringMap{
			"Name": pulumi.String("mongodb-sg"),
		},
	})
	if err != nil {
		return err
	}

	// Create a new security group to allow ssh access to the database servers.
	sshSg, err := ec2.NewSecurityGroup(ctx, "ssh-sg", &ec2.SecurityGroupArgs{
		Ingress: ec2.SecurityGroupIngressArray{
			ec2.SecurityGroupIngressArgs{
				Protocol:   pulumi.String("tcp"),
				FromPort:   pulumi.Int(22),
				ToPort:     pulumi.Int(22),
				CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")},
			},
		},
		Tags: pulumi.StringMap{
			"Name": pulumi.String("ssh-sg"),
		},
	})
    if err != nil {
		return err
	}

    // Create a new security group to allow public egress access to the database servers.
    // This is required so that our database can connect to the internet to download packages.
    publicSg, err := ec2.NewSecurityGroup(ctx, "public-egress-sg", &ec2.SecurityGroupArgs{
		Egress: ec2.SecurityGroupEgressArray{
			ec2.SecurityGroupEgressArgs{
				Protocol:   pulumi.String("-1"),
				FromPort:   pulumi.Int(0),
				ToPort:     pulumi.Int(0),
				CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")},
			},
		},
		Tags: pulumi.StringMap{
			"Name": pulumi.String("public-egress-sg"),
		},
	})
	if err != nil {
		return err
	}

	ami, err := ec2.LookupAmi(ctx, &ec2.LookupAmiArgs{
		Filters: []ec2.GetAmiFilter{
			{
				Name: "name",
				// 22.04 not yet supported according to mongodb documentation.
				Values: []string{"ubuntu/images/hvm-ssd/ubuntu-focal-20.04-arm64-server-*"},
			},
		},
		Owners:     []string{"099720109477"},
		MostRecent: pulumi.BoolRef(true),
	}, nil)
	if err != nil {
		return err
	}

	mongodbAdminPass, exists := ctx.GetConfig("mongodb:adminPass")
	if !exists {
		return fmt.Errorf("expected mongodb admin password to be set in Pulumi.tutorial.yaml")
	}
	mongodbAppPass, exists := ctx.GetConfig("mongodb:appPass")
	if !exists {
		return fmt.Errorf("expected mongodb app password to be set in Pulumi.tutorial.yaml")
	}

	instanceType, exists := ctx.GetConfig("mongodb:instanceType")
	if !exists {
		return fmt.Errorf("expected mongodb instance type to be set in Pulumi.tutorial.yaml")
	}

    // In AWS lingo, the script that runs when an instance is created is called a user data script.
	userData, err := getUserData()

    // Creating a new instance with the user data script and the security groups we created earlier.
	_, err = ec2.NewInstance(ctx, "mongodb-server-1", &ec2.InstanceArgs{
		Ami:                      pulumi.String(ami.Id),
		AssociatePublicIpAddress: pulumi.Bool(true),
		EbsBlockDevices: ec2.InstanceEbsBlockDeviceArray{
			ec2.InstanceEbsBlockDeviceArgs{
				DeviceName:          pulumi.String("/dev/sda1"),
				DeleteOnTermination: pulumi.Bool(false),
				VolumeSize:          pulumi.Int(100), // 100 GB allocated.
				VolumeType:          pulumi.String("gp3"), // gp3 is the latest and greatest.
				Tags: pulumi.StringMap{
					"Name": pulumi.String("mongodb-server-1"),
				},
			},
		},
		EbsOptimized: pulumi.Bool(true),
		InstanceType: pulumi.String(instanceType),
		KeyName:      pulumi.String("mongodb-key-pair"),
		Tags: pulumi.StringMap{
			"Name": pulumi.String("mongodb-server"),
		},
		VpcSecurityGroupIds: pulumi.StringArray{mongodbSg.ID(), sshSg.ID(), publicSg.ID()},
		UserData:            pulumi.Sprintf(userData, mongodbAdminPass, mongodbAppPass),
	})

	return err
}

func getUserData() (string, error) {
	data, err := os.ReadFile("./mongodb.sh")
	return string(data), err
}

One final step

Run pulumi up to deploy the stack. This will take a few minutes.

We are done. Congratulations!

Conclusion

In this tutorial, we learned how to deploy a MongoDB server on AWS using Pulumi. We also learned how to use Pulumi to create a security group, a key pair, and an instance. We also learned how to use Pulumi to create a user data script to run when the instance is created.

Again, as a reminder, this is not a production-ready setup. You need backups, monitoring, and deploy atleast 3 mongodb instances in a replica set. This is just a simple tutorial to get you started with Pulumi.