Measurable Indifference

Following on from my last post, I mentioned that we would be looking to move from Elastic Beanstalk to EC2 spot requests. EB was useful. Initially, it was a great starting point. But after our foray into Elastic Beanstalk to get the website live, I've come to the following conclusion:
EB is for people with new projects that just want to get started and don't much care how anything works or how much it costs[^1].
I’m not pointing fingers because this was me not too long ago. Something I’ve learned since leaving full-time work is that it pays to know how to actually put it together yourself so that you can cut out the bits that you don't need and minimise the costs that you can.

Here’s what I did…

In my last post, we distilled the dizzying morass that was our AWS billing page into 3 distinct actions we can take to reduce our total spending on the website.

• Remove all static IPv4 addresses

• Decommission the Network Load Balancer that was doing nothing but routing https to http and handling the SSL for us

• Move from Elastic Beanstalk to a spot fleet to host the Django app

The last three months of AWS bills has felt like a warm hug from my wallet

Now you might be thinking, “well, yeah of course it costs less, we now have to do all the server management and scaling ourselves!” To which I say the short answer is yes… and no… twice…

Yes, it’s a bit more work: we will need to manage the server configuration now. But that’s good! we want to know exactly what’s running on our instance (instead of vaguely knowing that Django is being used to handle http requests somehow).

And “no”: The whole server configuration can be defined in a single script that gets run every time the server starts. No ongoing management required, just set and forget.

Yes, scaling is something to manage: If we were managing any kind of load on my website, we would have to set up scaling rules.

But “no” because: Until we start seeing the thousands of readers and clients that my mum says I deserve, we can stick to the smallest morsel of compute power that AWS can charge for.

Server Configuration in one file:

First, we switched off the EB which removed all the resources in commissioned including the expensive Load Balancer and the unnesesary static IP addresses.

We then needed to define our own basic VPC. This can be just a standard public VPC with an internet gateway and 3 public subnets, one in each ap-southeast-2 availability zone (we have no backend database so public subnets will do just fine for our purposes).

Next, we moved the server to a spot fleet request (‘fleets’ can be configured to be 1 instance). The request uses a configuration object called a ‘launch template’ containing all our capacity and compute preferences and configuration information.

The spot request will look for the cheapest of any t2.micro, t2.small, t3.micro, or t3.small available in any AZ in ap-southeast-2 and assign it to the appropriate subnet in our VPC. The instance is then automatically given a public IP address by our VPC.

The launch template also includes a "user data" section that is executed every time the server instance starts up. If you’re more familiar with Docker, you can think of the ‘launch template’ as AWS lingo for a dockerfile and the confusingly named "user data" section is an entrypoint script that gets called in your image's dockerfile. The point is, all the code in “user data” gets run. First thing. As soon as our instance is running.
This is very important because spot instances will get interrupted semi-regularly by aws (I've found mine are replaced at least once per day).

Every time our EC2 instance changes, our “user data” should perform the following actions:

1. Find the instance’s public IP address (These first three steps are required because we now have no load balancer and therefore no static IP address to always route traffic to. So whenever a new server is started, we have to directly edit the A record in our DNS settings to tell the internet that the website's host server is now here).

2. Change our Route53 A record to set samjdrew.com to go to this new instance’s public IP.

3. Search our code packages S3 bucket for the latest version of the website and unzip it.

4. Install Python 3.11 and NGINX.

5. In the unzipped source code, activate Django's python venv and install requirements.

6. Configure gunicorn daemon workers to run the Django app.

7. Start gunicorn and NGINX.

8. use certbot-nginx to assign a free SSL certificate and set up https on nginx [²].

The interested reader can check out the actual code over on my website repo.

Do diagrams help? In my head, it’s simpler than this makes it look.

This is everything we need. And that’s how I’ve left it for the past 3 months. Of course, now and again, I check in on the website as often as my ego prompts me to but it’s been up every time I’ve checked. I half expected there to be some disarray somehow manifesting from my negligence but it looks like we’ve written a good system.

Let’s briefly consider scalability

The magic word of cloud platform engineering - scalability - is a term thrown around so much, you’d think it’s a requirement for any cloud-hosted solution. There’s a section in chapter one of Martin Kleppmann’s Designing Data-Intensive Applications:

[Scalability] is not a one-dimensional label that we can attach to a system: it is meaningless to say “X is scalable” or “Y doesn’t scale.” Rather, discussing scalability means considering questions like “If the system grows in a particular way, what are our options for coping with that growth?” and “How can we add computing resources to handle the additional load?” -

The rest of the section goes on to point out that before we discus scalability, we need to have a way to measure load on the system and the performance of the system. We also need to know what our performance goals for the system are. Do we want the 99.9th percentile to experience webpage load times of less than 1 second? Is that doable? How much work will that take? What about the 99.99th percentile?

In a commercial setting, this makes a lot of sense. Having a way to reduce your cloud spending when traffic is low and only increase when demand for your service goes up is the Great Cloud Dream™. Scaling up and/or out at peak volume periods and scaling back down/in when traffic usually dies down is a neat, effective way to minimise wasted resources and is one of the major selling points of Cloud resources over on-prem. Autoscaling is the sledgehammer approach and is only really appropriate when you’re dealing with unpredictable loads.

Scalability is a tool - or maybe a power-tool, to present a more visual analogy. It can and should be used but only AFTER:

1. Defining how you’ll measure performance and load

2. Using those definitions to define your Service Level Objectives (SLO and SLA if you’re working with a client)

3. Setting up billing alerts (which should be done as soon as you set up any cloud hosting account that’s tied to your wallet)

For this website, there’s no sense in implementing an autoscaling group with a maximum instance count greater than 1 because our 1 instance should be able to handle hundreds of requests per second.
As far as we’re concerned, we can be content in the knowledge that the small handful of clients or employers who bother to click on our website are getting our website.

The left-overs

There were some issues beneath the surface that I haven’t addressed:

• Github actions were broken since moving away from Elastic Beanstalk meant we had to come up with a way of getting my source code onto the server (this was particularly embarrassing for me to ignore for so long as someone who once had DevOps in his job title).

  • This is fixed now. After merging to master, github actions sends a zipped package of the code to an S3 bucket, removes the “-latest” suffix from the old package and appends it to the new one, temporarily scales the spot fleet to 2 instances and then terminates the old instance after 3 minutes. The new instance handles all the server set up and domain redirection as described above and a new version of our website is live about 4 minutes after we merge to master.

• The pages on certifications and experience still haven’t been implemented, which is hilarious for a website that’s supposed to be a portfolio landing page

• we don’t talk about the photography website.

Jumping back into this project after three months, after not thinking about or working on anything Python/Django, I found it daunting trying to remember how Django organises itself. Python/Django has taken a sharp nose-dive in my list of priorities and unfortunately, the website has been in limbo because of that.

I’ll always be intrigued by cloud platforms and automating build systems. However, in my spare time, I’ve been trying to expand into a few new areas that I’m not familiar enough with.

In the past 3 months, I've:

• learned to use Cmake/ninja to build large C projects

• been re-learning C/C++ and assembly

• Learned to draw a triangle in OpenGL

• reading:

  • Martin Kleppmann's Designing Data-Intensive systems and

  • Thorsten Ball’s Writing an Interpreter in Go

• Been working on my own indie game in Godot 4

The point is, everything I learned about django/python and HTML/CSS while throwing this website together this year is buried under several dense layers of unrelated CS stuff that I had to sift through to write this. The Python/Django atrophy has begun and I don’t want to strain myself trying to strengthen those muscles again when it’s no longer a thing I need for my job and I have other priorities preceding it.

This isn’t the post for discussing learning goals and professional growth[³]. I just wanted to wrap up the last outstanding thing I said I’d write about with regards to creating and hosting a website.

Some day I may start measuring data from my nginx access logs and use that data to make a more informed decisions around scaling and traffic. But at the moment, I have other things I’m excited to learn about and the website is up so I’m not fussed.

Just one more thing, my old domain mountainbean.online expired this week and my domain registrar wanted me to pay $80 US to renew it which was never going to happen. Thus you may have noticed I’ve moved to samjdrew.com. If the savvy reader has any idea where I can get better domain registrars that don’t hold my domain to ransom every year, please write in and let me know. Feedback is also welcome.

[^1]: I had a whole analogy here where you would ask your mate to by groceries and cook dinner for you but it turns out your mate got kickbacks from the grocery store so he buys you a bunch of stuff you don’t need to run up your grocery bill. It’s a bit cynical and the analogy was not exactly relatable so decided not to force my up-and-coming novella “Stretchy- Legume Steve” upon you.

[²]: It’s worth noting somewhere that I don’t have to ever worry about renewing my certificates. My servers are rebooted around every 2 days and a new certificate is generated each time. (Edit: turns out this is a problem because if I ever go over 5 certificates in a week, certbot stops giving me new certs that week. I’ll have to work in a way to store, access and renew my certs.

[³]: but if it were, it’d be inspirational af.

In the act of procuring cloud infrastructure for my website, I've taken some shortcuts and I thought I'd carefully measured the price of those shortcuts and was willing to pay for them. Despite that, when my budget alerts went off a mere 10 days into the month, I was surprised and also kind of annoyed. How can this be?

So for this dev log, I've decided to have a look at my bills and see what we can do.

Oh boy. I don't know why I bothered spending so much time time tinkering my S3 media file hosting solution. That and most of the other costs related to my website server are dwarfed by this huge, honking Load balancer bill!

Load balancer clearly costs more than any other service. However VPC is unusually High too...

So yeah what's all that for just a micro T3 EC2 box quietly running a Django app that no one but me uses? I see 3 chunky bars evoking my ire.

• Load Balancer

• VPC

• EC2 - Compute

Let's address the Load balancer first.

Balance Deez

I don't see myself as a conspiracy person, but there's a reason AWS tells you "the simplest" way to enable https on your server is by attaching a load balancer. You can see it here in their documentation on the Load Balancer pricing page.

Straight out of the gate, we're spending upwards of $18 per month just to keep the load balancer on 24/7. That's not even considering the LCUs (Capacity units) used. $18/month is already too much and I'll need to take action to fix this (the action is me ripping it out like a weed, in case you were wondering).

Next up is the VPC. What's that charge for?

Well, dear reader, much like a decently priced rental in Brisbane City, public addresses on the internet are in short supply (that is, IPv4 addresses). This makes them valuable. Valuable enough that as of February 2024, Amazon has been charging 0.5 cents per hour for the pleasure of using one. Fast-forward to last month, when I asked Elastic Beanstalk to put my website online. It assumed I was a man of vastly more means than I really am and decided that I must want 3 availability zones each with a public IP address.

I was just a poor developer, trying to find a cheap way to host my website. Now look at me, I have a leak in my bank account of exactly 1.5 cents/hour and I get to think to myself, "boy am I glad my website is highly available".

1.5 cents per hour makes Sam a poor boy

Finally EC2 pricing. There's not much to say.

EC2 charges based on the type of EC2 and data transferred in a given time period. Data transfer is negligible at this stage and the EC2 type is T3-micro, the 2nd cheapest on-demand instance AWS offers. However, there are still some neat little FinOps tricks I can explore here.

Yes, FinOps is a real word

First off, the ALB has to go. There's no sense in my using a Load balancer anyway because my website traffic is featherweight. If that ever changes, I'll scale it up myself. For now paying $18 per month for a fancy server to route https traffic to my single node website is farcical. It's doing nothing but serving the SSL certificate which I can get Nginx to do on the instance for free.

While reconfiguring things, we could reduce our VPC costs by switching to IPv6. This would remove all the IPv4 tolls and our only charges will come from traffic (if we ever break out of free tier). Unfortunately, IPv6 presents a lot of it's own challenges. Namely that neither my home broadband service provider nor my mobile service provider support IPv6. The funny part is that even AWS has spotty IPv6 support! The AWS Instance Connect tool that AWS provides to connect to instances through your browser ALSO doesn't actually support IPv6. Why is any of that relevant? Because practically, yes I can configure and populate an IPv6-only VPC and subnets regardless of whether my ISP or Instance Connect supports IPv6. However, I won't be able to reach them for configuring or debugging my instances because I can't SSH in from home. It seems no one can actually handle the protocol that has been getting "adopted" since 2012.

Well okay. I can at least remove the 2 IPv4 addresses that I'm not using. That'll reduce the VPC costs by 1 cent/hour saving me up to $7.44/month (USD btw so that's actually around 14 dollaroos 🦘).

Finally we need to reduce EC2 costs. Savings plans are an option. According to AWS documentation, this would bring the hourly price of my instance down by 22%. However, I've been wanting to try out creating a spot fleet. It's essentially an autoscaling group but the instances are only ever spot instances. I can configure a spot fleet with a desired capacity of only 1 instance. Spot instances can get reclaimed by AWS at any time as other users are willing to pay the on-demand price. However if you have a spot fleet set up, the instance will be replaced automatically (if possible). Looks like this should bring my EC2 instance costs down around by 35%. If my website goes down for a few minutes or hours between spot instances, I'll live.

"...And a very familiar feeling is starting to come over me. I feel like someone is trying to teach me something." - Jeff Winger, Community Episode: Conspiracy Theories and Interior Design

To a seasoned Platform Engineer, I'm sure this is some kindergarten-level course-correction. It's not tricky to point out where I've taken shortcuts. I've done the cloud certification exams, I've heard of these issues before and I've been working in AWS for a few years. That's why I know that these are common problems and where to find the solutions. But I've never actually had to foot the bill for my shortcuts. That's what this little project is for (among other things). Unlike Jeff Winger, I'm here to learn and it's all part of the experience.

Keep an eye out for my next post where I'll go through implementing my cost-saving changes and how much my AWS bill actually changes.

Now look... no one's saying you must only ever use Terminal to get around, but the thought of clicking my mouse that many times just to push to production makes my little DevOps tummy hurt.

The Django tech stack ✨idea✨

Configuring prod data in dev

So this is just so I don't have to muck around setting up and spending money on a DB. Photo details are all stored in SQLite. That gets uploaded with my code to the repo. I then have Github Actions push that to Elastic Beanstalk, as my manager says, "auto-magically". No dev DB/prod DB disparity. It's just my website. For this moment, the data can be the same.
Now what about the photos and any other static files? I should move those to S3.

Firstly, I've set up a bucket with public read access, uploaded my photos there, set up Django to map the media URLs to the s3 bucket. Then I'll sit back and use all the spare time I now have googling ways I could've done it better.

MEDIA_URL = "/photo_files/"
if DJANGO_WEBSITE_ENVIRONMENT == "PROD" or DJANGO_WEBSITE_ENVIRONMENT == "BUILD":
    AWS_STORAGE_BUCKET_NAME = "mountainbean-photos-bucket-shepherds-pie"
    AWS_S3_REGION_NAME = "ap-southeast-2"
    AWS_ACCESS_KEY_ID = getenv("S3_WRITE_USER_ACCESS_KEY")
    AWS_SECRET_ACCESS_KEY = getenv("S3_WRITE_SECRET_ACCESS_KEY")

    AWS_S3_CUSTOM_DOMAIN = f"{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com"

    STORAGES = {
        "default": {
            "BACKEND": "storages.backends.s3.S3Storage",
            "OPTIONS": {
                "location": "media"
            },
        },
        "staticfiles": {
            "BACKEND": "storages.backends.s3.S3Storage",
            "OPTIONS": {
                "location": "static"
            }
        }
    }

Using the django_storages library, I've configured the public S3 bucket as a backend with a service role that has write-access permission on the bucket. An admin with access to post a new photo on the website (me) now has permissions to upload to the bucket from the website and that photo can be viewed by any visitor via a url to the photo location in the public bucket.

So what's with the if() statement?

When the website is being hosted by elastic beanstalk, I want Django to give out the correct urls to the photos in the S3 bucket. Likewise, when Github actions is deploying the code (there the environment variable will be BUILD) and running collectstatic, I want django to send the static files to S3, not just collect them into a local directory. However, when I'm just working on this on my machine, I don't want those files going anywhere.

So I set: MEDIA_ROOT = BASE_DIR / "uploads" in my Settings.py and:

if settings.DJANGO_WEBSITE_ENVIRONMENT != "PROD":
    urlpatterns += static(settings.MEDIA_URL,
                          document_root=settings.MEDIA_ROOT)

Map that directory to MEDIA_URL in my root urls.py so Django will send all my uploaded images there when I'm just in localhost.

The catch

I have one manual step left. When I make a commit or merge to master (triggering my Github workflow) I have to remember to copy my uploads folder to the public S3 bucket under the media key.
Lastly, I should address the issue of uploading my full-size images to the web. I've just shifted the problem from my repo to the S3 bucket. I still have very large JPEGs in my public S3 bucket that I (and any other visitor) will be requesting every time I visit my site.
AWS gives you a free 100GB of data transfer out of S3 per month. But if I'm not careful, my big fat JPEGs might use that up quickly. I'll look at resizing uploads next...

I've been working on a photography website as a project to force myself to actually learn Django. I enjoy coding and I love taking shitty photos but I've never deigned to host my own work anywhere other than Instagram. By setting a project for myself like this, not only do I hope I become less sucky at working with Django, a necessary skill in my particular role at my workplace, one ancillary outcome is that I get to host my photos somewhere more permanently.

The progenitor website starting page

It has three views.

• Starting page: Shows the three most recent photos

• All photos: Shows all uploaded photos

• Photo detail: Displays a single image with white margins

That's all it does right now. The photos are currently placeholders, They are my photos and they will appear in the final product but these particular photos are jpegs that have undergone at least 2x rounds of edits (jpeg's lossy compression gets applied for every edit + save). To be honest, they look fine to me but I'll want make sure the final product looks as good as I can make it. All static files (like the photos) are hosted locally for now. The plan is to put them in an S3 and get Django or Nginx to fetch them.

Photo detail view

Resourcing

Deciding on hosting raises some conflicting feelings. I want to take advantage of my somewhat working knowledge of AWS systems... but I also don't want to over-engineer this site that will likely only be visited by myself, my wife and a handful of job recruiters that actually vet me. I've never actually done this before either. So I'm not taking any shortcuts, nor am I jumping into architectures I have absolutely no business using.

Resourcing initial ✨idea✨

This is the basic plan. Chuck it on a tiny EC2. SQLite database to hold all the photo details including the S3 key (so it knows where to reach for it in the S3 bucket). The S3 bucket is just a regular bucket with public access blocked. I'll need to allow the server to access the photos on it via API key. Lets flesh this out a bit.

Resourcing ideal version 1

So I'm just using SQLite. This is a file-base database system. No need for a dedicated server which simplifies deployment drastically. The drawback here is that it's not quite as fast as a dedicated DB server like Postgres or MySQL would be. So I might have some delays in fetch times as the website grows. However, at this scale I think this will be fine. Similarly, hosting my photos on an S3 bucket means no need to spin up an additional private server. S3 is by far the cheapest option for file storage (probably) and the drawbacks are outweighed at this stage by my chief desires to spend no money and do no work.

First Deployment

My Django website's Elastic Beanstalk environment overview

Alright, I'm cheating. I'm using a barebone Elastic beanstalk application to spin up my EC2 in an "autoscaling group" of 1 instance, no DB, no VPC, no security groups. Just what is needed. In defence of my laziness, this is all I need at this stage. I wanted it live. It's now live (almost)

Deployment Woes

Right off the bat I ran into some problems getting a response from the EC2 instance deployed by elastic beanstalk. After re-deploying 4 times, re-checking my configuration each time, I was fed up. I've been working in AWS too long to get stuck at first deployment. Let's diagnose.

Elastic beanstalk is telling me it's deployed the EC2, added an EIP and added the EC2 to the environment. Good... So what's wrong? The problem is that AWS is getting no responses from the instance. Not just from the Nginx server, but from the whole instance. That tells me it's not connected to the internet. Lets check the instance settings.

Django website app EC2 instance Networking tab

Seeeeems fine... I didn't set any VPC settings so Elastic Beanstalk has taken care of all that for me. Let's look at the VPC and see if it has an Internet gateway.

Internet gateway generated by Elastic Beanstalk

Okay... is it routed?

Route table generated by Elastic Beanstalk

So that's weird right? There should be a route there for outbound packages (anything not going to 172.31.0.0/16). But there's not. Elastic Beanstalk generated everything I asked and then some, but stopped short of actually connecting the VPC to the internet. Let's add it in manually.

Route table with added route out to the internet gateway

I'll cut to the chase. That solved the issue. With my instance now connected to the internet, Elastic Beanstalk is getting responses and I'm able to connect on the default domain it's given me.

I'll leave it there for today

Literally best website you've ever seen

After adding the DNS entries for my domain and sorting out SSL, my site is officially on the web. I'm putting out my dirty html for the world to see. Instead of S3 hosting the images, the Nginx server on the EC2 is currently serving the sample images I loaded in with the code. I have many tasks left to do, but for today, this is enough.