My dumbass could not think of a better way to use a little free compute

I had some free compute from OCI's Always Free tier lying around and decided to put my personal site on it. To add a lil twist, I made 4 visual versions, and Envoy rotates between them every day.

I never thought I needed a personal site but it makes sense now

I never thought I would need a personal site until now because I never thought I would have enough content to put on it, at least at this early stage of my career. Then AI happened and made building things accessible, so everything in the tech space changed. You could easily solo-ship cool ideas: ideas that made sense, ideas that were just outright dumb, and ideas that had no business existing. Everything accelerated.

Fast-forward to today, I have so many projects that I think are cool, so I needed a space to share them. Obviously, I could just post them on social media and forget about them, but they would fade away with time. So the next obvious option was a personal website. I could fully manage it and control what content gets shown there and how.

So where should I host my personal site?

Now that we have established that I needed a personal site, the next obvious thing was buying a domain. I looked at a lot of options and where other people hosted their sites. The obvious one was just a [name].com, and for the first time in my life, the exact domain I wanted was available. So I bought aakashreddy.com.

I was looking for inspiration for the site. I wanted it to be unique not only visually but in how it was built as well.

On a regular day, while doomscrolling on X, I found something very interesting: https://github.com/nce/oci-free-cloud-k8s. This has detailed instructions for setting up a free Kubernetes cluster using OCI's Always Free tier. I was immediately interested, as this is the setup I usually work with for my job. Using it for a personal site would be cool because it reflects the tech I use for work and would definitely be unique: a static personal site served from a Kubernetes cluster. So I picked this idea and proceeded with it.

MVP1

The setup for using a static website was pretty straightforward from here. Set up the domain. Point it to the static IP for the Kubernetes LB. Direct the traffic to the pod serving the site. It is pretty simple. You can find the setup here: https://github.com/skyaara/oci-k8s-static-portfolio.

For MVP1, the static Astro SSG site was served directly from the nginx pod. It was functional. It was unique. The issue was that only I knew it was unique. A person viewing the site in their browser would not see the difference between it being hosted on Cloudflare Pages, a Kubernetes cluster, or the Moon. So I was still not satisfied with how uniquely it was built.

Inspiration for the current idea

If you are on Tech Twitter, you might remember a guy who got laid off from Atlassian and spoke about the company's internal infrastructure. He dropped an informative YouTube video about his tenure there: https://www.youtube.com/watch?v=55pTFVoclvE&t=1510s. A curious cat found its way. In the video, he spoke a lot about how good and configurable Envoy, an L7 proxy, actually was. I had only worked with nginx-based HTTP proxies until then, and Envoy piqued my interest. I went through the documentation, and I loved it. Anything we did with it could fit directly into serving our site from our Kubernetes cluster.

Current setup

The immediate basic setup I could come up with for Envoy was to use it as a router for different visual versions of the site. Initially, I thought I could serve a completely different site through the cluster each day, but then the sites might diverge and feel unrelated. So I converged on one shared site that cycles through canvas shaders while keeping the core content the same. This is the idea the current site is built on. The same setup can be done in multiple simpler ways for free, but the way I built it is stupid and dumb, so let's stick with it.

There are currently 4 different visual shaders for the site. Each of them is compiled into appN/current.js, where N is 1, 2, 3, or 4. The client requests a shader from the site using a static URL, and the request is routed through the Envoy filter, which cycles through the shaders each day. This configuration can be extended to any number of apps, and supporting new shaders requires just a small change to the Lua script.

Below is a simple Envoy Lua filter script. It leaves other hostnames alone. For shaders.aakashreddy.com, it allows only GET requests for /current.js, calculates the current Unix day, and cycles through the apps. It adds appN to the internal path so it can pick up the correct JS file for that day's version. This lets us serve different visual versions from the same public URL.

lua
function envoy_on_request(handle)
  local headers = handle:headers()
  local authority = string.gsub(headers:get(":authority") or "", ":%d+$", "")
  if authority ~= "shaders.aakashreddy.com" then return end

  if headers:get(":method") ~= "GET" then
    handle:respond({[":status"] = "405", ["allow"] = "GET"}, "Method Not Allowed\n")
    return
  end

  local path = string.match(headers:get(":path") or "/", "^[^?]*")
  if path ~= "/current.js" then
    handle:respond({[":status"] = "404"}, "Not Found\n")
    return
  end

  -- A Unix day counter continues cleanly across month and year boundaries.
  local utc_day = math.floor(os.time() / 86400)
  local app = (utc_day % 4) + 1

  -- The browser keeps requesting /current.js; this is an internal rewrite.
  headers:replace(":path", "/app" .. app .. "/current.js")
end

Kubernetes resource definitions

This could get very long for no reason, so let's intentionally keep it short.

Our setup currently consists of 2 namespaces, istio-system and site, and a native OCI load balancer with a static IP that we supply to Cloudflare DNS.

istio-system has istiod installed. It manages the Istio resources and Envoy ingress. Whenever a new deployment happens, istiod reads the updated config. If any config has changed, it sends the updated config via xDS to the relevant resources. This is a pretty standard custom Envoy setup that uses Istio.

site namespace has the actual shader files in a lightweight nginx pod that acts as a file system in our setup.

Client-to-origin request flow

The important final block: Security

Considering this whole site is unauthenticated, everyone should always be able to read it without restrictions. There are 2 important parts in the request flow: the part where the request reaches the load balancer and the part from the load balancer to the nginx pod.

The second part of the request flow is relatively easier. Istio and Envoy let us inspect L7 traffic here and allow only the requests we need. So we will get this out of the way first.

After the request reaches the load balancer, it flows to the IstioGateway, where TLS termination happens. It accepts requests only when the host is shaders.aakashreddy.com. A validated request gets passed on to the EnvoyFilter that only allows GET requests and returns 405 Method Not Allowed for others. It accepts only the /current.js path and returns 404 Not Found for any other path. It removes query parameters before matching and rewrites /current.js to /appN/current.js. Only verified requests that match the routing rules in the VirtualService are sent through the Kubernetes Service to the pod. The DestinationRule applies traffic policy to that destination. The site namespace enforces the Restricted Pod Security Standard. We run the nginx pod as a non-root Linux process with a read-only root filesystem. We allow only the Linux system calls required by nginx at runtime, restrict all other system calls, block privilege escalation, and do not supply a Kubernetes serviceAccountToken. Without a mounted token, the pod has no in-cluster identity for calling the Kubernetes API. We also duplicate the HTTP method and path validation as a final check. This is overkill, as we only intend to use nginx as a file server.

The above part sounds complicated, but it is mostly handled through widely accepted Kubernetes practices.

Now comes the part of the flow where we need to prevent most traffic from reaching the load balancer and serve it through Cloudflare instead. We have DNS records set for shaders.aakashreddy.com, proxy traffic through the Cloudflare edge to the load balancer's static IP, allowlist only Cloudflare's published public IPs, and set up aakashreddy.com on Cloudflare Pages. We also allow caching for shaders.aakashreddy.com, and we intend to serve most of our traffic through the CDN. I added protective WAF rules on Cloudflare that block any request other than GET shaders.aakashreddy.com/current.js, ignore all query parameters so that cache hits happen regardless of them, and add one-hour Cache-Control and rate limiting. None of those controls guarantees security on its own, but together they cut down what can reach the origin. Anything that does reach it still has to pass the network, Istio, Envoy, and pod-level controls.

Now that the inbound path is locked down to the traffic we expect, we should be done, right? Nope. Another question remains: how do we deploy the application if we do not allow any public requests to the OKE Kubernetes API? We use a service called OCI Bastion, an SSH-like managed service from OCI that creates a temporary path into the target subnet and forwards traffic to the private OKE Kubernetes API. When we want to deploy, we open a Bastion session through the authenticated OCI CLI, upload our public SSH key, and connect using the matching private key from a machine within an allowed client CIDR. We use Helm through that Bastion tunnel to apply the updated Kubernetes configuration.

Thank you for reading.