Isolation at Fleet Scale: How Astronomer Made Every Cluster Steadier and Cheaper to Run

HQ: 
NYC
Industry: 
DataOps, Cloud Infrastructure
Table of contents

Isolation at Fleet Scale: How Astronomer Made Every Cluster Steadier and Cheaper to Run

Astronomer runs their product Astro, an enterprise data orchestration platform powered by Apache Airflow, across a large fleet of Kubernetes clusters, many of them dedicated to a single customer. Enterprises pay for that isolation and in return don’t have to worry about a shared control plane, audit complexity, or noisy neighbors.

But providing that level of isolation has other costs since each cluster runs its own Airflow components, all on a dedicated platform node pool, before the customer has even scheduled any work. Each of those components reserves CPU and memory through its pod requests, whether it’s using them or not. A single shared platform would split that overhead across everyone, but running clusters separately means paying it again in each one.

The Challenge: Sizing a Fleet by Hand

This leaves the platform team with a tough question: how large should those pod requests be?

Sizing one cluster is an afternoon's work, but sizing an entire fleet is a different problem since every cluster has a different business behind it and the right answer keeps moving as Airflow versions ship and customers grow. There are simply more clusters needing attention than any company would sensibly staff for.

Human attention tends to go to outliers, the clusters that got big or noisy enough to outgrow the default. Everything else runs on that default, which is intentionally generous. A default that’s too high may buy a compute guarantee that holds for every customer but is ultimately paid for in idle capacity. It’s the infrastructure equivalent of building every meeting room to seat the company all-hands. Set it closer to the middle and eventually a pod runs out of memory in a customer’s cluster.

That padding has a second cost, and it’s one customers could feel more directly than the bill. The scheduler places pods by adding up their requests, not by what they actually use, so a node that could comfortably run several more pods shows as full and the next pod triggers a new node instead. When demand settles those extra nodes get drained and whatever was running on them has to move somewhere else. Across a whole fleet it means the infrastructure underneath customer workloads is constantly moving around.

The traditional answer here is the Kubernetes vertical pod autoscaler, which sets pod requests using a percentile of recent usage. Astronomer evaluated it, but a percentile of recent usage isn’t designed to account for the daily and weekly seasonality that define their customers’ Airflow workloads, where the usage profile at 3am on a Sunday is different from the one at 9am on a Tuesday. It also needed a long warm-up period, reacts to spikes after they happen rather than ahead of them, and wasn’t compatible with their end-to-end tests, so they decided it wasn’t the right fit.

The Solution: Predictive Sizing That Happens Per Cluster

Thoras observes and analyzes workload behavior, forecasts demand, and sets workload requests before demand arrives rather than reacting to a percentile of what already happened.

Across a fleet, it sizes each cluster against its own demand and seasonality instead of the busiest customer's peak. Pod requests in a cluster orchestrating thousands of tasks get sized for that load, and a cluster running a dozen Dags gets sized for that. Every cluster needing individual attention was always the right answer but now it doesn’t require a human engineer’s attention to achieve that.

What Changed for Astro Customers

Once requests reflect what workloads actually use, more pods fit onto the nodes that already exist. The same daily swing in Airflow work then moves a fraction of the nodes it used to, since all that padding was acting as a multiplier on how much the cluster had to grow and shrink to absorb it.

There are two things happening in that graph. The obvious one is the drop in total nodes. But the one that matters more is the one that customers feel - how much flatter the line gets afterward, since every bump is a node being added or taken away and something running on that node having to deal with it.

That plays out in a few places customers notice directly:

Core components stay where they are.

The Airflow scheduler and the Celery message broker run on that same platform pool, and every node that drains is another chance for one of them to get evicted and restarted. Fewer node removals means fewer restarts, and fewer restarts means fewer chances for scheduling gaps and catatonic workers.

Workers get interrupted less often.

A task running on a node that’s about to go away either gets evicted and retried, or it holds up the drain until it finishes. Less churn means fewer tasks end up in that position.

Tasks spend less time queued.

A worker pod that can be placed on a node that already exists starts in seconds, while one that has to wait on a new node being provisioned waits for potentially minutes.

Additional Results: The Fleet Got Smaller

In addition to the positive impact on node churn, across the platform node pool, fleet-wide, Astronomer's analysis puts the savings at over 10% of cloud spend annually, and growing as adoption of the platform increases. A forecast built from a few weeks of a cluster's history leaves more headroom than one built from months of it, and at the time of measurement most of the fleet was still early on that curve.

‍

‍

Sizing every cluster against its own demand also took a project off Astronomer's list. They had talked about doing the analysis to sort clusters into behavioral profiles and then making sizing decisions per bucket, which would have meant real work both in the analysis and in the tooling to use it. There’s no bucket to assign a cluster to when every cluster is already sized for itself.

‍

Results: Sizing That Keeps Up

Traditionally, a workload’s resource request is correct on the day it’s set and begins drifting immediately. Things like Airflow shipping a new version, Dag counts increasing, or a team changing how it batches work are all examples of organic change in the system that transform usage baselines and ultimately what causes resource requests to become stale. This is true of anything sized by hand, and many (we’d argue even “most”) workloads in Kubernetes are.

In Astronomer's observability and rollout data, workload requests kept declining well after each cohort was enabled, as Thoras continued rolling out and its forecasts continued getting more history to work from.

A True Partnership

Most companies running a fleet of clusters for their customers pay a similar overhead, and most arrived at the same default for the same sound reasons: a number set to be safe for the largest customer, applied to everyone, because sizing each cluster for itself was a major engineering challenge.

Every cluster in Astronomer’s fleet now gets a forecast of its own.

Trial Thoras for free, and start scaling immediately.

Read More Case Studies

Best Practices
Human-Centered Pull Requests: What Most PR Advice Is Missing
Read more
News
Fear, Uncertainty, and ...Utilization?
Read more
News
Scale Out or Size Right? A Guide to Horizontal Scaling and Pod Right-Sizing
Read more