How Expel Used Predictive Autoscaling to Reclaim $117K From One of Its Busiest Workloads

Right now, somewhere in Expel's infrastructure, one pod is politely asking a security vendor's API whether anything interesting has happened lately. A few pods over, another is hauling a forensic artifact off an endpoint device on the other side of the world. They're the same Kubernetes deployment, which means something (or someone) has to pick one memory request that works for both of them, and for every other task that deployment runs.

Expel runs 24x7 managed detection and response. Their analysts work out of a SaaS platform that ingests tens of billions of alerts and events every month on behalf of customers, and the machinery hauling all of it in is what Expel calls its tasking infrastructure: a scheduling and asynchronous job processing stack handling both periodic and ad-hoc work.

The worker doing that job, task-executor, accounts for more than 90 percent of tasking pods and runs many thousands of concurrent replicas. Nearly everything the platform detects flows through it. If it falls behind, analysts wait and detection is delayed, which is a serious problem.

That mix of work is also what makes the workload really tough to size. Expel's own engineers put it more rigorously in a post about migrating the system to GKE: their tasks "have widely different memory requirements with outliers requiring several standard deviations more than the mean."

There are two ways to pick that number, and neither one feels great.

Option A: Provision for the worst case. Every pod gets sized for the heaviest task it might ever run, then spends most of its life polling an API. This is the infrastructure equivalent of buying a moving truck year-round because you move once in July.

Option B: Provision closer to the average and risk the heavy tasks getting killed. Expel built to handle this scenario: robust checkpointing and retry logic means a sporadic OOMKill doesn't noticeably affect performance, but it’s still not ideal.

Expel picked A, which was the correct call. They had also already squeezed A about as hard as it can be squeezed: the Kubernetes vertical pod autoscaler ran on task-executor, tuned specifically for how unevenly this workload uses memory, and their broader migration work had cut tasking compute costs by nearly 70 percent.

But a tool that sizes from recent usage has to settle on one number, and that number has to cover both the heaviest task as well as the quietest weekend hour. Expel's tasking load rises and falls with the workday and thins out on weekends, so the quiet stretches are long ones. Sizing off a past percentile means using the busy-hour number straight through them, overnight and all weekend, because the bigger number is usually the safer choice.

The Solution: Capacity That Shows Up On Time

Thoras predictively right-sizes workloads rather than simply reacting to a historical percentile. It learns how a workload actually behaves, forecasts what it will need in the future, and allocates resources ahead of the demand instead of scrambling along behind it.

For task-executor, that means Expel no longer has to choose between paying for the heaviest task all day and paying to run it twice after an OOM. The capacity shows up when the heavy task does, and goes away when it finishes. Expel runs the workload near its real current requirement most of the time, with headroom already in place before the heavy tasks land.

Two things made this comfortable in their cluster. First, Kubernetes 1.33 in-place pod resizing, so Thoras adjusts CPU and memory requests on running pods without restarting them. Applying a new request used to mean evicting the pod and starting it over. On a stateless deployment of many thousands of replicas that already absorbs regular churn from preemptible nodes, that eviction churn went away entirely.

Second, Thoras ran on task-executor in vertical autonomous mode, making changes directly rather than filing a PR for a human to rubber-stamp. Besides, at this scale there is no manual alternative since actual resource requirements shift several times a day- so any number a person picks would already be aging out before it was rolled out.

Getting there took about two weeks. Thoras ran in observation mode first, making no changes while the team checked whether the predictions looked credible. Some classes of task-executor flipped to autonomous within a day, others took closer to ten. The workload had also run in staging for months beforehand, though staging was never going to settle it since usage doesn't look much like production for this kind of work.

What actually built confidence was agreeing in advance on what would count as a problem. The team picked the performance metrics that mattered, walked through SLOs with stakeholders, and settled on what would trigger a rollback before anything was switched on. Task execution throughput and latency were the primary signals, with node density, scheduling errors, and pod churn watched alongside them.

Results: $117,000 a Year, Found in a Finished Project

$117,000 per year, annualized, on task-executor alone, confirmed end-to-end in Expel's GCP billing console- not a napkin estimate or a number from the dashboard. The actual bill!

The baseline is fascinating to consider because that figure is measured against Expel's steady state after right-sizing with traditional vertical autoscaling. There was no low-hanging fruit here! This was a strong platform organization that had already done the work, measured it, and quite reasonably filed the workload under "handled." The $117K is purely the gap between sizing for what a workload used and sizing for what it’s about to need.

The OOMKill rate stayed flat across the switch, holding the same range as before, which for this system is unremarkable since killed tasks are checkpointed and retried. Task ack rate and the other throughput measures were flat on both sides of the transition, and none of the signals Expel had named as rollback triggers ever moved. This is what gave the Expel team the confidence that the platform was doing its job safely.

A True Partnership

Most platform teams have a task-executor somewhere: the workload at the center of everything, too spiky to run lean and far too important to experiment on. The standard move is to provision for the worst case forever and call it the cost of sleeping through the night. Which is fine, except the worst case only shows up for a few minutes and bills you for the year.

Thoras has been a great partner along this journey. They provided great support to get us onboarded and continual support as we expand our adoption and move more into autonomous mode to reduce the burden of continually and safely right sizing our clusters for efficiency gains. They’re always eager to get feedback on their offering and act on it quickly. Truly a great partnership!
_{- Blake Scrivner, Director of Engineering @ Expel} 

Expel has never treated efficiency as a project with an end date, which is why anyone went looking at a workload already marked done. On their most critical one, the next round was worth $117,000 a year, and nobody had to get paged to find it.

‍