Energy-Aware Inference Scheduling for Heterogeneous GPU Clusters

1. Introduction

The carbon and energy cost of LLM inference is now comparable to training [Patterson et al. 2022]. Production inference fleets are typically heterogeneous: an A100 generation purchased in 2021, an H100 generation purchased in 2023, and L4s for low-throughput tasks. These devices differ by 2-4$\times$ in energy-per-token at fixed model size, but nearly all schedulers ignore this and route purely by load.

We present JouleSched, an energy-aware scheduler that incorporates per-device energy profiles into the routing decision while honoring latency SLOs.

2. Problem Formulation

Consider a cluster with $N$ devices. Device $n$ processes requests at throughput $\mu_n$ (tokens/s) with energy cost $\epsilon_n$ (joules/token) and queue length $q_n$. For an incoming request $r$ with token estimate $\hat{t}_r$ and SLO deadline $D_r$, define

$$\mathrm{ETA}(n, r) = \frac{q_n + \hat{t}_r}{\mu_n}$$

We want to choose device $n^*$ minimizing

$$n^* = \arg\min_n \Big[ \epsilon_n \hat{t}_r + \lambda \cdot \max(0, \mathrm{ETA}(n,r) - D_r) \Big]$$

where $\lambda$ is a Lagrangian multiplier on SLO violation. The first term is the energy cost; the second penalizes deadline misses.

3. Method

3.1 Profiling

We profile each $(\text{device}, \text{model})$ pair offline, measuring energy via NVIDIA NVML's nvmlDeviceGetTotalEnergyConsumption and computing $\epsilon_n$ as a function of batch size and sequence length.

3.2 Online routing

The scheduler maintains a min-heap keyed on the JouleSched cost. SLO buffer $\lambda$ is auto-tuned via PI control to keep deadline-miss rate at a target $\rho^* = 1$:

$$\lambda_{t+1} = \lambda_t + K_p (\rho_t - \rho^$$) + K_i \sum_{\tau \le t}(\rho_\tau - \rho^)

def route(req, devices):
    best, best_cost = None, float('inf')
    for d in devices:
        eta = (d.queue + req.tokens) / d.throughput
        slack = max(0, eta - req.deadline)
        cost = d.epsilon * req.tokens + LAMBDA * slack
        if cost < best_cost:
            best, best_cost = d, cost
    return best

3.3 Workload-aware tier routing

We additionally classify incoming requests into interactive and bulk tiers; bulk requests (e.g., embedding jobs, batch summarization) tolerate higher latency and are aggressively routed to the most energy-efficient devices.

4. Experimental Setup

We simulate a 320-GPU cluster (160 A100s, 128 H100s, 32 L4s) using a discrete-event simulator calibrated against a 24-hour production trace from a major LLM API provider. Workload mix: 64% chat (interactive), 28% RAG (interactive), 8% batch.

5. Results

JouleSched delivers a 24.8% energy reduction at iso-SLO. The H100 utilization rises from 72% to 89% (since H100s are more efficient per token); L4s carry a larger share of bulk traffic (49% of bulk vs. 12% under round-robin).

6. Carbon Implications

At the marginal grid intensity of 0.42 kgCO$_2$/kWh, the cluster's annualized savings are approximately 1{,}710 metric tons of CO$_2$. While the absolute number depends on regional grid mix and cluster utilization, the fractional improvement is robust across grids.

7. Limitations

Profiles drift with driver updates and silicon aging; we re-profile monthly. The scheduler assumes accurate token-length estimates; we use a 95th-percentile prefix-conditioned predictor with 12% MAE, which suffices but degrades gracefully at higher error. Finally, JouleSched does not model power capping; integrating with frequency-scaling controllers is open work.

8. Conclusion

Energy-aware inference scheduling is a low-hanging-fruit optimization with material sustainability impact. JouleSched cuts energy by ~25% on a realistic heterogeneous cluster without compromising latency.

References

1. Patterson, D. et al. (2022). The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink.
2. Strubell, E. et al. (2019). Energy and Policy Considerations for Deep Learning in NLP.
3. Tang, X. et al. (2024). Power-aware Scheduling for LLM Inference.
4. NVIDIA. NVML API Reference.
5. Schwartz, R. et al. (2020). Green AI.