[PODCAST INTEL] Latent Space
"The 100,000 Sandbox Problem — Akshat Bubna, Modal CTO"
Guest: Akshat Bubna
Signal: 0.78 (HIGH)
Thesis: Agent workloads require fundamentally different cloud primitives than traditional compute—the limiting factor is no longer GPU throughput but CPU-GPU colocation, memory movement efficiency, and adaptive scaling across heterogeneous hardware. LLM-mediated permissions and soft guardrails alongside hard sandbox boundaries will become the kernel-level abstraction for agentic systems.
Key takeaways:
1. Speculative decoding via improved draft models (2-4x speedup via accept length) is multiplicative; kernel optimizations only yield single-digit percentage gains. This shifts inference optimization focus from kernels to algorithm design.
2. RL rollouts generate 100,000+ simultaneous sandbox spikes; inference autoscaling across regions is the true differentiator, not raw GPU capacity. Elastic scaling from 0 to 1500 GPUs per region per hour is the new baseline.
3. Agents are poor at using observability/logs to correct themselves without skills; LLMs can now one-shot Modal code, but struggle with reasoning about infrastructure state. This drives need for agentic benchmarks and tuned skills.
"The 100,000 Sandbox Problem — Akshat Bubna, Modal CTO"
Guest: Akshat Bubna
Signal: 0.78 (HIGH)
Thesis: Agent workloads require fundamentally different cloud primitives than traditional compute—the limiting factor is no longer GPU throughput but CPU-GPU colocation, memory movement efficiency, and adaptive scaling across heterogeneous hardware. LLM-mediated permissions and soft guardrails alongside hard sandbox boundaries will become the kernel-level abstraction for agentic systems.
Key takeaways:
1. Speculative decoding via improved draft models (2-4x speedup via accept length) is multiplicative; kernel optimizations only yield single-digit percentage gains. This shifts inference optimization focus from kernels to algorithm design.
2. RL rollouts generate 100,000+ simultaneous sandbox spikes; inference autoscaling across regions is the true differentiator, not raw GPU capacity. Elastic scaling from 0 to 1500 GPUs per region per hour is the new baseline.
3. Agents are poor at using observability/logs to correct themselves without skills; LLMs can now one-shot Modal code, but struggle with reasoning about infrastructure state. This drives need for agentic benchmarks and tuned skills.