Baseten’s Model Inference Platform Is Pulling Clients from Modal

The Inference War Nobody Saw Coming

Model inference – the act of running a trained AI model in production so it actually does something useful – has quietly become one of the most contested battlegrounds in the AI infrastructure stack. Baseten, a San Francisco-based startup that has spent the better part of three years building tooling for production model deployment, is now drawing serious attention from teams that originally built their workflows on Modal. The shift is not a dramatic exodus, but the pattern is consistent enough that Baseten’s growth curve has become hard to ignore.

Baseten’s pitch is specific: it handles the full lifecycle of model serving, from cold start optimization to GPU autoscaling, without requiring engineering teams to architect that infrastructure themselves. The platform targets ML engineers who are tired of babysitting Kubernetes clusters but still need fine-grained control over how their models behave under load. That positioning sits in an interesting middle ground between fully managed black-box APIs and the raw compute marketplaces that give you nothing but a GPU and a bill.

Modal built a loyal following by making serverless compute genuinely pleasant to use.

Where Baseten Is Finding Its Edge

The core technical argument Baseten makes is around latency predictability. Production ML teams often care less about peak throughput and more about tail latency – the worst-case response time that determines whether a user-facing feature feels broken. Baseten’s architecture prioritizes keeping those tail latencies consistent, which matters enormously for applications like real-time document processing, code generation tools, and voice interfaces where a slow response is a broken experience. Modal, by contrast, was designed around a developer experience first, with compute efficiency as the primary optimization target rather than inference-specific latency guarantees.

Baseten also offers what it calls Truss, an open-source model packaging standard the company developed to make deploying any model – whether from Hugging Face, a fine-tuned internal checkpoint, or a third-party provider – follow a consistent pattern. The open-source angle matters because it lowers the switching cost for teams evaluating the platform. A team can package their model using Truss, test it against Baseten, and still retain the ability to deploy elsewhere without a full rewrite. That kind of portability removes one of the biggest objections enterprise buyers raise when committing to a new infrastructure vendor.

GPU availability has also become a quiet differentiator. During periods when H100 access tightened across the industry, Baseten reportedly maintained more consistent availability for its contracted customers than some competitors by negotiating capacity directly with cloud providers rather than relying entirely on spot markets. For production teams running revenue-generating features, the ability to guarantee capacity matters more than saving a few cents per GPU-hour on spot pricing.

The Modal Comparison Is Not Clean-Cut

Framing this as a clean win for Baseten against Modal misses some important nuance. Modal remains genuinely excellent for a specific type of workload: batch processing jobs, scheduled inference pipelines, and developer experimentation where cold start times are acceptable and the simplicity of Python-native deployment is worth more than latency guarantees. Teams building internal tooling, running nightly fine-tuning jobs, or prototyping new model integrations are not the customers Baseten is pulling. The movement is happening among teams that started on Modal because it was the fastest path to production, then found themselves needing features that Modal was never designed to provide at scale.

What Baseten offers that increasingly resonates with growing teams is a set of enterprise-facing controls: private deployments in customer-owned cloud accounts, audit logging, role-based access controls, and dedicated support contracts. Those features do not matter at all when you are a five-person startup deploying a side project. They matter enormously when a Fortune 500 company asks your legal team whether your model infrastructure is SOC 2 compliant and whether your vendor can sign a business associate agreement. Baseten has spent time building the compliance and security apparatus that enterprise procurement requires, and that investment is now converting deals that a simpler developer-tools company could not close.

The pricing structure also favors teams with predictable, sustained workloads. Modal’s serverless model prices well for variable or bursty usage, but teams running models continuously against steady traffic often find that Baseten’s reserved capacity pricing ends up cheaper when all costs are totaled. Enterprise buyers running cost-benefit analyses across a full quarter of inference spend are increasingly landing on Baseten as the more economical option for their specific usage patterns.

What This Means for the Broader Inference Market

Baseten’s traction reflects a maturation happening across the ML infrastructure space – teams that once prioritized ease of deployment are now asking harder questions about control, cost, and compliance, and the vendors that built for those requirements from the beginning are starting to collect the contracts that prove it. The real test for Baseten will come as hyperscalers including AWS, Google, and Azure continue pushing their own managed inference products deeper into the enterprise stack, because at that point, the question of whether an independent inference platform can hold pricing power against the companies that already own the compute layer becomes very sharp very fast.