LinkedIn is a leader in AI recommender systems, having developed them over the last 15-plus years. But getting to a next-gen recommendation stack for the job-seekers of tomorrow required a whole new technique. The company had to look beyond off-the-shelf models to achieve next-level accuracy, latency, and efficiency.

“There was just no way we were gonna be able to do that through prompting,” Erran Berger, VP of product engineering at LinkedIn, says in a new Beyond the Pilot podcast. “We didn’t even try that for next-gen recommender systems because we realized it was a non-starter.”

Instead, his team set to develop a highly detailed product policy document to fine-tune an initially massive 7-billion-parameter model; that was then further distilled into additional teacher and student models optimized to hundreds of millions of parameters. 

The technique has created a repeatable cookbook now reused across LinkedIn’s AI products. 

“Adopting this eval process end to end will drive substantial quality improvement of the likes we probably haven’t seen in years here at LinkedIn,” Berger says. 

Why multi-teacher distillation was a ‘breakthrough’ for LinkedIn 

Berger and his team set out to build an LLM that could interpret individual job queries, candidate profiles and job descriptions in real time, and in a way that mirrored LinkedIn’s product policy as accurately as possible. 

Working with the company’s product management team, engineers eventually built out a 20-to-30-page document scoring job description and profile pairs “across many dimensions.” 

“We did many, many iterations on this,” Berger says. That product policy document was then paired with a “golden dataset” comprising thousands of pairs of queries and profiles; the team fed this into ChatGPT during data generation and experimentation, prompting the model over time to learn scoring pairs and eventually generate a much larger synthetic data set to train a 7-billion-parameter teacher model.

However, Berger says, it’s not enough to have an LLM running in production just on product policy. “At the end of the day, it’s a recommender system, and we need to do some amount of click prediction and personalization.” 

So, his team used that initial product policy-focused teacher model to develop a second teacher model oriented toward click prediction. Using the two, they further distilled a 1.7 billion parameter model for training purposes. That eventual student model was run through “many, many training runs,” and was optimized “at every point” to minimize quality loss, Berger says. 

This multi-teacher distillation technique allowed the team to “achieve a lot of affinity” to the original product policy and “land” click prediction, he says. They were also able to “modularize and componentize” the training process for the student.

Consider it in the context of a chat agent with two different teacher models: One is training the agent on accuracy in responses, the other on tone and how it should communicate. Those two things are very different, yet critical, objectives, Berger notes. 

“By now mixing them, you get better outcomes, but also iterate on them independently,” he says. “That was a breakthrough for us.” 

Changing how teams work together

Berger says he can’t understate the importance of anchoring on a product policy and an iterative eval process. 

Getting a “really, really good product policy” requires translating product manager domain expertise into a unified document. Historically, Berger notes, the product management team was laser focused on strategy and user experience, leaving modeling iteration approaches to ML engineers. Now, though, the two teams work together to “dial in” and create an aligned teacher model. 

“How product managers work with machine learning engineers now is very different from anything we’ve done previously,” he says. “It’s now a blueprint for basically any AI products we do at LinkedIn.”

Watch the full podcast to hear more about: 

• How LinkedIn optimized every step of the R&D process to support velocity, leading to real results with days or hours rather than weeks; 

• Why teams should develop pipelines for plugability and experimentation and try out different models to support flexibility; 

• The continued importance of traditional engineering debugging.

You can also listen and subscribe to Beyond the Pilot on Spotify, Apple or wherever you get your podcasts.

When OpenAI went down in December, one of TrueFoundry’s customers faced a crisis that had nothing to do with chatbots or content generation. The company uses large language models to help refill prescriptions. Every second of downtime meant thousands of dollars in lost revenue — and patients who could not access their medications on time.

TrueFoundry, an enterprise AI infrastructure company, announced Wednesday a new product called TrueFailover designed to prevent exactly that scenario. The system automatically detects when AI providers experience outages, slowdowns, or quality degradation, then seamlessly reroutes traffic to backup models and regions before users notice anything went wrong.

“The challenge is that in the AI world, failover is no longer that simple,” said Nikunj Bajaj, co-founder and chief executive of TrueFoundry, in an exclusive interview with VentureBeat. “When you move from one model to another, you also have to consider things like output quality, latency, and whether the prompt even works the same way. In many cases, the prompt needs to be adjusted in real-time to prevent results from degrading. That is not something most teams are set up to manage manually.”

The announcement arrives at a pivotal moment for enterprise AI adoption. Companies have moved far beyond experimentation. AI now powers prescription refills at pharmacies, generates sales proposals, assists software developers, and handles customer support inquiries. When these systems fail, the consequences ripple through entire organizations.

Why enterprise AI systems remain dangerously dependent on single providers

Large language models from OpenAI, Anthropic, Google, and other providers have become essential infrastructure for thousands of businesses. But unlike traditional cloud services from Amazon Web Services or Microsoft Azure — which offer robust uptime guarantees backed by decades of operational experience — AI providers operate complex, resource-intensive systems that remain prone to unexpected failures.

“Major LLM providers experience outages, slowdowns, or latency spikes every few weeks or months, and we regularly see the downstream impact on businesses that rely on a single provider,” Bajaj told VentureBeat.

The December OpenAI outage that affected TrueFoundry’s pharmacy customer illustrates the stakes. “At their scale, even seconds of downtime can translate into thousands of dollars in lost revenue,” Bajaj explained. “Beyond the economic impact, there is also a human consequence when patients cannot access prescriptions on time. Because this customer had our failover solution in place, they were able to reroute requests to another model provider within minutes of detecting the outage. Without that setup, recovery would likely have taken hours.”

The problem extends beyond complete outages. Partial failures — where a model slows down or produces lower-quality responses without going fully offline — can quietly destroy user experience and violate service-level agreements. These “slow but technically up” scenarios often prove more damaging than dramatic crashes because they evade traditional monitoring systems while steadily eroding performance.

Inside the technology that keeps AI applications online when providers fail

TrueFailover operates as a resilience layer on top of TrueFoundry’s AI Gateway, which already processes more than 10 billion requests per month for Fortune 1000 companies. The system weaves together several interconnected capabilities into a unified safety net for enterprise AI.

At its core, the product enables multi-model failover by allowing enterprises to define primary and backup models across providers. If OpenAI becomes unavailable, traffic automatically shifts to Anthropic, Google’s Gemini, Mistral, or self-hosted alternatives. The routing happens transparently, without requiring application teams to rewrite code or manually intervene.

The system extends this protection across geographic boundaries through multi-region and multi-cloud resilience. By distributing AI endpoints across zones and cloud providers, health-based routing can detect problems in specific regions and divert traffic to healthy alternatives. What would otherwise become a global incident transforms into an invisible infrastructure adjustment that users never perceive.

Perhaps most critically, TrueFailover employs degradation-aware routing that continuously monitors latency, error rates, and quality signals. “We look at a combination of signals that together indicate when a model’s performance is starting to degrade,” Bajaj explained. “Large language models are shared resources. Providers run the same model instance across many customers, so when demand spikes for one user or workload, it can affect everyone else using that model.”

The system watches for rising response times, increasing error rates, and patterns suggesting instability. “Individually, none of these signals tell the full story,” Bajaj said. “But taken together, they allow us to detect early signs that a model is slowing down or becoming unreliable. Those signals feed into an AI-driven system that can decide when and how to reroute traffic before users experience a noticeable drop in quality.”

Strategic caching rounds out the protection by shielding providers from sudden traffic spikes and preventing rate-limit cascades during high-demand periods. This allows systems to absorb demand surges and provider limits without brownouts or throttling surprises.

The approach represents a fundamental shift in how enterprises should think about AI reliability. “TrueFailover is designed to handle that complexity automatically,” Bajaj said. “It continuously monitors how models behave across many customers and use cases, looks for early warning signs like rising latency, and takes action before things break. Most individual enterprises do not have that kind of visibility because they are only able to see their own systems.”

The engineering challenge of switching models without sacrificing output quality

One of the thorniest challenges in AI failover involves maintaining consistent output quality when switching between models. A prompt optimized for GPT-5 may produce different results on Claude or Gemini. TrueFoundry addresses this through several mechanisms that balance speed against precision.

“Some teams rely on the fact that large models have become good enough that small differences in prompts do not materially affect the output,” Bajaj explained. “In those cases, switching from one provider to another can happen with some visible impact — that’s not ideal, but some teams choose to do it.”

More sophisticated implementations maintain provider-specific prompts for the same application. “When traffic shifts from one model to another, the prompt shifts with it,” Bajaj said. “In that case, failover is not just switching models. It is switching to a configuration that has already been tested.”

TrueFailover automates this process. The system dynamically routes requests and adjusts prompts based on which model handles the query, keeping quality within acceptable ranges without manual intervention. The key, Bajaj emphasized, is that “failover is planned, not reactive. The logic, prompts, and guardrails are defined ahead of time, which is why end users typically do not notice when a switch happens.”

Importantly, many failover scenarios do not require changing providers at all. “It can be routing traffic from the same model in one region to another region, such as from the East Coast to the West Coast, where no prompt changes are required,” Bajaj noted. This geographic flexibility provides a first line of defense before more complex cross-provider switches become necessary.

How regulated industries can use AI failover without compromising compliance

For enterprises in healthcare, financial services, and other regulated sectors, the prospect of AI traffic automatically routing to different providers raises immediate compliance concerns. Patient data cannot simply flow to whichever model happens to be available. Financial records require strict controls over where they travel. TrueFoundry built explicit guardrails to address these constraints.

“TrueFailover will never route data to a model or provider that an enterprise has not explicitly approved,” Bajaj said. “Everything is controlled through an admin configuration layer where teams set clear guardrails upfront.”

Enterprises define exactly which models qualify for failover, which providers can receive traffic, and even which regions or model categories — such as closed-source versus open-source — are acceptable. Once those rules take effect, TrueFailover operates only within them.

“If a model is not on the approved list, it is simply not an option for routing,” Bajaj emphasized. “There is no scenario where traffic is automatically sent somewhere unexpected. The idea is to give teams full control over compliance and data boundaries, while still allowing the system to respond quickly when something goes wrong. That way, reliability improves without compromising security or regulatory requirements.”

This design reflects lessons learned from TrueFoundry’s existing enterprise deployments. A Fortune 50 healthcare company already uses the platform to handle more than 500 million IVR calls annually through an agentic AI system. That customer required the ability to run workloads across both cloud and on-premise infrastructure while maintaining strict data residency controls — exactly the kind of hybrid environment where failover policies must be precisely defined.

Where automatic failover cannot help and what enterprises must plan for

TrueFoundry acknowledges that TrueFailover cannot solve every reliability problem. The system operates within the guardrails enterprises configure, and those configurations determine what protection is possible.

“If a team allows failover from a large, high-capacity model to a much smaller model without adjusting prompts or expectations, TrueFailover cannot guarantee the same output quality,” Bajaj explained. “The system can route traffic, but it cannot make a smaller model behave like a larger one without appropriate configuration.”

Infrastructure constraints also limit protection. If an enterprise hosts its own models and all of them run on the same GPU cluster, TrueFailover cannot help when that infrastructure fails. “When there is no alternate infrastructure available, there is nothing to fail over to,” Bajaj said.

The question of simultaneous multi-provider failures occasionally surfaces in enterprise risk discussions. Bajaj argues this scenario, while theoretically possible, rarely matches reality. “In practice, ‘going down’ usually does not mean an entire provider is offline across all models and regions,” he explained. “What happens far more often is a slowdown or disruption in a specific model or region because of traffic spikes or capacity issues.”

When that occurs, failover can happen at multiple levels — from on-premise to cloud, cloud to on-premise, one region to another, one model to another, or even within the same provider before switching providers entirely. “That alone makes it very unlikely that everything fails at once,” Bajaj said. “The key point is that reliability is built on layers of redundancy. The more providers, regions, and models that are included in the guardrails, the smaller the chance that users experience a complete outage.”

A startup that built its platform inside Fortune 500 AI deployments

TrueFoundry has established itself as infrastructure for some of the world’s largest AI deployments, providing crucial context for its failover ambitions. The company raised $19 million in Series A funding in February 2025, led by Intel Capital with participation from Eniac Ventures, Peak XV Partners, and Jump Capital. Angel investors including Gokul Rajaram and Mohit Aron also joined the round, bringing total funding to $21 million.

The San Francisco-based company was founded in 2021 by Bajaj and co-founders Abhishek Choudhary and Anuraag Gutgutia, all former Meta engineers who met as classmates at IIT Kharagpur. Initially focused on accelerating machine learning deployments, TrueFoundry pivoted to support generative AI capabilities as the technology went mainstream in 2023.

The company’s customer roster demonstrates enterprise-scale adoption that few AI infrastructure startups can match. Nvidia employs TrueFoundry to build multi-agent systems that optimize GPU cluster utilization across data centers worldwide — a use case where even small improvements in utilization translate into substantial business impact given the insatiable demand for GPU capacity. Adopt AI routes more than 15 million requests and 40 billion input tokens through TrueFoundry’s AI Gateway to power its enterprise agentic workflows.

Gaming company Games 24×7 serves machine learning models to more than 100 million users through the platform at scales exceeding 200 requests per second. Digital adoption platform Whatfix migrated to a microservices architecture on TrueFoundry, reducing its release cycle sixfold and cutting testing time by 40 percent.

TrueFoundry currently reports more than 30 paid customers worldwide and has indicated it exceeded $1.5 million in annual recurring revenue last year while quadrupling its customer base. The company manages more than 1,000 clusters for machine learning workloads across its client base.

TrueFailover will be offered as an add-on module on top of the existing TrueFoundry AI Gateway and platform, with pricing following a usage-based model tied to traffic volume along with the number of users, models, providers, and regions involved. An early access program for design partners opens in the coming weeks.

Why traditional cloud uptime guarantees may never apply to AI providers

Enterprise technology buyers have long demanded uptime commitments from infrastructure providers. Amazon Web Services, Microsoft Azure, and Google Cloud all offer service-level agreements with financial penalties for failures. Will AI providers eventually face similar expectations?

Bajaj sees fundamental constraints that make traditional SLAs difficult to achieve in the current generation of AI infrastructure. “Most foundational LLMs today operate as shared resources, which is what enables the standard pricing you see publicly advertised,” he explained. “Providers do offer higher uptime commitments, but that usually means dedicated capacity or reserved infrastructure, and the cost increases significantly.”

Even with substantial budgets, enterprises face usage quotas that create unexpected exposure. “If traffic spikes beyond those limits, requests can still spill back into shared infrastructure,” Bajaj said. “That makes it hard to achieve the kind of hard guarantees enterprises are used to with cloud providers.”

The economics of running large language models create additional barriers that may persist for years. “LLMs are still extremely complex and expensive to run. They require massive infrastructure and energy, and we do not expect a near-term future where most companies run multiple, fully dedicated model instances just to guarantee uptime.”

This reality drives demand for solutions like TrueFailover that provide resilience regardless of what individual providers can promise. “Enterprises are realizing that reliability cannot come from the model provider alone,” Bajaj said. “It requires additional layers of protection to handle the realities of how these systems operate today.”

The new calculus for companies that built AI into critical business processes

The timing of TrueFoundry’s announcement reflects a fundamental shift in how enterprises use AI — and what they stand to lose when it fails. What began as internal experimentation has evolved into customer-facing applications where disruptions directly affect revenue and reputation.

“Many enterprises experimented with Gen AI and agentic systems in the past, and production use cases were largely internal-facing,” Bajaj observed. “There was no immediate impact on their top line or the public perception of the enterprise.”

That era has ended. “Now that these enterprises have launched public-facing applications, where both the top line and public perception can be impacted if an outage occurs, the stakes are much higher than they were even six months ago. That’s why we are seeing more and more attention on this now.”

For companies that have woven AI into critical business processes — from prescription refills to customer support to sales operations — the calculus has changed entirely. The question is no longer which model performs best on benchmarks or which provider offers the most compelling features. The question that now keeps technology leaders awake is far simpler and far more urgent: what happens when the AI disappears at the worst possible moment?

Somewhere, a pharmacist is filling a prescription. A customer support agent is resolving a complaint. A sales team is generating a proposal for a deal that closes tomorrow. All of them depend on AI systems that depend on providers that, despite their scale and sophistication, still go dark without warning.

TrueFoundry is betting that enterprises will pay handsomely to ensure those moments of darkness never reach the people who matter most — their customers.

Researchers at Google have developed a technique that makes it easier for AI models to learn complex reasoning tasks that usually cause LLMs to hallucinate or fall apart. Instead of training LLMs through next-token prediction, their technique, called internal reinforcement learning (internal RL), steers the model’s internal activations toward developing a high-level step-by-step solution for the input problem. 

Ultimately, this could provide a scalable path for creating autonomous agents that can handle complex reasoning and real-world robotics without needing constant, manual guidance.

The limits of next-token prediction

Reinforcement learning plays a key role in post-training LLMs, particularly for complex reasoning tasks that require long-horizon planning. However, the problem lies in the architecture of these models. LLMs are autoregressive, meaning they generate sequences one token at a time. When these models explore new strategies during training, they do so by making small, random changes to the next single token or action. This exposes a deeper limitation: next-token prediction forces models to search for solutions at the wrong level of abstraction, making long-horizon reasoning inefficient even when the model “knows” what to do.

This token-by-token approach works well for basic language modeling but breaks down in long-horizon tasks where rewards are sparse. If the model relies solely on random token-level sampling, the probability of stumbling upon the correct multi-step solution is infinitesimally small, “on the order of one in a million,” according to the researchers.

The issue isn’t just that the models get confused; it’s that they get confused at the wrong level. In comments provided to VentureBeat, Yanick Schimpf, a co-author of the paper, notes that in a 20-step task, an agent can get lost in the minute details of a single step, or it can lose track of the overall goal.

“We argue that when facing a problem with some abstract structure… [goal-oriented exploration] is what you want,” Schimpf said. By solving the problem at the abstract level first, the agent commits to a path, ensuring it doesn’t “get lost in one of the reasoning steps” and fail to complete the broader workflow.

To address this, the field has long looked toward hierarchical reinforcement learning. HRL attempts to solve complex problems by decomposing them into a hierarchy of temporally abstract actions (high-level subroutines that represent different stages of the solution) rather than managing a task as a string of tokens. 

However, discovering these appropriate subroutines remains a longstanding challenge. Current HRL methods often fail to discover proper policies, frequently “converging to degenerate options” that do not represent meaningful behaviors. Even sophisticated modern methods like GRPO (a popular RL algorithm used for sparse-reward tasks) fail in complex environments because they cannot effectively bridge the gap between low-level execution and high-level planning.

Steering the LLM’s internal thoughts

To overcome these limitations, the Google team proposed internal RL. Advanced autoregressive models already “know” how to perform complex, multi-step tasks internally, even if they aren’t explicitly trained to do so.

Because these complex behaviors are hidden inside the model’s residual stream (i.e., the numerical values that carry information through the network’s layers), the researchers introduced an “internal neural network controller,” or metacontroller. Instead of monitoring and changing the output token, the metacontroller controls the model’s behavior by applying changes to the model’s internal activations in the middle layers.

This nudge steers the model into a specific useful state. The base model then automatically generates the sequence of individual steps needed to achieve that goal because it has already seen those patterns during its initial pretraining. 

The metacontroller operates through unsupervised learning and does not require human-labeled training examples. Instead, the researchers use a self-supervised framework where the model analyzes a full sequence of behavior and works backward to infer the hidden, high-level intent that best explains the actions.

During the internal RL phase, the updates are applied to the metacontroller, which shifts training from next-token prediction to learning high-level actions that can lead to the solution.

To understand the practical value of this, consider an enterprise agent tasked with code generation. Today, there is a difficult trade-off: You need “low temperature” (predictability) to get the syntax right, but “high temperature” (creativity) to solve the logic puzzle.

“Internal RL might facilitate this by allowing the model to explore the space of abstract actions, i.e. structuring logic and method calls, while delegating the token-level realization of those actions to the robust, lower-temperature distribution of the base model,” Schimpf said. The agent explores the solution without breaking the syntax.

The researchers investigated two methods for applying this controller. In the first, the base autoregressive model is pretrained on a behavioral dataset and then frozen, while the metacontroller is trained to steer the frozen model’s residual stream. In the second, the metacontroller and the base model are jointly optimized, with parameters of both networks updated simultaneously. 

Internal RL in action

To evaluate the effectiveness of internal RL, the researchers ran experiments across hierarchical environments designed to stump traditional learners. These included a discrete grid world and a continuous control task where a quadrupedal “ant” robot must coordinate joint movements. Both environments used sparse rewards with very long action sequences.

While baselines like GRPO and CompILE failed to learn the tasks within a million episodes due to the difficulty of credit assignment over long horizons, internal RL achieved high success rates with a small number of training episodes. By choosing high-level goals rather than tiny steps, the metacontroller drastically reduced the search space. This allowed the model to identify which high-level decisions led to success, making credit assignment efficient enough to solve the sparse reward problem.

Notably, the researchers found that the “frozen” approach was superior. When the base model and metacontroller were co-trained from scratch, the system failed to develop meaningful abstractions. However, applied to a frozen model, the metacontroller successfully discovered key checkpoints without any human labels, perfectly aligning its internal switching mechanism with the ground-truth moments when an agent finished one subgoal and started the next.

As the industry currently fixates on reasoning models that output verbose “chains of thought” to solve problems, Google’s research points toward a different, perhaps more efficient future.

“Our study joins a growing body of work suggesting that ‘internal reasoning’ is not only feasible but potentially more efficient than token-based approaches,” Schimpf said. “Moreover, these silent ‘thoughts’ can be decoupled from specific input modalities — a property that could be particularly relevant for the future of multi-modal AI.”

If internal reasoning can be guided without being externalized, the future of AI agents may hinge less on prompting strategies and more on how well we can access and steer what models already represent internally. For enterprises betting on autonomous systems that must plan, adapt, and act over long horizons, that shift could matter more than any new reasoning benchmark.

As agentic AI moves from experiments to real production workloads, a quiet but serious infrastructure problem is coming into focus: memory. Not compute. Not models. Memory.

Under the hood, today’s GPUs simply don’t have enough space to hold the Key-Value (KV) caches that modern, long-running AI agents depend on to maintain context. The result is a lot of invisible waste — GPUs redoing work they’ve already done, cloud costs climbing, and performance taking a hit. It’s a problem that’s already showing up in production environments, even if most people haven’t named it yet.

At a recent stop on the VentureBeat AI Impact Series, WEKA CTO Shimon Ben-David joined VentureBeat CEO Matt Marshall to unpack the industry’s emerging “memory wall,” and why it’s becoming one of the biggest blockers to scaling truly stateful agentic AI — systems that can remember and build on context over time. The conversation didn’t just diagnose the issue; it laid out a new way to think about memory entirely, through an approach WEKA calls token warehousing.

The GPU memory problem

“When we’re looking at the infrastructure of inferencing, it is not a GPU cycles challenge. It’s mostly a GPU memory problem,” said Ben-David.

The root of the issue comes down to how transformer models work. To generate responses, they rely on KV caches that store contextual information for every token in a conversation. The longer the context window, the more memory those caches consume, and it adds up fast. A single 100,000-token sequence can require roughly 40GB of GPU memory, noted Ben-David.

That wouldn’t be a problem if GPUs had unlimited memory. But they don’t. Even the most advanced GPUs top out at around 288GB of high-bandwidth memory (HBM), and that space also has to hold the model itself.

In real-world, multi-tenant inference environments, this becomes painful quickly. Workloads like code development or processing tax returns rely heavily on KV-cache for context.

“If I’m loading three or four 100,000-token PDFs into a model, that’s it — I’ve exhausted the KV cache capacity on HBM,” said Ben-David. This is what’s known as the memory wall. “Suddenly, what the inference environment is forced to do is drop data,” he added.

That means GPUs are constantly throwing away context they’ll soon need again, preventing agents from being stateful and maintaining conversations and context over time

The hidden inference tax

“We constantly see GPUs in inference environments recalculating things they already did,” Ben-David said. Systems prefill the KV cache, start decoding, then run out of space and evict earlier data. When that context is needed again, the whole process repeats — prefill, decode, prefill again. At scale, that’s an enormous amount of wasted work. It also means wasted energy, added latency, and degraded user experience — all while margins get squeezed.

That GPU recalculation waste shows up directly on the balance sheet. Organizations can suffer nearly 40% overhead just from redundant prefill cycles This is creating ripple effects in the inference market.

“If you look at the pricing of large model providers like Anthropic and OpenAI, they are actually teaching users to structure their prompts in ways that increase the likelihood of hitting the same GPU that has their KV cache stored,” said Ben-David. “If you hit that GPU, the system can skip the prefill phase and start decoding immediately, which lets them generate more tokens efficiently.”

But this still doesn’t solve the underlying infrastructure problem of extremely limited GPU memory capacity.

Solving for stateful AI

“How do you climb over that memory wall? How do you surpass it? That’s the key for modern, cost- effective inferencing,” Ben-David said. “We see multiple companies trying to solve that in different ways.”

Some organizations are deploying new linear models that try to create smaller KV caches. Others are focused on tackling cache efficiency.

“To be more efficient, companies are using environments that calculate the KV cache on one GPU and then try to copy it from GPU memory or use a local environment for that,” Ben-David explained. “But how do you do that at scale in a cost-effective manner that doesn’t strain your memory and doesn’t strain your networking? That’s something that WEKA is helping our customers with.”

Simply throwing more GPUs at the problem doesn’t solve the AI memory barrier. “There are some problems that you cannot throw enough money at to solve,” Ben-David said.

Augmented memory and token warehousing, explained

WEKA’s answer is what it calls augmented memory and token warehousing — a way to rethink where and how KV cache data lives. Instead of forcing everything to fit inside GPU memory, WEKA’s Augmented Memory Grid extends the KV cache into a fast, shared “warehouse” within its NeuralMesh architecture.

In practice, this turns memory from a hard constraint into a scalable resource — without adding inference latency. WEKA says customers see KV cache hit rates jump to 96–99% for agentic workloads, along with efficiency gains of up to 4.2x more tokens produced per GPU.

Ben-David put it simply: “Imagine that you have 100 GPUs producing a certain amount of tokens. Now imagine that those hundred GPUs are working as if they’re 420 GPUs.”

For large inference providers, the result isn’t just better performance — it translates directly to real economic impact.

“Just by adding that accelerated KV cache layer, we’re looking at some use cases where the savings amount would be millions of dollars per day,” said Ben-David

This efficiency multiplier also opens up new strategic options for businesses. Platform teams can design stateful agents without worrying about blowing up memory budgets. Service providers can offer pricing tiers based on persistent context, with cached inference delivered at dramatically lower cost.

What comes next

NVIDIA projects a 100x increase in inference demand as agentic AI becomes the dominant workload. That pressure is already trickling down from hyperscalers to everyday enterprise deployments— this isn’t just a “big tech” problem anymore.

As enterprises move from proofs of concept into real production systems, memory persistence is becoming a core infrastructure concern. Organizations that treat it as an architectural priority rather than an afterthought will gain a clear advantage in both cost and performance.

The memory wall is not something organizations can simply outspend to overcome. As agentic AI scales, it is one of the first AI infrastructure limits that forces a deeper rethink, and as Ben-David’s insights made clear, memory may also be where the next wave of competitive differentiation begins.

Presented by NetSuite

Most companies racing from startup to an industry leader face a choice: limp along with scrappy early systems or endure a costly platform migration.

DoorDash did neither. The local-commerce giant scaled from its 2013 founding through IPO and global expansion — acquiring the Helsiniki-based technology company Wolt in 2022 and UK-based Deliveroo in 2025 — while keeping its original Oracle NetSuite business system. Today, it serves over 50 million consumers in more than 40 countries.*

Chief Accounting Officer Gordon Lee says the secret is building a scalable ecosystem that allows teams to use tools that work best for them.

Choosing flexibility over uniformity

When DoorDash selected NetSuite as its corporate financial control center, it wasn’t looking for a system to enforce uniformity. It sought a scalable platform that could connect all its systems, from ERP, CRM, HR, sourcing, and more.

“Our philosophy has been to create a platform that allows our customers and business partners to use whatever tools work best for them,” Lee says. “When we’re managing growth, the majority of the conversation is about managing expectations — what people expect when you grow from A to B.”

The migration question

Two years after its founding, DoorDash surpassed one million deliveries and expanded into Canada. As the company scaled, Lee faced growing pressure from vendors insisting that rapid growth required a new enterprise platform.

He ran the numbers. The move to another platform could cost millions and consume months of his team’s focus.

Instead, DoorDash stayed with NetSuite, which continued to scale alongside the company’s growth. Built on Oracle Cloud Infrastructure, NetSuite delivers the performance and reliability of an enterprise platform without the cost or disruption of migration.

Lee concluded: “Why do I bother to move? I already have the scalability I need from NetSuite.”

Today, DoorDash’s NetSuite backend provides enterprise-grade security while its familiar front end provides the team flexibility, creating a stable, modern foundation for sustained, high-velocity growth.

Expanding the menu without the technical indigestion

That flexibility soon proved invaluable. The ability to add new applications quickly — without long, costly integrations — became a major advantage during hypergrowth.

For example, as DoorDash expanded from restaurant delivery into grocery, convenience, and retail, Lee turned to NetSuite’s inventory modules to handle the distinct demands of those new categories.

“The flexibility to have and not have, and turn the switch on and off, is easy because it’s all integrated,” he explains.

Today, DoorDash’s technology stack spans multiple systems — all integrating seamlessly with NetSuite as the financial hub. “They do it, and you’re done,” Lee says.

Embedding expertise to scale smarter, not bigger

For Lee, true partnerships turn vendors into part of the team — and that’s exactly how he describes NetSuite Advanced Customer Support (ACS).

“They are here with us every week. They know all my schematics, they know all my data infrastructure, they know all my database structure within NetSuite. Essentially, they are an extension of my team,” Lee explains.

Close collaboration benefits both parties. DoorDash keeps NetSuite attuned to the realities of hypergrowth and gets instant feedback on technology capability and scalability. In turn, NetSuite stays close to a marquee customer. Interaction is ongoing — and frank, according to Lee.

“We work directly with NetSuite ACS and often ask, ‘Can NetSuite do this?’ If they can prove it can, we stay with NetSuite.”

Another benefit is the ability to extend DoorDash’s expertise without expanding headcount.

“If someone says to me, ‘Gordon, you’re just an accountant. How do you know about systems? I say, I don’t. I have a network guy with us, an expert.’ That’s the kind of partner I want to surround myself with, so that I can grow beyond what I am.”

By embedding expertise within our partnerships, DoorDash scales with precision and control. Lee says the model applies to other companies preparing for IPOs or global expansion. He adds that sustainable growth depends as much on shared understanding as on technology itself.

Too often, finance and IT “look at the same requirement but see completely different things,” Lee says, describing what he calls the “blue versus purple” problem. “The accountant doesn’t understand the configuration of the system,” he explains. “The IT guy doesn’t understand what the accountant was trying to tell them.”

NetSuite bridges that gap. With a unified data model and built-in best practices across finance, operations, and more, it keeps teams aligned and information consistent. That close collaboration, Lee notes, is what keeps rollouts smooth, data clean, and growth sustainable at any stage.

AI strategy: Trust only internal data, get data ducks in a row

Lee plans to test the NetSuite AI Connector Service — which supports Model Context Protocol (MCP) and lets customers connect their own AI to NetSuite — to see how faster access to accurate data can drive growth.

By implementing an internal instance, Lee is less worried about disruptive errors from LLMs trained on public data sources.

“Think about a generative AI chatbot. When you ask a question, it can reflect many perspectives,” he explains. On the other hand, a chatbot trained on private enterprise systems benefits from “a clean data infrastructure.”

Lee is taking a methodical approach: first get data pristine, then train AI on domain-specific terminology, and finally see how internal AI can both find the right information and automate downstream accounting processes to save resources and accelerate growth.

Betting long-term on its original financial core

From early growth to major acquisitions that helped expand its footprint across the globe, DoorDash has relied on NetSuite as a consistent foundation for innovation and scale.

Lee credits NetSuite’s flexible architecture and close partnership with helping enable DoorDash as it continued to scale and cement itself as a leader in local commerce globally.

His mantra is simple: “Focus on growth instead of churning through vendors.”

* Based on the combined numbers for DoorDash, Wolt, and Deliveroo, measured as of September 2025.

Sponsored articles are content produced by a company that is either paying for the post or has a business relationship with VentureBeat, and they’re always clearly marked. For more information, contact sales@venturebeat.com.