Why China’s AI Companies Are Stretching Every Nvidia GPU
China’s average daily AI token calls exceeded 140 trillion in March 2026, up from 100 billion in early 2024. Inference software is becoming the fastest way to increase useful output from scarce hardware: firms can route simpler requests to domestic chips while reserving Nvidia capacity for complex, high value tasks.
Published byEdited with GPT-5.6 LunaImages generated with GPT Image 1.5
China’s average daily AI token calls exceeded 140 trillion in March 2026, up from 100 billion in early 2024.
Inference software is becoming the fastest way to increase useful output from scarce hardware: firms can route simpler requests to domestic chips while reserving Nvidia capacity for complex, high value tasks.
The commercial problem is acute because premium inference consumes more compute while Chinese token prices are estimated to average about one sixth of comparable U.S.
Why are Chinese AI companies optimizing inference software and stretching scarce Nvidia GPU capacity as AI shifts from model training to larChina’s AI companies are optimizing inference to stretch a constrained mix of Nvidia and domestic computing hardware.
AI Prompt
Create a landscape editorial hero image for this Studio Global article: Why are Chinese AI companies optimizing inference software and stretching scarce Nvidia GPU capacity as AI shifts from model training to lar. Article summary: Chinese AI companies are shifting effort from training ever-larger models to serving huge volumes of real-time requests. They are optimizing inference software—such as scheduling workloads across mixed hardware, improvin. Topic tags: general, news, general web, user generated. Style: premium digital editorial illustration, source-backed research mood, clean composition, high detail, modern web publication hero. Use reference image context only for broad subject, composition, and topical grounding; do not copy the exact image. Avoid: logos, brand marks, copyrighted characters, real person likenesses, fake screenshots, UI text, readable text, watermarks, charts w
openai.com
China’s AI bottleneck is increasingly appearing after models have been trained. As consumer and enterprise applications move into large-scale deployment, companies must serve enormous volumes of requests—many of them generated by agents that reason, call tools and complete multi-step tasks. That makes inference capacity, not just model-training capacity, a central constraint.
The shift from training to serving
Training creates a model; inference is the later stage in which that model produces responses for users. Unlike training, some inference workloads can be adapted to domestic hardware. But the distinction is not absolute: demanding applications still require a combination of computing performance, memory, reliability and mature software that is difficult to replace.
Chinese AI companies therefore face a mixed-hardware problem. Domestic accelerators can handle parts of the inference workload, while a limited pool of high-end Nvidia processors must be reserved for the tasks least amenable to substitution. Industry reporting identifies complex coding workloads as a particular pressure point.
Why coding and AI agents consume scarce capacity
A simple question-and-answer interaction may require relatively modest computation. A coding assistant or autonomous agent can be much more demanding: it may need to maintain a long context, generate code, use external tools, inspect the result and revise its answer repeatedly.
That changes the economics of deployment. The relevant question is not how many chips a company owns, but how much reliable, high-quality work those chips can complete. If domestic processors perform less effectively on complex coding and reasoning workloads, shifting all inference away from Nvidia hardware can reduce quality or efficiency. Scarce Nvidia capacity must then be allocated to the jobs where the difference matters most.
Studio Global AI
Continue your research
This page includes a source-backed answer you can continue inside Studio Global.
What is the short answer to "Why China’s AI Companies Are Stretching Every Nvidia GPU"?
China’s average daily AI token calls exceeded 140 trillion in March 2026, up from 100 billion in early 2024.
What are the key points to validate first?
China’s average daily AI token calls exceeded 140 trillion in March 2026, up from 100 billion in early 2024. Inference software is becoming the fastest way to increase useful output from scarce hardware: firms can route simpler requests to domestic chips while reserving Nvidia capacity for complex, high value tasks.
What should I do next in practice?
The commercial problem is acute because premium inference consumes more compute while Chinese token prices are estimated to average about one sixth of comparable U.S.
Why China’s AI Companies Are Stretching Every Nvidia GPU
China’s average daily AI token calls exceeded 140 trillion in March 2026, up from 100 billion in early 2024. Inference software is becoming the fastest way to increase useful output from scarce hardware: firms can route simpler requests to domestic chips while reserving Nvidia capacity for complex, high value tasks.
Published byEdited with GPT-5.6 LunaImages generated with GPT Image 1.5
China’s average daily AI token calls exceeded 140 trillion in March 2026, up from 100 billion in early 2024.
Inference software is becoming the fastest way to increase useful output from scarce hardware: firms can route simpler requests to domestic chips while reserving Nvidia capacity for complex, high value tasks.
The commercial problem is acute because premium inference consumes more compute while Chinese token prices are estimated to average about one sixth of comparable U.S.
Why are Chinese AI companies optimizing inference software and stretching scarce Nvidia GPU capacity as AI shifts from model training to larChina’s AI companies are optimizing inference to stretch a constrained mix of Nvidia and domestic computing hardware.
AI Prompt
Create a landscape editorial hero image for this Studio Global article: Why are Chinese AI companies optimizing inference software and stretching scarce Nvidia GPU capacity as AI shifts from model training to lar. Article summary: Chinese AI companies are shifting effort from training ever-larger models to serving huge volumes of real-time requests. They are optimizing inference software—such as scheduling workloads across mixed hardware, improvin. Topic tags: general, news, general web, user generated. Style: premium digital editorial illustration, source-backed research mood, clean composition, high detail, modern web publication hero. Use reference image context only for broad subject, composition, and topical grounding; do not copy the exact image. Avoid: logos, brand marks, copyrighted characters, real person likenesses, fake screenshots, UI text, readable text, watermarks, charts w
openai.com
China’s AI bottleneck is increasingly appearing after models have been trained. As consumer and enterprise applications move into large-scale deployment, companies must serve enormous volumes of requests—many of them generated by agents that reason, call tools and complete multi-step tasks. That makes inference capacity, not just model-training capacity, a central constraint.
The shift from training to serving
Training creates a model; inference is the later stage in which that model produces responses for users. Unlike training, some inference workloads can be adapted to domestic hardware. But the distinction is not absolute: demanding applications still require a combination of computing performance, memory, reliability and mature software that is difficult to replace.
Chinese AI companies therefore face a mixed-hardware problem. Domestic accelerators can handle parts of the inference workload, while a limited pool of high-end Nvidia processors must be reserved for the tasks least amenable to substitution. Industry reporting identifies complex coding workloads as a particular pressure point.
Why coding and AI agents consume scarce capacity
A simple question-and-answer interaction may require relatively modest computation. A coding assistant or autonomous agent can be much more demanding: it may need to maintain a long context, generate code, use external tools, inspect the result and revise its answer repeatedly.
That changes the economics of deployment. The relevant question is not how many chips a company owns, but how much reliable, high-quality work those chips can complete. If domestic processors perform less effectively on complex coding and reasoning workloads, shifting all inference away from Nvidia hardware can reduce quality or efficiency. Scarce Nvidia capacity must then be allocated to the jobs where the difference matters most.
Studio Global AI
Continue your research
This page includes a source-backed answer you can continue inside Studio Global.
What is the short answer to "Why China’s AI Companies Are Stretching Every Nvidia GPU"?
China’s average daily AI token calls exceeded 140 trillion in March 2026, up from 100 billion in early 2024.
What are the key points to validate first?
China’s average daily AI token calls exceeded 140 trillion in March 2026, up from 100 billion in early 2024. Inference software is becoming the fastest way to increase useful output from scarce hardware: firms can route simpler requests to domestic chips while reserving Nvidia capacity for complex, high value tasks.
What should I do next in practice?
The commercial problem is acute because premium inference consumes more compute while Chinese token prices are estimated to average about one sixth of comparable U.S.
Token counts are useful indicators of demand, but they do not describe identical workloads. Simple, low-cost output can be directed to cheaper hardware. High-quality output from a stronger model, a longer reasoning process or repeated tool calls requires more compute and may be more valuable to customers.
This is why a surge in total tokens can understate the shortage of premium inference. The market may be producing more tokens overall while experiencing a sharper shortage of the hardware needed to produce reliable results for complex tasks. Investment in high-quality token capacity and systems that combine heterogeneous computing resources reflects that distinction.
Demand has grown faster than capacity
China’s average daily token calls exceeded 140 trillion in March 2026, compared with about 100 billion at the beginning of 2024—a rise of more than 1,000 times according to figures reported from China’s National Data Administration.
The increase signals a change in how AI is being used. Models are moving beyond experimentation and isolated prompts into assistants, enterprise software and agents that perform real-world workflows. Inference is consequently becoming a major driver of computing consumption rather than a secondary step after training.
Why software optimization is the fastest lever
Adding high-end processors is difficult when export controls restrict access to advanced Nvidia products, supply is limited and hardware ecosystems are expensive to replace. China’s AI developers also face substantial transition costs because existing models, tools and workflows are deeply tied to established software platforms.
Software optimization cannot create new silicon, but it can increase the amount of useful work extracted from each available processor. The practical approaches include:
Workload routing: sending simpler requests to cheaper or domestic accelerators and reserving Nvidia GPUs for complex tasks.
More efficient serving: improving how requests are grouped and scheduled so hardware spends less time idle.
Resource-aware deployment: matching model size, context length and response strategy to the value and difficulty of each request.
Heterogeneous infrastructure: integrating different chip architectures instead of requiring every workload to run on one platform.
These measures are a bridge between rising demand and constrained supply. They are not a complete substitute for frontier hardware, especially when quality-sensitive workloads remain dependent on a narrower set of processors.
The commercial squeeze
The shortage is harder to solve because inference prices are under pressure. Jefferies analysts estimated that Chinese models cost, on average, about one-sixth as much per token as offerings from major U.S. companies.
Low prices can accelerate adoption, but they also limit how much revenue is available to fund scarce premium compute. At the same time, coding assistants and agentic products can consume more resources than simple chat. Chinese AI companies therefore face a difficult combination: demand is expanding rapidly, customers expect inexpensive access, and the most capable inference capacity cannot be expanded or replaced easily.
That tension has already appeared in service availability. A report on the market described major Chinese model developers and cloud providers throttling services, rationing access or raising prices as demand exceeded available computing resources, particularly for resource-intensive coding assistants.
The bigger implication
China’s AI infrastructure challenge is not simply a shortage of chips. It is a shortage of the right combination of advanced processors, compatible software and economical capacity for high-value inference.
Domestic hardware can absorb more of the workload as software improves and deployments are designed around its strengths. But the current evidence points to a transitional strategy: optimize every layer of inference, use domestic chips where they are effective and preserve scarce Nvidia capacity for the hardest jobs. That can extend existing resources, but sustained growth in agent-based AI will ultimately depend on whether China can expand both its hardware supply and the software ecosystem that makes different processors practical to use.
scmp.comChina's top AI is still trained on Nvidia chips. What is delaying a ...
Token counts are useful indicators of demand, but they do not describe identical workloads. Simple, low-cost output can be directed to cheaper hardware. High-quality output from a stronger model, a longer reasoning process or repeated tool calls requires more compute and may be more valuable to customers.
This is why a surge in total tokens can understate the shortage of premium inference. The market may be producing more tokens overall while experiencing a sharper shortage of the hardware needed to produce reliable results for complex tasks. Investment in high-quality token capacity and systems that combine heterogeneous computing resources reflects that distinction.
Demand has grown faster than capacity
China’s average daily token calls exceeded 140 trillion in March 2026, compared with about 100 billion at the beginning of 2024—a rise of more than 1,000 times according to figures reported from China’s National Data Administration.
The increase signals a change in how AI is being used. Models are moving beyond experimentation and isolated prompts into assistants, enterprise software and agents that perform real-world workflows. Inference is consequently becoming a major driver of computing consumption rather than a secondary step after training.
Why software optimization is the fastest lever
Adding high-end processors is difficult when export controls restrict access to advanced Nvidia products, supply is limited and hardware ecosystems are expensive to replace. China’s AI developers also face substantial transition costs because existing models, tools and workflows are deeply tied to established software platforms.
Software optimization cannot create new silicon, but it can increase the amount of useful work extracted from each available processor. The practical approaches include:
Workload routing: sending simpler requests to cheaper or domestic accelerators and reserving Nvidia GPUs for complex tasks.
More efficient serving: improving how requests are grouped and scheduled so hardware spends less time idle.
Resource-aware deployment: matching model size, context length and response strategy to the value and difficulty of each request.
Heterogeneous infrastructure: integrating different chip architectures instead of requiring every workload to run on one platform.
These measures are a bridge between rising demand and constrained supply. They are not a complete substitute for frontier hardware, especially when quality-sensitive workloads remain dependent on a narrower set of processors.
The commercial squeeze
The shortage is harder to solve because inference prices are under pressure. Jefferies analysts estimated that Chinese models cost, on average, about one-sixth as much per token as offerings from major U.S. companies.
Low prices can accelerate adoption, but they also limit how much revenue is available to fund scarce premium compute. At the same time, coding assistants and agentic products can consume more resources than simple chat. Chinese AI companies therefore face a difficult combination: demand is expanding rapidly, customers expect inexpensive access, and the most capable inference capacity cannot be expanded or replaced easily.
That tension has already appeared in service availability. A report on the market described major Chinese model developers and cloud providers throttling services, rationing access or raising prices as demand exceeded available computing resources, particularly for resource-intensive coding assistants.
The bigger implication
China’s AI infrastructure challenge is not simply a shortage of chips. It is a shortage of the right combination of advanced processors, compatible software and economical capacity for high-value inference.
Domestic hardware can absorb more of the workload as software improves and deployments are designed around its strengths. But the current evidence points to a transitional strategy: optimize every layer of inference, use domestic chips where they are effective and preserve scarce Nvidia capacity for the hardest jobs. That can extend existing resources, but sustained growth in agent-based AI will ultimately depend on whether China can expand both its hardware supply and the software ecosystem that makes different processors practical to use.
scmp.comChina's top AI is still trained on Nvidia chips. What is delaying a ...
