How Jetson AGX Thor and AGX Orin Run Reasoning AI at the Edge

Running multi-step reasoning models on an embedded device has traditionally meant trading away either model capability or responsiveness. NVIDIA’s September 4 developer guide makes the case that this is changing: optimized open models, low-precision inference, and faster decoding can put capable reasoning and agentic AI directly on Jetson AGX Orin and Jetson AGX Thor modules. 1

The headline result is up to 6.28× higher decode throughput than BF16 on Jetson AGX Thor when NVFP4 quantization and speculative decoding are combined. That is a model- and workload-specific benchmark result—not a universal tokens-per-second guarantee—but it illustrates why local AI agents are becoming more practical for robotics, vehicles, and industrial equipment. 1

The deployment stack: serve locally, reduce precision, decode more efficiently

NVIDIA’s guide identifies Nemotron 3.5 Lightning and Qwen3.8-27B as options for Jetson AGX Orin and Jetson AGX Thor. Developers can serve these models with vLLM on Jetson, then use two complementary techniques to increase inference efficiency. 1

NVFP4 quantization

NVFP4 uses four-bit floating-point representations to reduce the work and memory required for model operations. Lower memory movement is particularly important in embedded inference, where the practical limit is often not just raw compute but moving weights and activations quickly enough during generation. 1

Jetson Thor is especially suited to this approach because NVIDIA says the platform’s Blackwell GPU includes native FP4 support through its Transformer Engine. 3

Speculative decoding

Speculative decoding separates proposal from verification. A smaller draft model proposes several next tokens; the main model verifies them together and still makes the final decision. When several tokens are accepted, generation advances multiple tokens in one verification step rather than one token at a time. 1

The benefit depends on how often draft tokens are accepted, so performance varies by model, prompt, and decoding configuration. This is why speculative decoding should be treated as a workload optimization, not a fixed multiplier.

Nemotron 3.5 Lightning vs. Qwen3.8-27B

The two models represent different edge-deployment tradeoffs.

Nemotron 3.5 Lightning: capacity with fewer active parameters

Nemotron 3.5 Lightning is a 30-billion-parameter mixture-of-experts model, but it activates only 3 billion parameters per token. NVIDIA describes it as a model for the execution layer of long-running, always-on agents. 2

That architecture can be attractive when an agent needs to generate frequent responses or repeatedly work through operational loops: interpret an event, call an approved tool, inspect the result, and decide whether to continue or escalate. The relevant measure is not only total parameter count, but active parameters per token and sustained generation speed.

Qwen3.8-27B: dense computation for harder decisions

Qwen3.8-27B is a dense model, meaning all 27 billion parameters participate for each token. NVIDIA positions it as a strong Jetson option alongside Nemotron 3.5 Lightning. 1

A dense model can make sense when an application has fewer decisions but places more value on each one. The tradeoff is that activating the entire network per token can demand more compute than a sparse MoE model.

Choose by workflow, not headline parameter count

NVIDIA’s guidance is effectively to benchmark against the real application: its prompt lengths, response lengths, tools, latency budget, memory limits, and pattern of decisions. 1

A response-heavy agent may favor a sparse model with fast output generation. A system making fewer, more complex judgments may accept slower generation in exchange for the behavior it gets from a dense model. Neither architecture is automatically better for every edge deployment.

What the 6.28× Jetson AGX Thor result means

NVIDIA reports a maximum 6.28× decode-throughput improvement relative to BF16 after combining NVFP4 quantization with speculative decoding on Jetson AGX Thor. The preferred speculative-decoding configuration differed by model: NVIDIA found DSpark worked best for Nemotron 3.5 Lightning, while DFlash2 worked best for Qwen3.8-27B. 1

The result matters because decoding—the step-by-step creation of output tokens—often defines an interactive agent’s perceived responsiveness. It should not, however, be read as a blanket speed claim for every prompt, model, or deployment. Draft-model quality, token acceptance rate, batch size, context length, and runtime configuration all affect real throughput. 1

Why local inference changes the edge-AI design

An on-device model avoids making every reasoning request depend on a remote data center. For physical systems, that can mean less network-related delay, continued operation when connectivity is intermittent, and the ability to keep sensor streams and operational logs on the device. NVIDIA specifically frames edge reasoning and agentic AI as valuable in situations where data locality and real-time response matter. 1

This does not eliminate the role of cloud infrastructure. Organizations may still use centralized systems for model training, fleet management, large-scale analysis, or tasks that exceed the device’s available memory and compute. The important shift is that a local connection to a data center no longer has to be on the critical path for every decision.

Applications that benefit most

NVIDIA highlights in-cabin assistants, real-time anomaly detection, and robots operating in harsh or remote environments. 1

These examples share a practical requirement: the system must interpret local context and respond quickly enough to be useful. Potential fits include:

• robots that need to reason over observations and task status;
• industrial systems that interpret equipment events or operational logs;
• vehicle assistants that must respond within an in-cabin interaction;
• remote inspection or monitoring equipment that cannot assume reliable connectivity.

The best candidates are not merely devices that can run an LLM. They are systems where response time, privacy or data-locality requirements, and offline resilience have direct operational value.

Where AGX Thor fits in NVIDIA’s Jetson roadmap

Jetson AGX Thor is NVIDIA’s high-end platform for demanding physical-AI workloads. NVIDIA says it combines a Blackwell GPU with 128 GB of memory and delivers up to 2,070 FP4 TFLOPS within a 130 W power envelope. 3

That hardware profile aligns with the guide’s focus on NVFP4, high-throughput decoding, and larger local reasoning models. AGX Orin remains relevant for established embedded deployments, while Thor targets more demanding generative and multimodal workloads.

At the entry tier, NVIDIA has announced Jetson Orin Nano 2 for robotics, delivery and inspection drones, and vision-AI systems. NVIDIA says the module and developer kit are expected in the first half of 2027. 1

Together, the portfolio points to a tiered approach: smaller systems for entry-level edge AI, AGX Orin for mature embedded deployments, and AGX Thor for applications that need more memory and compute for local generative reasoning.

Bottom line

NVIDIA’s guide does not claim that every frontier model can now run comfortably on every embedded module. Its more useful conclusion is narrower: compact reasoning models, vLLM deployment, NVFP4 quantization, and speculative decoding can make capable local agents feasible on Jetson-class hardware. 1

For developers, the next step is empirical rather than theoretical: test the target model on the target Jetson device using the real prompts, tools, context windows, and response-time requirements of the product. The 6.28× result shows the upside of optimization; the application benchmark determines whether that upside becomes a usable edge-AI system. 1