What is APXInf, the open-source embodied edge inference engine released by Infinigence AI with Tsinghua University and Shanghai Jiao Tong University, and how do its Rust runtime, Python bindings, model-porting workflow, and pipeline, graph, kernel, and quantization optimizations

APXInf is an open-source inference engine from Infinigence AI, Tsinghua University, and Shanghai Jiao Tong University for running embodied-AI policies close to a robot’s sensors and actuators. Its aim is to reduce the delay between an observation and a model-generated action; it is not itself the robot’s motor controller. 144

• How it works: A small Rust runtime manages execution and memory, while Python bindings let developers call policies from familiar robotics code. Its documented model-porting workflow starts with a fixed reference checkpoint and test inputs, then checks the port against reference outputs and intermediate values before optimizing it. 35
• Where the speed comes from: Pipeline scheduling targets delays across the full inference path; graph capture and reusable buffers avoid repeated setup; tuned GPU kernels accelerate expensive operations; and FP8 or INT8 quantization can reduce computation and memory traffic. Together, these are intended to make action generation fast and predictable enough for on-robot use. 3135
• Reported Thor result: For π0.5 in FP8 on Jetson Thor, the published comparison says end-to-end inference fell from 278 ms for the unoptimized baseline to under 26 ms, roughly a 10.7× latency reduction. It reports 38.46 inferences per second, versus about 3.6 per second implied by 278 ms. That is an inference rate, not proof that every robot’s complete sensor-to-actuator loop runs at 38.46 Hz. Some reports say 46 Hz, but the repeatedly stated 26 ms/38.46 Hz figures are mutually consistent, so 38.46 Hz is the more defensible figure for this comparison. 1486
• Models, hardware, and integration: Launch coverage names π0.5 and WALL-OSS and RTX 4090, Jetson Orin, and Jetson Thor; the repository specifically describes its highly optimized first-release π0.5 path on Thor and Orin, with BF16, FP8, and INT8 options. APXInf is the inference/deployment side of the RLinf ecosystem: RLinf documents APXInf-accelerated π0.5 evaluation while retaining OpenPI’s transforms. The robot-facing package also offers an OpenPI-compatible WebSocket server, which is an interface compatibility claim—not a claim that every OpenPI checkpoint is already ported. 14515
• Roadmap and limits: The stated plans include more VLA, VLM, and world models, with Qwen and GR00T being adapted, plus further platform/backend work; plans are not current support guarantees. The Thor number should not be transferred to another checkpoint, precision, device, or power limit without rerunning the benchmark: each can change latency, and inference speed alone does not establish closed-loop task success or sustained performance under a robot’s power and thermal constraints. 145