Kog Inference Engine: 3,000 Tokens/s on Standard GPUs Without Custom Silicon

The core innovation is a monokernel — a single, persistent GPU-resident program that runs the entire LLM decode pass (prefill, decode, LM-head sampling) in one shot, eliminating the per-token kernel launch overhead that plagues standard stacks . Standard inference frameworks like vLLM, SGLang, and TensorRT-LLM launch one GPU kernel per token, each paying roughly 4.5 μs launch overhead plus HBM restart latency . Kog's engine bypasses standard communication libraries and eliminates grid synchronizations that it measured as consuming 35% of per-token generation time .

The "30x faster" claim is calibrated against typical decoding speeds of 100–300 tokens/s for 2B–8B models on high-end GPUs, compared to Kog's 3,000 tokens/s .

Tech Preview Results and Clear Limitations

Results:

• 3,000+ output tokens per second per request on a single 8× AMD MI300X node with the Laneformer 2B model (FP16, batch size 1, no speculative decoding)
• 2,100 output tokens/s per request on 8× NVIDIA H200 under the same conditions
• Single-user latency low enough for real-time interactive use — the demo went live at playground.kog.ai
• The Laneformer 2B model achieved 45.1% HumanEval+ and 51.6% MBPP+ in greedy decoding

Limitations:

• The preview only ran on a 2.3B-parameter model (Laneformer 2B). Larger models were not demonstrated
• It was single-request, single-user — not a batched multi-user production serving setup, and did not show throughput under concurrent load
• Only two GPU types were supported: AMD MI300X and NVIDIA H200
• At 11 employees, the company is very small for the scope of its ambition

Target Use Cases and Early Business Traction

Kog targets three main scenarios:

• Agentic AI workflows — the company argues GPUs are poorly suited to the iterative, low-latency demands of AI agents, and that its engine fixes this
• Real-time LLM inference — coding copilots, chatbots, and applications where sub-100ms per-token latency matters
• Self-hosted enterprise inference — companies that already own datacenter GPUs and want to avoid buying dedicated inference hardware

On the business side, Kog reported 200+ tangible business leads as of August 2026, according to CEO Delalleau . The tech preview hit the front page of Hacker News in May 2026 . The company has raised a seed round co-led by Varsity VC, whose partner Kamel Zeroual was Delalleau's co-founder at his first startup, Stribe .

Why a Solid-State Physicist Turned Hacker Runs This Company

CEO Gaël Delalleau brings an unusual background to AI infrastructure: solid-state physics and offensive cybersecurity .

• Solid-state physics trained him to reason from first principles about how physical hardware behaves at a low level — electrons, memory bandwidth, transistor-level constraints. His team treats GPU optimization as a physics problem rather than a software engineering problem
• Offensive cybersecurity gave him a mentality of adversarial reverse-engineering. Kog's engine examines GPUs at the assembly and binary levels, finding hidden headroom that standard drivers and libraries don't expose — a classic "attacker's mindset" applied to hardware performance
• Delalleau is not a traditional ML researcher; his first startup (Stribe, a TechCrunch50 2009 alum) was unrelated to AI. He describes himself as a first-principles thinker who designs algorithms from the hardware up

Scaling Challenges: Team Size, Chip Support, and Model Scale

Kog faces three significant scaling hurdles:

• Team size: With just 11 employees, porting its deeply hand-optimized monokernel to new GPU architectures is labor-intensive
• New chip support: The engine currently supports only AMD MI300X and NVIDIA H200. Each new GPU requires manual reverse-engineering at the assembly level. CEO Delalleau acknowledges this is a bottleneck, though he expects newer GPU architectures with higher memory bandwidth to be easier targets
• Model scale: Scaling from 2B-parameter models to full-scale LLMs (70B+) represents what Delalleau calls "a huge leap." The techniques that work for a small model may not trivially extend to much larger memory and compute footprints

The Make-or-Break Milestone: Large Language Models

Kog's next critical milestone is proving its approach works on large language models (70B+ parameters). CEO Delalleau has stated the company is now focused on scaling its techniques to full-scale LLMs. Confidence comes from newer GPU architectures with substantially higher memory bandwidth — a resource Kog believes remains fundamentally underutilized by standard software stacks . The company has not announced a specific date, but this demonstration is widely seen as the make-or-break proof point for the entire thesis.