How Encord Uses Brain Wave Sensors, EMG, and Teleoperation to Build Training Data for Physical AI

EMG (Muscle Signal) Sensors

Encord is simultaneously testing sensors strapped to the forearm that detect electrical signals in muscles. Standard video of hands often occludes fine finger positions, so the company hopes to reconstruct 3D hand pose from these EMG signals, giving models a richer understanding of dexterous manipulation .

Teleoperation Data

At its San Leandro R&D facility, Encord operates leader-follower robotic arms. A human operator directly controls one arm while a second arm mimics the movement, producing high-fidelity data for tasks such as pouring coffee, stacking poker chips, plugging and unplugging ethernet cables, and household object manipulation . According to Encord, every humanoid robotics company has requested these task-specific datasets .

Egocentric (First-Person) Video

Pilots wear head-mounted cameras that capture what they see, supplemented by data drawn from several factories around the globe. This provides a scalable source of real-world observation data .

In-Field and Lab-Based Collection

Encord deploys dedicated in-field operators and operates reconfigurable lab environments to collect embodiment-specific data, including LiDAR, point cloud, and synchronized multimodal sensor streams . The collection protocols are designed backward from a customer's training pipeline, so that every episode is classified, synchronized, and aligned to the target model . The company also captures failure modes through a deployment feedback loop: when a model fails in the field, the failure is captured via remote teleoperation and fed back into data collection policies .

Key Technical Challenges

Proving the Value of Neural Data

Zander Labs' approach infers mental states from brain activity at a coarse level — measuring how much cognitive effort a pilot is using, for example — rather than decoding specific thoughts . The trial must validate whether this signal actually improves robotic model performance compared to standard video and teleoperation data alone. If the improvement is marginal, the cost of adding EEG to every data collection session may not be justified.

Multimodal Synchronization and Annotation

Physical AI requires sensors such as video, depth, LiDAR, audio, IMU, and now EEG/EMG to be precisely time-aligned and annotated together. Encord has built support for native formats (MCAP, ROSBAG) and sensor fusion, but managing the complexity of synchronizing 2D, 3D, and biosignal streams remains a hard engineering problem .

Data Diversity and Failure Mode Capture

Models need exposure to varied environments, lighting, operators, and failure cases. Encord incorporates a deployment feedback loop, but failure data is widely considered underrated yet essential for robust physical AI . Without enough edge cases, models trained on even the richest data will generalize poorly.

Key Economic Challenges

Extreme Cost of Physical Data Creation

Unlike LLMs, which train on text scraped from the internet, physical AI requires proprietary, real-world data generated by expensive hardware, human operators, and controlled environments. Encord's model represents a deliberate shift toward more expensive, specialized manufacturing of data — it scales less cheaply than web scraping .

High Capital Requirements

Encord has raised $110 million total, including a $60 million Series C in February 2026 led by Wellington Management . The company must spend heavily on facilities (its Bay Area R&D lab), robotic hardware, trained personnel ("pilots"), and data annotation infrastructure — all before most customers have deployed production robots at scale.

Simulation Gap

Simulation can generate large synthetic datasets cheaply, but it systematically fails to capture real-world physics, edge cases, and fine-grained manipulation. Encord and others are betting that only real human demonstration data — with its cost and mess — will bridge the sim-to-real gap needed for general-purpose robotics .

Market Timing Risk

Encord anticipates that over 400 million intelligent robots will come online in the next four years, pushing the physical AI industry past $30 billion annually . If adoption is slower than expected, the demand for expensive custom training data may not materialize fast enough to sustain the current infrastructure spending.

Bottom Line

Encord is pioneering experimental biosignal capture (brain waves and muscle sensors) alongside teleoperation, egocentric video, and lab-based physical data collection to address the industry's shortage of real-world training data. The approach is still unproven at scale: the brain wave trial is in active evaluation, and the core economic tradeoff is that physical data is far more expensive to produce than web-scale text data. Success depends on whether these richer data modalities meaningfully improve robotic model performance and whether the physical AI market grows fast enough to justify the upfront investment.