TinyFish Launches Open-Source BigSet: AI Agents That Build Live Datasets from Plain English

How the Multi-Agent Pipeline Works

BigSet is not a single large language model call. It is a multi-agent system that delegates tasks to specialized sub-agents, with the orchestrator managing the overall workflow.

The documented pipeline works as follows :

1. Orchestrator Agent receives the user's natural language description and infers the target schema—determining which columns, data types, and entities the final table will contain. According to sources, this step uses models like Claude Sonnet for complex reasoning and planning .
2. Research Sub-Agents then fan out to execute the actual work. These agents are dispatched to conduct live-web research, browse real pages, extract specific data points, and collect evidence . The research is not just a semantic search; the agents are designed to visit and parse real web sources.
3. Verification Sub-Agents cross-check the findings. Sources indicate that BigSet verifies gathered data against its sources to flag or prevent hallucinated rows, a critical step for production-grade datasets .
4. Aggregation and Export — The collected rows are deduplicated and compiled into a clean table, which the user can download as a CSV or XLSX file .

The system is built for recurring work. Users can set a refresh cadence—from every 30 minutes to weekly—and the agents will re-run the full research and validation pipeline on schedule, ensuring datasets stay current without manual intervention .

License, Self-Hosting, and Data Sovereignty

BigSet is released under the AGPL-3.0 license, which is a strong copyleft license . For teams that modify the software and make it available as a network service, the license typically requires that the modified source code is made available to users. This legal framework contrasts with permissive licenses or proprietary SaaS models.

The system is packaged for Docker-based self-hosting . This means the entire pipeline—the browser-based web interactions, the LLM-backed agent reasoning, and the extraction logic—runs on your own infrastructure. For industries with strict compliance requirements, this deployment model avoids the data sovereignty risks of routing sensitive business intelligence queries through a third-party cloud API or platform. The data extraction process, therefore, remains within a controlled environment where you manage access, networking, and logs.

BigSet vs. Exa Websets: A Competitive Comparison

The most direct competitive comparison is with Exa Websets, a product from Exa.ai that also builds structured data collections from the web. While both products aim to turn the web into a queryable database, their technical approaches and market positioning differ .

The Hallucination Question

Hallucination is a critical concern for any system that autonomously builds datasets. Trendshift coverage of BigSet highlights a comparison where both BigSet and a competitor ran the same query and the competitor returned hallucinated data . The exact nature of that competitor is not explicitly named in the source, but the comparison is framed against a well-funded rival.

Exa addresses hallucination head-on through a documented three-step detection workflow that uses its own search capabilities: extracting claims from text, searching for evidence, and verifying claims against the retrieved evidence . Exa’s Websets product also runs agentic workflows on search candidates to verify that they match the user's exact query before adding them to the final set .

Sources covering BigSet’s launch describe a dedicated verification stage, where sub-agents verify findings against their sources before the table is finalized . The specific architecture of this verification agent—whether it is a single-step check or a more complex multi-pass system—is not fully detailed in the available materials. However, its purpose in the pipeline is clear: to prevent rows without source citations from reaching the exported dataset.

Key Differences at a Glance

Positioning and Implications

BigSet is not just a new tool; it represents a shift in how data pipelines can be built. For a small team or a busy analyst, compressing what has traditionally been a multi-hour process of writing and debugging scrapers into a 2- to 5-minute natural-language interaction is a significant time-saver .

The AGPL-3.0 license and self-hosted Docker deployment also position BigSet as an answer to growing concerns around data privacy and pipeline lock-in. By running the system locally, companies avoid routing proprietary business data through a third party and gain the freedom to modify the pipeline as their needs change.

TinyFish, which has raised $47 million and counts Google, DoorDash, and Amazon among its customers for its broader infrastructure platform, appears to be using BigSet to drive adoption for its core web agent technology while providing a genuinely useful, free tool to the community .