Parquet as the Interchange Format for Agent-Consumed Analytics

Parquet and DuckDB let agents query governed analytics without touching production databases.

· 8 min read
Cover illustration for “Parquet as the Interchange Format for Agent-Consumed Analytics”
MCP and Headless Analytics · October 26, 2026 · 8 min read · 1,876 words

For API-first and agent-first teams, a headless analytics backend should serve governed datasets without requiring an opinionated frontend or access to production databases. Parquet provides the interchange contract, DuckDB supplies the query runtime, and MCP exposes the data to agents. This architecture makes analytics readable, inexpensive to query repeatedly, and separate from the live systems that produced it.

What Parquet actually is and why its design choices matter for analytics

Parquet organizes data in three nested layers. Row groups are horizontal slices of a dataset, column chunks hold one column within each row group, and pages are the units where encoding and compression are applied. This structure makes selective reading possible. A query touching two columns can read their column chunks without reading the rest of the table. Within those chunks, pages hold values encoded for their characteristics and then compressed with a codec such as Zstd or Snappy.

The footer tells a reader where to look. It contains the schema, the locations of row groups and column chunks, and per-column-chunk statistics including minimum values, maximum values, and null counts. These details are available before column data is decompressed. Column pruning uses the layout to avoid unneeded columns, while predicate pushdown uses statistics to skip row groups whose ranges cannot contain a match. An unfiltered COUNT(*) can return from row counts in the metadata without reading column data.

Dictionary encoding represents repeated values compactly, run-length encoding handles consecutive repeats, delta encoding suits sorted data, and bit packing suits small integers. Compression follows that encoding step. Consequently, country, device, and event type columns can shrink substantially through dictionary encoding and run-length encoding, while high-entropy floating-point values compress far less.

Why the format's self-describing structure is what agents need from a data contract

An agent has no ambient understanding of a dataset. Before constructing a query, it needs to discover which columns exist and how their values are represented. Parquet’s footer provides column names, physical types, and logical types without requiring a lookup against a live database. The structure travels with the data it describes. That makes it a more dependable structural contract than separate documentation that may no longer match the payload.

This changes the requirements for analytical access. Querying Postgres requires a connection, credentials, and knowledge of the database schema, while inspecting a Parquet footer requires file access and a reader library. A pre-calculated, governed copy thus lets the agent find the data it needs without having production credentials. DuckDB makes this discovery queryable through metadata functions that expose schema details, row counts, and column statistics as rows.

However, structural discovery is not a substitute for metric definitions. A footer can describe a column’s type, but the governed dataset must still encode the agreed meaning of the metric that column represents. The distinction is important for agent accuracy. Self-description removes uncertainty about the payload’s structure, while pre-modeling removes the need for each agent to reconstruct business logic.

How columnar layout and Zstd compression make agent queries cheap to run repeatedly

Agent analytics commonly involves repeated, narrow, read-only requests. Daily active users, revenue by segment, and churn rate may each need only a small part of a larger metrics dataset. Column pruning skips columns that those reads do not need. As a result, adding columns to the underlying dataset need not increase the column data scanned by an unchanged query. That cost profile is well suited to agents answering questions for many users or workflows.

MotherDuck’s Parquet benchmark used an 11.2-million-row NYC TLC yellow taxi dataset in DuckDB v1.5.5. Its Zstd Parquet file was smaller than the same data as CSV, and query speed ranged from significantly to dramatically faster depending on how many columns the query touched. The variation is central to interpreting the result. It supports the case for selective analytical reads. SQL is also the query interface DuckDB uses to read Parquet.

For a request covering the last seven days, timestamp statistics allow the reader to skip groups whose ranges fall entirely outside that window. Only potentially relevant groups need further reading. Unlike a query against production Postgres, that scan consumes no production connection and introduces no analytical contention with production writes. Repeated queries still require execution, but their work remains separate from the transactional system and limited to the relevant parts of the file.

Why DuckDB is the query engine that closes the loop between Parquet files and agents

A storage contract needs an executable query path. DuckDB supplies that path by reading Parquet directly, without a separate server process, schema migration, or persistent database connection. An agent runtime can embed it, point it at a local path or object storage, and issue SQL in-process. The same reader supports both column queries and metadata inspection. This gives the agent a practical route from discovering a dataset to calculating a result.

Parquet format v2 added encodings aimed at numeric data, and DuckDB reads both v1 and v2 without requiring the user to choose a version. Compatibility does not become an agent decision. An MCP server can therefore expose the governed dataset as a Resource while DuckDB supplies the query runtime, with neither step requiring access to the production Postgres instance.

Parquet is the at-rest format, while Arrow is the in-flight, in-memory format. Both are open and columnar, which makes their interchange low-overhead. That conversion is not zero-copy, however. Reading Parquet into Arrow requires decoding and decompressing encoded columns into Arrow’s memory layout, so the runtime still performs work between the stored file and the agent’s usable results.

Why agents must never query production databases directly, and what Parquet changes about that

Analytical requests that agents make on their own can use up connection pools, and paths that bypass row-level security can reveal records the agent is not allowed to see. A successful login is not sufficient protection. Each action must be checked against policy based on what the request contains, what actions came before it, and what is happening in the business at the time. Periodic audits cannot substitute for enforcement during rapid, autonomous activity.

The analytical layer is where policy should be enforced. Attribute-based access control can restrict records through row-level security and redact sensitive fields through column-level masking. A fraud detection agent, for example, can get transaction amounts and timestamps without getting payment card numbers or customer names. A governed Parquet payload contains only the authorized rows and columns. This removes the gap in which an agent’s query could reach unauthorized source data, while access to the payload must still reflect the policy governing each action.

The U.S. National Security Agency noted in a May 2026 report that MCP’s rapid proliferation had outpaced development of its security model. That concern strengthens the case for reducing what an agent-facing interface can reach. A pre-calculated, read-only dataset keeps that interface from running queries on live production data. It also makes the governed copy, rather than an audit log after the fact, the boundary for data exposure. That copy must be exposed without reopening access to its source.

How MCP turns a governed Parquet dataset into a queryable resource for any agent

MCP’s Resource primitive fits a pre-calculated, read-only analytical payload. A server can expose a governed Parquet dataset and its schema as context, rather than provide an unrestricted route to production queries. The agent receives data with a defined boundary. MCP’s other primitives complement that Resource, with Tools representing actions such as running a query and Prompts providing reusable analytical workflow templates. DuckDB can execute the analytical query against the exposed file.

MCP servers are connectors that depend on prior work to fix data quality, normalize schemas, and transform raw API responses into usable metrics. The semantic layer must encode metric definitions into the governed dataset so that agents consume agreed calculations. Without that preparation, MCP offers convenient access to an ungoverned source.

MCP’s July 28, 2026 specification revision made the protocol core stateless, enabling horizontal scaling behind a plain load balancer. That supports shared dataset services under concurrent agent traffic. Platform adoption extends the potential reach of this interface. Informatica announced native MCP support across its Intelligent Data Management Cloud at Informatica World 2026, with servers live or in preview across AWS, Microsoft Foundry, Google Cloud, Databricks Agent Bricks, Snowflake Cortex AI, and Salesforce Agentforce. Salesforce announced more than 60 new MCP tools at TrailblazerDX on April 15, 2026, but connector availability does not remove the need for governed analytical inputs.

Where Parquet's current defaults fall short and what practitioners need to know before shipping

Parquet needs more than its default settings to work well. Dictionary encoding loses its advantage when almost every string is distinct, as can happen with URLs, request IDs, user agents, and long text attributes. Floating-point columns also compress less effectively than low-cardinality strings or integers. The 2026 additions include ALP encoding and improved floating-point handling, but reader support is not uniform. Compression expectations therefore need to reflect actual column distributions and the engines that will consume them.

Row-group size determines how precisely statistics can exclude data. Groups that are too large make predicate pushdown less selective, while groups that are too small increase footer overhead relative to the data. Neither extreme is a universal default. Teams should benchmark writer settings on a sample that reflects their real data before they publish a governed dataset. Synthetic data may compress well even when its distributions differ from those that production queries will read.

Schema changes need just as much care before release. Additive changes, including new columns and widened types, are usually safe, while removed or renamed columns and narrowed types can cause quiet downstream failures. Agent consumers need a stable, versioned contract. The 2026 format additions also include a native variant type and first-class geospatial types, with embedding and File logical types proposed to address needs beyond Parquet’s 2013 assumptions. Teams should verify reader compatibility before depending on new logical types, because support arrives unevenly across engines.

How can human analysts and AI agents share the same governed Parquet datasets?

A dashboard, a Slack report, and an AI agent should answer the same question from identical metric definitions. Pre-modeled, pre-calculated Parquet datasets make that possible without each consumer having to write its own SQL for the business logic. Metrics are calculated once and shared through DuckDB or MCP, while each consumer can show them in a different way. The frontend no longer determines where the metric is defined.

This separation gives API-first teams control over delivery. The same governed dataset can support an inbox report, a dashboard, or usage analytics inside an AI agent without making a prebuilt dashboard UI the center of the architecture. When both consume the same file, the dashboard’s calculation cannot diverge from the agent’s merely because each reconstructed the metric differently. Governance and schema belong to the shared data layer.

For Supabase teams, Postgres remains the transactional home. Running analytical workloads on the primary damages transactional performance, so the recommended pattern is to model metrics into governed Parquet datasets and query them through DuckDB. Keeping analytical and agent traffic away from production Postgres is the practical criterion for a headless analytics backend. Its analytical contract must let humans and agents trust the same meaning without leaving either of them to protect the live database.

More in MCP and Headless Analytics