What would a data processing framework look like if we built it today?
With modern workloads, unstructured data, and LLM inference as first-class compute.
That experiment became fenic.
Here's the bar we set for ourselves. A modern data processing engine should:
1) Feel familiar to engineers
People already invested years learning SQL and DataFrames. Asking them to throw that away is not pragmatic.
2) Be fast
Polars, DuckDB, and modern warehouses raised expectations. "It works" isn't enough; it has to be pleasantly fast.
3) Be easy to run, ideally local-first
If the first step is "run 10 containers," you've lost most developers. DuckDB proved local OLAP can be a default experience.
4) Treat inference as first-class compute
Not a bolt-on UDF. Inference changes fault tolerance, execution strategy, caching, and optimization.
5) Treat unstructured data as first-class data
Text is not a niche anymore. Markdown isn't "just formatting" anymore—it's semi-structured data. PDFs, transcripts, docs: these are core inputs now.
6) Be usable by a new user: LLMs and agents
This was the most surprising requirement: you're not just building for humans. You're building for models that will write code, call tools, and compose workflows.
If you can meet these requirements, a new engine is worth building.
Here's how we approached them in fenic.
The fenic approach
Familiar API: start where engineers (and models) already are
We started with a familiar DataFrame API and a PySpark-style mental model.
Why PySpark?
- data engineers already know it,
- the DataFrame paradigm mixes declarative and imperative logic cleanly,
- and (importantly) LLMs are already good at it.
That last point matters more than people expect. If you're building a system that agents will use, you want the interface to be something models can reliably reason about.
Performance + local DX: build on proven engines
Rather than reinvent physical execution, fenic builds on successful modern engines—Polars and DuckDB.
A good mental model is:
fenic plans compile down into Polars and/or DuckDB plans. Polars and DuckDB implement the physical operators. fenic decides how to execute.
This gives you the best of both worlds:
- fast local execution,
- strong ergonomics,
- and the ability to build higher-level optimization and planning above them.
And none of this works without Apache Arrow as the common substrate. Arrow is what makes moving data between these components consistent, safe, and fast.
Inference without UDF opaqueness: semantic operators
We didn't want "LLM calls" to be a bag of ad-hoc UDFs.
We wanted inference to feel like any other compute: you describe what you want, and the system figures out how to get it.
So we introduced semantic operators (inspired by research like Lotus and similar systems coming out of Berkeley).
Why semantic operators matter:
- Declarative inference: You ask for the output; the system manages batching, retries, caching, throttling, etc.
- Composability: Semantic steps become first-class nodes in a plan—chainable like filters and joins.
- Optimization becomes possible: Once inference is visible to the planner, you can do real optimizations:
- reorder operations to reduce the number of LLM calls,
- push down filters,
- cache intermediates aggressively,
- and structure execution to improve throughput.
UDFs hide all of that.
Reliability: async execution + types end-to-end
Inference failures don't look like "a worker died."
They look like:
- rate limits,
- token limits,
- timeouts,
- partial streaming responses,
- model hiccups,
- and non-deterministic output variance.
In practice, inference behaves more like embedding REST calls into a pipeline than distributing CPU work across a cluster.
So fenic leans into:
- a robust async execution model,
- retries + backoff,
- caching,
- and type reasoning across the pipeline, even when parts are powered by LLMs.
Types become a reliability tool, not just a developer convenience.
Unstructured-first: new data types for modern inputs
If you take LLM-centric workflows seriously, you end up spending a lot of time manipulating text.
But "text" isn't one thing anymore. Markdown is semi-structured. Transcripts have speaker structure. Documents have sections and hierarchy. PDFs and HTML have layout and noise.
So fenic allows introducing new data types that make these operations safer and more ergonomic. A simple example is a Markdown type that understands structure instead of treating everything as a raw string.
Built for agents: MCP as a first-class interface
This is where fenic took a turn.
Building on PySpark helped models write "basic" fenic code quickly. But as we extended the API, we hit a ceiling: models can't use what they can't see.
So we built an MCP server as a documentation + context layer. The goal was simple:
- give any model a minimal set of tools to learn fenic on demand,
- grounded in real docs and code.
That alone made a huge difference in the quality of generated fenic code.
But then we realized something bigger:
A pipeline isn't just something you run. It's also something you can expose as a tool.
So fenic introduced the idea that any DataFrame, no matter how complex, can be registered in a catalog as a tool, similar to how you'd register a view or UDF in a database.
From there, fenic can expose those tools automatically through MCP via the CLI.
That means:
- you can define data + business logic declaratively,
- register it once,
- and make it callable by any LLM/agent as a tool interface.
This is the moment where "data processing framework" and "agentic system framework" collapse into the same thing.
Why fenic now calls itself "context engineering"
If someone visits the fenic repo today, they'll see it described as:
declarative context engineering for agents.
That might sound far away from the question we started with.
But it's the natural evolution of the same core idea.
Building agentic systems is an exercise in state management under constraints:
- what information to provide,
- when to provide it,
- how to represent it,
- how to keep it coherent,
- and how to do all of that efficiently.
That's a data problem.
And it's an optimization problem.
fenic is built to let you describe the shape of the result, whether you're a human engineer or an agent composing tools, and let the system handle the rest: planning, execution, reliability, and performance.
Get started
fenic is still an experiment.
And that's intentional. The "right" form factor for these systems is still emerging, and the most important input isn't a product roadmap.
It's creativity from the people who try to build real things with it.
fenic is open source. Try it, break it, tell us what's missing and let's build it together.