fenic is a semantic DataFrame library. A PySpark-style API for building AI and LLM pipelines over messy, unstructured data. Its local engine is Polars.
This post is about one specific problem we hit while building it, and the part of Polars that solved it: the expression plugin system. We ended up writing nine Rust plugins that extend Polars' expression engine. What follows is both why we went that route and how they're built, with the real code.
tl;dr. The operations an AI pipeline needs over text (chunking, prompt templating, jq, fuzzy matching, markdown and transcript parsing, richer type casts) aren't in Polars. Doing them as Python UDFs is slow, and it breaks composition. Writing them as Polars expression plugins in Rust, via pyo3-polars, turns them into native expressions. They run in-engine over Arrow, they keep their declared types, and they compose with built-in ops in a single expression tree. If you're weighing a UDF against a plugin, this is the case for the plugin.
Why we built this
What is fenic
fenic is a DataFrame library for building AI and LLM pipelines, with an API modeled on PySpark. If you come from Polars, the important thing to know is that Polars is the core execution engine for fenic.
Every DataFrame operation you write becomes a Polars expression, or a plan over pl.DataFrames. The semantic operators, the LLM map/extract/classify, the embeddings, the similarity joins, all sit on top of that same machinery. So in practice, what fenic can do is bounded by what we can express in Polars.
That's a deliberate bet. Polars gives us a fast, columnar, Arrow-native engine with a real expression language and a lazy optimizer. We had no interest in rebuilding any of it. We wanted to add to it.
What we wanted to do
The workloads fenic targets are pipelines over unstructured text. Documents, chat logs, transcripts, scraped JSON, markdown. Concretely, that means row-level operations Polars has no native equivalent for:
- Chunk a document into overlapping windows sized by token count, for embedding or retrieval.
- Render a prompt template per row. Real Jinja, with a struct of columns as the variables, ahead of an LLM call.
- Query JSON columns with jq, and parse markdown into a structured AST.
- Fuzzy-match strings (six edit-distance metrics) for dedup and joins.
- Parse transcripts (SRT/WebVTT) into typed, timestamped cue records.
- Cast values into fenic's richer logical types (embeddings, markdown, typed structs) that Polars' physical dtypes don't model directly.
Each of these has to run over a whole column, produce a typed result, and slot into the middle of a larger DataFrame pipeline. Not sit off to the side.
Where the obvious approaches fell short
The reflexive way to add a custom operation to Polars is a Python UDF. map_elements for per-row work, map_batches for whole-Series work. For genuinely opaque, IO-bound work, like an LLM API call, that's still the right tool, and fenic uses map_batches for exactly that. But for the text operations above, UDFs cost too much on three axes.
Speed. map_elements runs Python per row, under the GIL, with a Python-object round-trip for every value. For tokenizing or fuzzy-matching millions of rows, that's the bottleneck. Not the work itself.
Composition. This is the one that actually hurt. Polars is fast because it plans and executes an expression tree as a whole. The moment one step is an opaque Python callback, the engine can't see through it. It becomes an optimization barrier, forces a materialization, and breaks the single-pass pipeline. Chain three UDFs and you've got three trips out of the engine and back.
Types. A UDF's output type is something you assert loosely and hope holds. We needed operations that produce real, declared dtypes, like List<String>, Struct, fixed-size embedding arrays, so the rest of the plan can type-check against them.
The alternatives were worse. Forking Polars to add native kernels means owning a fork forever. Doing the text work outside the DataFrame, preprocess then load, throws away the laziness and composition that are the whole reason to use Polars in the first place. We wanted these operations to be first-class citizens of the expression engine. Not neighbors of it.
Why plugins
Polars has a purpose-built answer for exactly this: expression plugins. You write a kernel in Rust, register it, and it becomes a normal pl.Expr. To the engine, it's indistinguishable from a built-in. That checks every box the UDF route missed.
It runs in-engine, in Rust, over Arrow buffers. Vectorized, parallelizable, eligible for streaming, with no Python and no per-row object round-trip. It returns a pl.Expr, so it composes. Plugins chain with each other and with native ops in one expression tree the engine plans as a whole. It declares its output dtype, so the plan stays typed through the custom step. And Rust gives us a mature ecosystem for the hot loops (jaq, minijinja, rapidfuzz, tiktoken, a markdown parser) without reimplementing any of it.
The rule we settled on isn't "rewrite everything in Rust." Native Polars stays the fast path. A plugin only fills a gap Polars can't express: a tokenizer, a jq engine, a regex whose capture-group index is itself a column. Even the fuzzy matcher keeps just its six primitive kernels in Rust and composes the higher-order ratios from ordinary Polars expressions.
pyo3-polars generates the FFI, the Arrow marshaling, and the keyword-argument bridge, so the cost of writing one is low. We wrote nine.
What it bought us
The end state is worth stating before the mechanics. Every one of fenic's text operations is now a native Polars expression. They run inside the engine over shared Arrow memory, they keep their types, and here's the part that matters most. They compose with native Polars ops in a single expression, with zero Python round-trips.
Parsing a markdown document, filtering its AST with jq, indexing into the result, and casting it into a typed struct is one expression. Polars plans and executes it in a single pass, custom Rust and built-in list ops side by side. We'll come back to that exact example once the pieces are on the table.
The rest of this post is how it's built, from the Python registration down to the Arrow memory, using nothing but the real code.
How it's built: an implementation walkthrough
The mental model: two thin layers around a contract
A Polars expression plugin is two small pieces of code with a well-defined contract between them:
- Python side. Register a function so it looks like native Polars:
expr.my_namespace.my_op(...). - Rust side. A function that takes
&[Series], returnsPolarsResult<Series>, and declares its output dtype.
Everything between them is generated for you by pyo3-polars. Moving Series across the FFI boundary via Arrow, marshaling keyword arguments, wiring the symbol lookup. You never touch the FFI by hand.
Here's the entire Python surface for fenic's json.jq operator:
# src/fenic/_backends/local/polars_plugins/json.py
from pathlib import Path
import polars as pl
from polars.plugins import register_plugin_function
PLUGIN_PATH = Path(__file__).parents[3]
@pl.api.register_expr_namespace("json")
class Json:
"""Namespace for JSON-related operations on Polars expressions."""
def __init__(self, expr: pl.Expr) -> None:
self.expr = expr
def jq(self, query: str) -> pl.Expr:
return register_plugin_function(
plugin_path=PLUGIN_PATH,
function_name="jq_expr",
args=self.expr,
kwargs={"query": query},
is_elementwise=True,
)Two Polars APIs are doing the work:
@pl.api.register_expr_namespace("json")bolts a.jsonaccessor onto everypl.Exprin the process. After import,pl.col("payload").json.jq(".name")is a legal expression anywhere Polars expressions are legal.register_plugin_function(...)returns a normalpl.Exprthat, when the engine evaluates it, dlopens the compiled library atplugin_path, looks up the symbol namedfunction_name, hands it the inputSeries, and reads back the result.
That's the whole idea. .json is not special-cased anywhere in Polars. It's a user-registered namespace, and fenic registers nine of them (json, jinja, markdown, chunking, tokenization, fuzz, regexp, dtypes, transcript). The plugin looks exactly like a built-in because, to the expression engine, there's no meaningful difference.
(New to plugins? The Polars plugin docs and the pyo3-polars repo are the canonical references.)
Walking through one plugin, end to end
The Python jq method above points at a Rust symbol called jq_expr. Here it is, in full, on the other side of the boundary:
// rust/src/json/mod.rs
use polars::prelude::*;
use polars_arrow::array::ValueSize;
use pyo3_polars::derive::polars_expr;
use serde::Deserialize;
use serde_json::Value;
#[derive(Deserialize)]
struct JqKwargs {
query: String,
}
fn jq_output(_: &[Field]) -> PolarsResult<Field> {
Ok(Field::new(
"jq".into(),
DataType::List(Box::new(DataType::String)),
))
}
#[polars_expr(output_type_func=jq_output)]
fn jq_expr(inputs: &[Series], kwargs: JqKwargs) -> PolarsResult<Series> {
// Compile the jq filter ONCE, reuse it for every row.
let filter = jq::build_jq_query(&kwargs.query)
.map_err(|e| PolarsError::ComputeError(e.to_string().into()))?;
let jq_inputs = RcIter::new(core::iter::empty());
let ca = inputs[0].str()?;
let mut builder =
ListStringChunkedBuilder::new("jq".into(), ca.len(), ca.get_values_size() * 5);
for opt_str in ca.into_iter() {
if let Some(s) = opt_str {
match serde_json::from_str::<Value>(s) {
Ok(val) => {
let v: Val = val.into();
let results = filter
.run((Ctx::new([], &jq_inputs), v))
.collect::<Result<Vec<_>, _>>();
match results {
// an empty jq result set -> null
Ok(values) if values.is_empty() => builder.append_null(),
Ok(values) => {
let strs: Vec<String> = values.iter().map(|v| v.to_string()).collect();
builder.append_values_iter(strs.iter().map(|s| s.as_str()));
}
// a failed query aborts the whole batch — it does NOT null the row
Err(e) => return Err(PolarsError::ComputeError(
format!("jq query execution failed: {e}. Query: '{}'", kwargs.query).into(),
)),
}
}
// malformed JSON is unreachable: the column is typed JsonType, so
// upstream validation guarantees every non-null row is valid JSON.
Err(e) => unreachable!("Invalid JSON: {s} ({e})"),
}
} else {
builder.append_null();
}
}
Ok(builder.finish().into_series())
}Everything you need to understand the contract is in that snippet:
- Signature. The
#[polars_expr(...)]attribute macro (frompyo3_polars::derive) turns an ordinary Rust function into an exported, C-ABI symbol that Polars can dlopen and call. Your function just receivesinputs: &[Series]and returnsPolarsResult<Series>. Nounsafe, no manual pointer juggling. The macro generates the Arrow marshaling. - Typed inputs.
inputs[0].str()?downcasts the firstSeriesto aStringChunked. If the column isn't a string, you get a cleanPolarsErrorinstead of a segfault. - Do work once, not per row. The jq filter is compiled a single time (
build_jq_query) and reused across every row. This is a recurring pattern. Hoist compilation, allocation sizing, and setup out of the row loop. - Build the output with a builder.
ListStringChunkedBuilder, pre-sized with a capacity estimate, produces the output column. The null semantics are deliberate.append_null()fires for a null input row or an empty jq result. A failed jq query aborts the batch with aComputeError. And malformed JSON hits anunreachable!(), because the input column is typedJsonTypeand upstream validation guarantees every non-null row is valid JSON. (That guarantee is a nice side effect of building on a typed engine. The kernel gets to assume its inputs.) - Return a
Series.builder.finish().into_series().
The jaq crate (a pure-Rust jq) does the actual filtering. Polars provides the columnar plumbing. The plugin is the seam between them.
Output types, and why they matter
Look again at the attribute: #[polars_expr(output_type_func=jq_output)]. That jq_output function returns DataType::List(String) and never touches the data. This is the single most important thing to get right about plugins, and the easiest to get wrong.
Polars resolves the schema of an expression before it executes it. A plugin is opaque Rust. The engine can't infer what comes out by looking at your loop. So you must declare the output dtype up front, and the declaration has to run without the data. fenic uses all three declaration styles the macro supports, picked by how much the output type depends on.
1. Static. The type never changes. All six fuzzy-match kernels always return a Float64:
// rust/src/fuzz/mod.rs
#[polars_expr(output_type=Float64)]
fn normalized_indel_similarity(inputs: &[Series]) -> PolarsResult<Series> { /* ... */ }tokenization.count_tokens is likewise a fixed #[polars_expr(output_type=UInt32)].
2. From input fields. The type is a function of the inputs. jq and text chunking both always produce List<String>, declared by a small function that receives the input Fields (and here ignores them).
3. From the arguments. The type depends on kwargs. This is where it gets interesting. fenic's dtypes.cast kernel implements casting into fenic's own logical type system, so its output dtype is literally an argument, passed as a serialized JSON type descriptor. The output-type function deserializes that JSON to compute the Polars dtype at plan time:
// rust/src/dtypes/mod.rs
#[derive(Deserialize, Debug)]
pub struct CastKwargs {
source_dtype: String, // JSON-encoded fenic type
dest_dtype: String, // JSON-encoded fenic type
}
// Runs during schema inference only — never sees the data.
fn fenic_dtype_str_to_polars_dtype(
_input_fields: &[Field],
kwargs: CastKwargs,
) -> PolarsResult<Field> {
let fenic_type = serde_json::from_str::<FenicDType>(&kwargs.dest_dtype)
.expect("Invalid Fenic type string");
Ok(Field::new("casted".into(), fenic_type.canonical_polars_type()))
}
#[polars_expr(output_type_func_with_kwargs=fenic_dtype_str_to_polars_dtype)]
fn cast_expr(inputs: &[Series], kwargs: CastKwargs) -> PolarsResult<Series> {
let src_type = serde_json::from_str::<FenicDType>(&kwargs.source_dtype)
.expect("Invalid Fenic type string");
let dest_type = serde_json::from_str::<FenicDType>(&kwargs.dest_dtype)
.expect("Invalid Fenic type string");
cast_series_to_fenic_dtype(&inputs[0], &src_type, &dest_type)
}The lesson: output_type_func_with_kwargs lets a plugin's result type be computed from its configuration, decoded at schema-resolution time. That's what makes it possible to bolt an entirely foreign type system (fenic's logical types, including things like fixed-size embedding arrays) onto Polars' physical dtypes while keeping lazy schema resolution intact. Your plugin declares, up front and data-free, exactly what shape it will produce. (One honest caveat, since we're holding this up as exemplary. The real code .expect()s on a malformed type descriptor, so a bad JSON string panics the schema-resolution path rather than surfacing a clean PolarsError. Hardening it would map that to a ComputeError.)
Get this wrong, declare List<String> and then build a Float64, and you don't get a compile error. You get a runtime failure when the produced column doesn't match the promised schema. The declaration is a contract you're on the hook for.
Getting parallelism and broadcasting right
Every fenic plugin passes is_elementwise=True. That flag is a promise to the engine: this function treats each row independently. Row i of the output depends only on row i of the input. No ordering, no windows, no cross-row state.
That promise is worth a lot. It makes the function eligible to run in Polars' streaming engine, and it lets the optimizer reason about it: predicate, projection, and slice pushdown, plus a group_by fast path where the function runs over the flattened series instead of once per group. It is also load-bearing. If you set is_elementwise=True on a function that actually looks across rows, Polars may hand it a partial batch (a streaming morsel), or run it over flattened groups inside a group_by. Either of which silently breaks cross-row logic. The flag isn't a hint. It's an invariant you're asserting.
"Elementwise" still has to cope with broadcasting. The common case is where one argument is a scalar literal (length 1) and another is a full column. fenic handles this two ways depending on the kernel.
For the fuzzy matchers, which take two string columns, it leans on a Polars helper that does broadcasting and null handling for you:
// rust/src/fuzz/mod.rs
#[polars_expr(output_type=Float64)]
fn normalized_indel_similarity(inputs: &[Series]) -> PolarsResult<Series> {
// Use broadcast_binary_elementwise for string operations since it handles nulls safely.
// binary_elementwise_values could process garbage data at null positions ...
let left = inputs[0].str()?;
let right = inputs[1].str()?;
let similarity: Float64Chunked = arity::broadcast_binary_elementwise(
left,
right,
|left: Option<&str>, right: Option<&str>| match (left, right) {
(Some(left), Some(right)) => {
Some(indel::normalized_similarity(left.chars(), right.chars()) * 100.0)
}
_ => None,
},
);
Ok(similarity.into_series())
}Two things worth stealing here. First, broadcast_binary_elementwise from polars::chunked_array::ops::arity handles length-1 broadcasting and threads Option<&str> through so nulls are explicit. The closure decides what a null means (here, null in, null out). Second, note the code comment. We deliberately avoided the *_values variant that skips the null check, because for UTF-8 string work you do not want the kernel touching the bytes at a null slot. That's a decision you only make after it has already bitten you once.
The regex kernel takes the manual route because it needs a per-call compile cache anyway:
// rust/src/regex/mod.rs
#[polars_expr(output_type=Int32)]
fn regexp_instr(inputs: &[Series]) -> PolarsResult<Series> {
let text_series = inputs[0].str()?;
let pattern_series = inputs[1].str()?;
let idx_series = inputs[2].i64()?;
let len = text_series.len();
let mut regex_cache: HashMap<String, Regex> = HashMap::new();
// A length-1 series is a literal that should be broadcast to every row.
let pattern_is_literal = pattern_series.len() == 1;
let idx_is_literal = idx_series.len() == 1;
// ... per-row loop reads index 0 for literals, i otherwise ...
}The broadcasting is a one-liner (if pattern_is_literal { 0 } else { i }). The HashMap<String, Regex> compile cache means a literal pattern is compiled once, and every subsequent row is a cache hit, via the Entry API:
fn get_or_compile_regex<'a>(
regex_cache: &'a mut HashMap<String, Regex>,
pattern: &str,
) -> PolarsResult<&'a Regex> {
match regex_cache.entry(pattern.to_string()) {
Entry::Occupied(entry) => Ok(entry.into_mut()),
Entry::Vacant(entry) => {
let regex = Regex::new(pattern).map_err(/* ComputeError */)?;
Ok(entry.insert(regex))
}
}
}Because this cache lives for the duration of one call on one chunk, even a per-row-varying pattern column only ever compiles each distinct pattern once.
The Arrow boundary, and zero copy
Two of fenic's plugins, the Jinja renderer and the markdown/JSON path, need to turn arbitrary Arrow values (strings, ints, structs, lists, nested combinations) into a foreign representation. A minijinja::Value for template rendering, or a serde_json::Value for JSON emission. This is the point where a naive plugin reaches for Series iterators, or worse, materializes Python objects. fenic reads the Arrow arrays directly. It's a deliberate rewrite of an earlier AnyValue-based path that required unnecessary allocations. Reading the Arrow buffer is zero-copy. The only allocation is the one output value the converter actually has to build.
The whole thing hangs off one trait and one generic function. The trait abstracts "how do I build a value of type V from an Arrow scalar":
// rust/src/arrow_scalar_extractor.rs
trait ArrowToValue: Sized {
fn from_null() -> Self;
fn from_str(s: &str) -> Self;
fn from_bool(b: bool) -> Self;
fn from_i64(i: i64) -> Self;
fn from_f64(f: f64) -> Self;
// ... every primitive width ...
fn from_struct(fields: Vec<(String, Self)>) -> Self;
fn from_list(values: Vec<Self>) -> Self;
}It's implemented once for minijinja::Value and once for serde_json::Value. Then a single generic converter walks any Arrow array and produces either. The two public methods just pin the concrete type:
pub struct ArrowScalarConverter;
impl ArrowScalarConverter {
pub fn to_jinja(&self, array: &dyn Array, row_idx: usize) -> PolarsResult<JinjaValue> {
self.convert(array, row_idx)
}
pub fn to_json(&self, array: &dyn Array, row_idx: usize) -> PolarsResult<JsonValue> {
self.convert(array, row_idx)
}
fn convert<V: ArrowToValue>(&self, array: &dyn Array, row_idx: usize) -> PolarsResult<V> {
if array.is_null(row_idx) {
return Ok(V::from_null());
}
match array.dtype() {
ArrowDataType::Utf8 => {
let a = downcast_array!(array, Utf8Array<i32>);
Ok(V::from_str(a.value(row_idx)))
}
ArrowDataType::LargeUtf8 => {
let a = downcast_array!(array, Utf8Array<i64>);
Ok(V::from_str(a.value(row_idx)))
}
ArrowDataType::Utf8View => {
let a = downcast_array!(array, Utf8ViewArray);
Ok(V::from_str(a.value(row_idx)))
}
// Binary, BinaryView, Boolean, Int8..Int64, UInt8..UInt64, Float32/64 ...
ArrowDataType::Struct(_) => self.convert_struct(downcast_array!(array, StructArray), row_idx),
ArrowDataType::List(_) => self.convert_list(downcast_array!(array, ListArray<i32>), row_idx),
ArrowDataType::LargeList(_) => self.convert_large_list(/* ListArray<i64> */),
ArrowDataType::FixedSizeList(_, _) => self.convert_fixed_size_list(/* ... */),
_ => Err(PolarsError::ComputeError(/* unsupported */)),
}
}
}A few things worth pulling out:
- It works on
&dyn Arrayfrompolars_arrow, not onSeries. It downcasts to the concrete Arrow array (Utf8ViewArray,StructArray,ListArray<i32>, …) and reads the value atrow_idxin place. No per-value boxing into a PolarsAnyValue, no Python objects. Utf8Viewis handled explicitly, alongsideUtf8andLargeUtf8. This is the detail that bites people. Modern Polars represents strings asUtf8Viewby default. But data zero-copy-imported from external Arrow (pyarrow, some IPC/Parquet paths) can still arrive asUtf8orLargeUtf8, which is exactly why the converter matches all three. A plugin that handles only one layout will fall through and error on perfectly ordinary string columns. If you read Arrow arrays directly, you have to know every physical layout Polars might hand you.- Nesting is recursion.
convert_structcallsconverton each field.convert_listcallsconverton each element. AList<Struct<...>>just falls out of the recursion. One function handles arbitrarily nested Arrow. - One implementation, two outputs. Adding a third target (say, a MessagePack value) is one more
impl ArrowToValue. The traversal never changes.
This is the difference between a plugin that works on strings and one that treats the Arrow memory as a first-class input.
Beyond scalars: returning List<Struct>
Plugins aren't limited to scalar columns. fenic's transcript parser takes raw SRT/WebVTT text per row and emits a list of typed cue records. That's a List<Struct> column with a fixed unified schema, regardless of input format. Its output-type function declares the shape of a single cue as a Struct (and validates the format name while it's there):
// rust/src/transcript/mod.rs
fn convert_format_to_struct(_: &[Field], kwargs: TranscriptFormatKwargs) -> PolarsResult<Field> {
// ... validate kwargs.format against the parser registry ...
Ok(Field::new(
"output".into(),
DataType::Struct(vec![
Field::new("index".into(), DataType::Int64),
Field::new("speaker".into(), DataType::String),
Field::new("start_time".into(), DataType::Float64),
Field::new("end_time".into(), DataType::Float64),
Field::new("duration".into(), DataType::Float64),
Field::new("content".into(), DataType::String),
Field::new("format".into(), DataType::String),
]),
))
}And the rows are assembled with AnyValue, building each struct explicitly and collecting them into a list:
let values = vec![
entry.index.map_or(AnyValue::Null, AnyValue::Int64),
entry.speaker.as_ref().map_or(AnyValue::Null, |s| AnyValue::StringOwned(s.clone().into())),
AnyValue::Float64(entry.start_time),
// end_time, duration, content, format ...
];
struct_values.push(AnyValue::StructOwned(Box::new((values, fields))));
// ...
Ok(Some(AnyValue::List(Series::new("".into(), struct_values))))The kernel builds each cue as an AnyValue::StructOwned, its fields lining up with the cue schema from the output-type function, and collects a row's cues into an AnyValue::List. So the column comes out as List<Struct>. A parse failure on one row becomes a null for that row, rather than a hard error for the whole batch. Structured output plus honest null semantics is what lets a downstream .unnest() or .explode() treat the result like any other native nested column.
Configuration across the boundary
The bridge for keyword arguments is just serde. On the Python side you build a plain dict (marshaling enums to their string values):
# src/fenic/_backends/local/polars_plugins/chunking.py
chunk_kwargs = {
"desired_chunk_size": desired_chunk_size,
"chunk_overlap": chunk_overlap,
"chunk_length_function_name": chunk_length_function_name.value, # enum -> str
"chunking_character_set_name": chunking_character_set_name.value,
"chunking_character_set_custom_characters": chunking_character_set_custom_characters,
}
return register_plugin_function(
plugin_path=PLUGIN_PATH, function_name="text_chunk_expr",
args=expr, kwargs=chunk_kwargs, is_elementwise=True,
)On the Rust side you declare a #[derive(Deserialize)] struct with matching field names, and it arrives populated:
// rust/src/chunking/mod.rs
#[derive(Deserialize)]
pub struct TextChunkKwargs {
desired_chunk_size: usize,
chunk_overlap: usize,
chunk_length_function_name: String,
chunking_character_set_name: String,
chunking_character_set_custom_characters: Option<Vec<String>>,
}Polars serializes the kwargs dict, and the derive macro deserializes it into your struct via serde before your function runs. The only discipline it demands is that the two sides agree on names and shapes. There's no schema enforcing it, so the Python dict and the Rust struct are a contract you keep by hand. Option<T> fields map cleanly to Python None.
Building and shipping
The whole thing is one Rust crate compiled to a single dynamic library, built by maturin. The relevant Cargo.toml:
[lib]
name = "polars_plugins"
crate-type = ["cdylib", "rlib"]
[dependencies]
polars = { version = "0.51.0", features = ["dtype-struct", "dtype-array", "timezones"] }
polars-arrow = "0.51.0"
pyo3 = { version = "0.25", features = ["abi3-py39"] }
pyo3-polars = { version = "0.24.0", features = ["derive", "dtype-struct"] }
serde = { version = "1.0.219", features = ["derive"] }
# jaq-core / jaq-json, minijinja, rapidfuzz, regex, tiktoken-rs, markdown ...cdylib produces the loadable library. rlib keeps it usable from Rust integration tests and benchmarks. pyo3-polars with the derive feature is what gives you the #[polars_expr] macro. Note the version coupling. pyo3-polars pins exact polars, polars-arrow, and pyo3 versions, and your plugin links against Arrow types that must be ABI-compatible with the Polars that loads it. So the whole stack has to move as one.
On the Python packaging side, maturin is told to place the built module inside the fenic package:
# pyproject.toml
[tool.maturin]
python-source = "src"
module-name = "fenic._polars_plugins"So the compiled artifact lands at fenic/_polars_plugins.abi3.so. That's why every plugin file computes PLUGIN_PATH = Path(__file__).parents[3]. From .../fenic/_backends/local/polars_plugins/json.py, three parents up is .../fenic, the package directory that holds the .so. register_plugin_function(plugin_path=...) points there, and Polars finds the library by name.
And because that Rust stack pins in lockstep, the Python dependency is pinned tight too: polars>=1.34.0,<1.36.0. This is the plugin author's tax. Bump polars on its own and you get two incompatible copies of the polars crates in the dependency graph, and a build that fails with type mismatches in every #[polars_expr] function. The crates move together, deliberately. A jump to a new Polars can even mean real code changes, not just a lockfile edit.
The benefits for fenic
Earlier I said the payoff was composition. Now that the pieces are on the table, here it is.
Everything composes. Because every plugin returns a plain pl.Expr, plugins compose with each other and with native Polars operations in a single expression. This is fenic's implementation of "extract code blocks from a markdown document." Four operations, three of them custom Rust plugins, one of them a built-in:
# src/fenic/_backends/local/transpiler/expr_converter.py
return (
self._convert_expr(logical.expr)
.markdown.to_json() # Rust plugin: markdown -> JSON AST
.json.jq(logical.jq_query) # Rust plugin: jq filter over the AST
.list.get(0) # NATIVE Polars list op, interleaved
.dtypes.cast(source_dtype, dest_dtype) # Rust plugin: cast into a struct type
)That .list.get(0) sitting between .json.jq(...) and .dtypes.cast(...) is the whole point. To the engine there is no boundary between the plugins and the native op. It's one expression tree, planned once, executed in one pass over the data, entirely inside Polars. No intermediate Python lists, no per-step Series-to-Python-to-Series marshaling, no leaving the engine and coming back.
It runs the other direction too, with native expressions feeding a plugin. fenic builds LLM prompts by packing columns into a native pl.struct and handing it to the Jinja plugin:
struct_expr = pl.struct(column_exprs) # native
return struct_expr.jinja.render(template=..., strict=...) # plugin renders per rowThe plugin receives a struct column, and the ArrowScalarConverter from earlier is what lets the Rust side read that struct's fields as Jinja variables. Native op produces the struct, plugin consumes it, one expression. Once your custom kernels are pl.Expr, the entire Polars expression language, every native string, list, struct, temporal, and arithmetic op, becomes glue you can put between them.
One build, two surfaces. A quieter benefit of writing the logic in Rust. You can expose slices of it as plain Python functions too, not only as expressions. fenic validates jq filters and regex patterns at plan time, when a query is being built, before any data flows, using the exact same Rust that executes them:
// rust/src/lib.rs
#[pymodule]
fn _polars_plugins(_py: Python<'_>, m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add_function(wrap_pyfunction!(json::py_validate_jq_query, m)?)?;
m.add_function(wrap_pyfunction!(regex::py_validate_regex, m)?)?;
Ok(())
}py_validate_jq_query calls the same build_jq_query the jq_expr kernel uses, and raises a Python ValueError on a bad filter. So an invalid query is caught the moment the user writes it, with the identical parser that will run it later, rather than blowing up mid-execution. This pattern started with regex. Python's re was accepting patterns that then failed under Polars' Rust regex engine at runtime, and validating with the same engine ended the divergence. The Rust crate is both a plugin library and an ordinary pyo3 extension module. You get both surfaces from one build.
Add it up, and the plugin system is what let fenic put an entire AI-oriented operation set (tokenizing, templating, parsing, matching, casting) inside Polars rather than beside it. The semantic layer sits on top. The fast, typed, composable substrate underneath is Polars, extended rather than wrapped.
Lessons for plugin authors
Distilled from building the nine:
- The output-type declaration is a hard contract, unchecked at compile time. Declare exactly what you build.
output_type_func_with_kwargsis the escape hatch when the type depends on configuration, enough to graft a whole foreign type system on, asdtypes.castdoes. - Match every physical Arrow layout, not just the logical dtype.
Utf8Viewas well asUtf8andLargeUtf8, or you'll error on perfectly ordinary string columns. And keep the Python kwargs dict and the RustDeserializestruct in lockstep. serde is the whole bridge, and nothing but you enforces it. is_elementwise=Trueis an invariant you assert, not a hint. And one you can break silently. While you're at it, hoist setup (compile, allocate, size builders) out of the row loop.- Split the macro shim from the logic. Keep
#[polars_expr]functions tiny and delegate to plain functions (e.g. Jinja'srender(...)) that are unit-testable withcargo testand no Python runtime. - Export the same Rust as plain pyfunctions for plan-time validation, so the code that checks a query is the code that runs it.
When to reach for a plugin, and where to look next
A plugin is the right tool when you need row-level work that Polars doesn't ship natively, you want it to run inside the engine (parallel, streamable, composable), and you can express the output as a declared dtype. That covers a lot. Parsing, tokenizing, templating, fuzzy matching, format conversion, custom casts. When the work is genuinely not elementwise, or it's an opaque IO-bound call (hitting a model API, say), the honest answer is still map_batches or map_elements, the Python escape hatches, and keeping the schema explicit there instead.
But for the case where Polars can't do something natively and you want it in-engine and composable, nine plugins deep, the system holds up. It doesn't feel like bolting something onto Polars. It feels like extending it.
If you want to see all nine in context, fenic is open source. The Rust kernels live in rust/src/ and the Python namespaces in src/fenic/_backends/local/polars_plugins/. If you're writing your own, the Polars plugin docs and pyo3-polars are the place to start. And we'd genuinely like to hear how it goes.
And if this post was worth your time, consider giving fenic a star on GitHub. It's the simplest way to help other people building on Polars find it, and it tells us these deep dives are worth writing.