Extending Polars with Rust Expression Plugins

In one line

The text operations an AI pipeline needs — chunking, prompt templating, jq, fuzzy matching, markdown and transcript parsing — aren’t in Polars, and Python UDFs are slow and break composition. Here’s why we wrote nine Rust expression plugins to extend Polars’ engine, and how they’re built, with the real code.

Opening excerpt

fenic is a semantic DataFrame library. A PySpark-style API for building AI and LLM pipelines over messy, unstructured data. Its local engine is Polars.

This post is about one specific problem we hit while building it, and the part of Polars that solved it: the expression plugin system. We ended up writing nine Rust plugins that extend Polars’ expression engine. What follows is both why we went that route and how they’re built, with the real code.

tl;dr. The operations an AI pipeline needs over text (chunking, prompt templating, jq, fuzzy matching, markdown and transcript parsing, richer type casts) aren’t in Polars. Doing them as Python UDFs is slow, and it breaks composition. Writing them as Polars expression plugins in Rust, via pyo3-polars , turns them into native expressions. They run in-engine over Arrow, they keep their declared types, and they compose with built-in ops in a single expression tree. If you’re weighing a UDF against a plugin, this is the case for the plugin.

(Excerpted from the original; full article via the source link below.)

This story hit the Hacker News front page today (43 points / 0 comments, via fenic.ai). Our Tech Radar aggregates daily signals on AI engineering, backend architecture and DevOps — browse the related services and further reading below, or get in touch with our team.