Implement a custom FileFormat

[ElSaico]

Hello,

I have a specific data transformation in mind: to skip the details, the input is a line-delimited JSON while the output is a list of partitioned files that follow an application's custom binary format.
With those specifications in mind, an Arrow dataset feels like an almost perfect fit... if I could implement my own FileFormat, that is. Would that even be possible?

[Mike-Animal-Counseling]

Arrow's C++ Dataset API supports this: implement a FileFormat backed by your own FileWriter and FileWriteOptions. The write side includes MakeWriter() and DefaultWriteOptions(); the existing formats are useful examples. Those interfaces are in file_base.h.

If you're using PyArrow, this is a native-binding job rather than a Python callback interface. Its FileFormat wrapper holds a C++ format object. A plain Python subclass doesn't provide the native writer needed by write_dataset; you'd also need binding support for your implementation.

For a Python-only pipeline, you can still use Arrow to stream the JSON input, then hand each batch to your own partition routing and binary writer:

import pyarrow.json as paj

with paj.open_json('input.jsonl') as reader:
    for batch in reader:
        route_and_write_batch(batch)  # Your application's function.

That function would own partition selection, file naming, and closing the custom writers. It doesn't plug the custom codec into Dataset's existing partitioned writer. If keeping that automatic Dataset machinery is the main requirement, the native FileFormat route is the one to investigate.

I checked the 25.0.0 Python bindings and ran the open_json batch-reading part locally. I haven't implemented your binary format or compiled a custom C++ writer.

AI-assisted research and writing.