Read pfeed data with polars - pfeed's Examples

import marimo as mo
import pfeed as pe
import polars as pl

Download data with pfeed¶

Download one day of 1-minute ETH perpetual bars from Bybit, and store them locally with ParquetIO().

feed = pe.Bybit().market_feed
df = feed.download(
    product="ETH_USDT_PERP",
    resolution="1m",
    start_date="2026-11-10",
    end_date="2026-11-10",
    io=pe.ParquetIO(),
).data.collect()
df

Running BYBIT dataflows: 100%|██████████| 1/1 [00:04<00:00,  4.98s/it]

data_path = pe.get_config().data_path

Read it back with polars¶

ParquetIO() stored the 1440 rows as a plain parquet file in a hive layout under <data_path>/parquet, where data_path is pfeed’s configured data folder:

env=BACKTEST
    └── data_layer=CLEANED
        └── data_domain=MARKET_DATA
            └── data_source=BYBIT
                └── data_origin=BYBIT
                    └── product_type=PERPETUAL
                        └── resolution=1_MINUTE
                            └── product=BYBIT_ETH_USDT_PERPETUAL
                                └── year=2026
                                    └── month=11
                                        └── day=10
                                            └── part-0.parquet

So polars can read them directly. With hive_partitioning=True, every key=value folder becomes a column, and filtering on those columns only reads the matching files.

polars_df = (
    pl.scan_parquet(
        data_path / "parquet" / "**" / "*.parquet", hive_partitioning=True
    )
    .filter(
        pl.col("data_source") == "BYBIT",
        pl.col("product") == "BYBIT_ETH_USDT_PERPETUAL",
        pl.col("resolution") == "1_MINUTE",
    )
    .select("date", "open", "high", "low", "close", "volume")
    .collect()
)
polars_df