Arrow Flight SQL

Yoda exposes its OLAP backend over Arrow Flight SQL — a gRPC protocol that streams columnar Arrow data directly to clients without any intermediate serialisation. It is a read-only endpoint: analytical SELECT queries run against DataFusion or DuckDB, while writes continue to go through the OLTP path.

Feature flag

# Cargo.toml
[dependencies]
yoda = { version = "...", features = ["flight-sql"] }

The flight-sql feature pulls in tonic 0.14 and arrow-flight 58. The Arrow version must stay pinned at 58 to match the rest of the workspace (see OLAP backends).

How it works: deferred execution

The Flight SQL protocol splits a query into two round-trips:

1. get_flight_info — the client sends the SQL string. The server encodes it into an opaque Ticket and returns a FlightInfo immediately. No query planning happens at this step.
2. do_get — the client presents the ticket. The server decodes the SQL, calls OlapEngine::query_stream(), and streams Arrow IPC record batches back over the gRPC connection.

This deferred pattern keeps get_flight_info latency negligible and avoids holding engine resources while the client is not yet ready to consume data.

Streaming behaviour by backend

Backend	query_stream() implementation
DataFusion	True streaming via DataFrame::execute_stream() — no full-result buffering
DuckDB	Collect-then-stream — results are fetched fully, then emitted as a stream

Compression

Compression is applied to the Arrow IPC buffers sent over the wire.

CompressionType	Description	When to use
Zstd (default)	Best size/CPU tradeoff for columnar data	Most deployments
Lz4	Faster decompression, larger wire footprint	CPU-constrained clients
None	Zero codec overhead	Loopback / high-bandwidth LAN

To override the default, construct the service manually:

use yoda_flight::{serve_flight_sql_with_compression, CompressionType};

serve_flight_sql_with_compression(olap, addr, CompressionType::Lz4).await?;

Authentication

Optional bearer-token auth protects the gRPC endpoint. When configured, every RPC must carry authorization: Bearer {token} in the gRPC metadata; requests without a valid token are rejected with UNAUTHENTICATED.

Empty tokens are ignored

If flight_auth_token is set to an empty string (e.g. YODA_FLIGHT_AUTH_TOKEN=""), the server logs a warning and starts without authentication rather than accepting every request that presents no token. This prevents misconfiguration from creating a false sense of security.

Read-only constraint

All write operations (do_put_*) return tonic::Status::Unimplemented. The OLAP layer carries no DML semantics — use the OLTP path or yd exec for INSERT / UPDATE / DELETE.

# Writes go through OLTP, not Flight SQL
yd exec --db app.db "INSERT INTO orders VALUES (1, 99.50)"

Configuration

Add to [engine] in your TOML config (see Configuration):

[engine]
oltp_path    = "app.db"
olap_backend = "datafusion"

# Enable the Arrow Flight SQL server on port 50051
flight_port  = 50051

# Optional bearer-token (or set YODA_FLIGHT_AUTH_TOKEN env var)
# flight_auth_token = "your-secret-token"

flight_auth_token falls back to the YODA_FLIGHT_AUTH_TOKEN environment variable, so you can keep the secret out of TOML files on disk.

Feature gate required

flight_port has no effect unless yoda-tui is built with --features flight-sql.

Python client example

Install the ADBC Flight SQL driver:

pip install adbc-driver-flightsql adbc-driver-manager pyarrow

Connect and run a query:

import adbc_driver_flightsql.dbapi as flight

# No auth
conn = flight.connect("grpc://localhost:50051")

# With bearer-token auth
conn = flight.connect(
    "grpc://localhost:50051",
    db_kwargs={"adbc.flight.sql.authorization_header": "Bearer your-secret-token"},
)

with conn.cursor() as cur:
    cur.execute("SELECT region, SUM(amount) AS total FROM orders GROUP BY region")
    table = cur.fetch_arrow_table()
    print(table.to_pandas())

conn.close()

The result is returned as a PyArrow Table — zero-copy interop with pandas, polars, or any Arrow-compatible library.

DBeaver and other GUI tools

Any Flight SQL–compatible client (DBeaver, DataGrip via the Flight SQL JDBC driver, etc.) can connect to localhost:50051 using the Arrow Flight SQL driver. No extra configuration is needed beyond the host, port, and optional bearer token.

Source

crates/yoda-flight/ — lib.rs (server entry points), service.rs (FlightSqlService implementation).