Skip to content

GH-622: Modular footer thrift definition - #623

Open
Jiayi-Wang-db wants to merge 3 commits into
apache:masterfrom
Jiayi-Wang-db:modular-footer-thrift-draft
Open

Jiayi-Wang-db wants to merge 3 commits into
apache:masterfrom
Jiayi-Wang-db:modular-footer-thrift-draft

Conversation

@Jiayi-Wang-db

@Jiayi-Wang-db Jiayi-Wang-db commented Sep 21, 2026 •

Copy link
Copy Markdown

Rationale for this change

Support flexible data layouts and statistics over column-chunk, page, or arbitrary row ranges, while allowing readers to fetch placement and statistics independently.

What changes are included in this PR?

ModularFooter.thrift defines the footer directory and its metadata streams:

  • Typed Parquet page streams located by offset, length, and codec.
  • Shared column-to-chunk and chunk-to-page indexes, allowing each column to have a different chunk count.
  • Direct chunk offsets and sizes, a dictionary table with per-chunk references, and optional row-group and page metadata.
  • Tagged row-range statistics with exact null, NaN, and value counts and typed min/max bounds. Chunk and page statistics reuse shared row positions.
  • PME structures for directory authentication, shared-stream protection, and separately encrypted column statistics.
  • A 4-byte little-endian directory length followed by the 4-byte magic; the length includes any directory signature.

The definition uses the ModularFooter root and the parquet.modular / org.apache.parquet.format.modular namespaces. Format-version and PME module-type allocations remain open.

Do these changes have PoC implementations?

The design and measurements describe the prototype readers and writers. Measurements cover plaintext metadata streams and pruning predicates; exact-count statistics, this directory serialization, encryption, and column sets are outside the measured scope. The linked design describes column sets as an optional extension for very wide files.

Validation: the definition parses, and Apache Thrift 0.24.0 generates Java and C++ bindings; the generated C++ passes a syntax check, and all 83 generated Java classes compile with the cached Thrift 0.23.0 runtime. Full Maven verification could not complete locally because dependency repositories were unavailable.

Related to #622.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant