Parquet file format
Apache Parquet, a columnar storage format for big-data analytics.
- MIME type
- application/vnd.apache.parquet
- Extensions
- .parquet
- Category
- Data & databases
Magic-number signature
| 0x00000000 | 50415231 | PAR1 | 'PAR1' header |
| -0x00000004 | 50415231 | PAR1 | 'PAR1' footer (last 4 bytes) |
About the Parquet format
Parquet stores a table column by column instead of row by row, so a query that touches three columns out of a hundred reads only those three, and each column compresses far better because neighbouring values share a type and usually a narrow range. Everything descriptive — the schema, the row groups, each column chunk's offset, codec and min/max statistics — is serialized with Thrift's compact protocol and written after the data, which is why the file is anchored at both ends: PAR1 at offset 0 announces the format to a scanner, and PAR1 at the very end anchors the footer a reader actually starts from.
Header layout, byte by byte
| Offset | Length | Field | Meaning |
|---|---|---|---|
| 0 | 4 | File magic 'PAR1' | The four bytes 50 41 52 31. It exists so a tool scanning the front of a file can name the format; no reader parses anything else from this end. A file written with an encrypted footer carries the magic PARE here and at the tail instead, which tells the reader to expect file crypto metadata ahead of the footer. |
| 4 | variable | First column chunk | Data begins immediately after the magic — page after page of encoded values, each introduced by a Thrift compact PageHeader. There is no schema, no index and no row count at the front, so the first megabyte of a Parquet file tells you almost nothing about what is in it. |
| variable | variable | FileMetaData (Thrift compact) | The schema, the row groups and every column chunk's offset, codec, encodings and statistics. It sits directly before the last eight bytes, and a reader finds its start by subtracting the footer length from that position — never by scanning forward from offset 0. |
| -8 | 4 | Footer length | Unsigned 32-bit little-endian byte count of the FileMetaData that precedes it. A reader seeks to the last eight bytes, checks the magic, reads this number and seeks back that many bytes. A wrong value here is fatal even when every data page is intact. |
| -4 | 4 | Trailing magic 'PAR1' | The same four bytes as offset 0 — PARE when the footer is encrypted. Because the format is anchored at the tail, these are the bytes every reader tests first, which is why a half-downloaded Parquet file is rejected outright despite a perfectly good header. |
Telling it apart from its neighbours
| Easily confused with | How to tell |
|---|---|
| ORC | The other columnar format in the same pipelines, and the footer-at-the-end design is the same. ORC opens with the three bytes 4F 52 43 ('ORC') rather than a four-byte magic, and its last byte is not a magic at all but the postscript length, a single value under 256. So read the last four bytes: PAR1 means Parquet, an arbitrary small byte means ORC. |
| Avro OCF | Avro object container files come out of the same ingestion jobs but are row-oriented and self-describing. The header is the four bytes 4F 62 6A 01 ('Obj' followed by 1), then a metadata map holding avro.schema and avro.codec, then a 16-byte sync marker. There is no trailing magic, so an Avro file can be appended to and still read while a Parquet file cannot. |
| Encrypted Parquet (PARE) | Same layout, different magic at both ends: PARE instead of PAR1, chosen deliberately so an older reader fails on the magic rather than on unreadable Thrift. When the footer is encrypted the four-byte length before the trailing magic covers the crypto metadata plus the encrypted footer, so the number is larger than the plaintext case for the same table. |
| SQLite database | Both get handed around as 'the analytics file' and both are single-file, but SQLite is a row store with its magic in the first sixteen bytes and no trailing marker whatsoever. If offset 0 reads PAR1 and the last four bytes read PAR1, nothing about the middle needs checking. |
How these files break
- Truncated tail. Anything that stops a transfer early — a killed object-store download, a writer process that died before close — leaves the head magic and gigabytes of intact pages behind a footer that was never written. Every engine reports the file as not a Parquet file, because the test they run is on the last four bytes, not the first four.
- A footer length that no longer lines up. Concatenating two Parquet files, appending a byte, or letting an editor add a final newline shifts the anchor: the trailing magic can still read PAR1 while the four bytes before it point into the middle of a page, and the Thrift parser fails on garbage instead of reporting that the file was edited.
- A flipped byte inside a data page. The footer parses, the schema lists, and the column reads correctly right up to the damaged page, where the decompressor errors out or values decode as nonsense. Per-page CRCs are optional in Parquet, so many writers leave nothing behind that can point at which page moved.
- Silent schema drift across a directory. Not corruption of one file but the failure people actually hit: each file is valid on its own, yet one of them types a column as int32 where the rest say int64, and the query engine refuses the whole dataset. The answer is in each file's footer, not in its data.
Where the format stands today
Parquet is the default storage format of the analytics world. Spark, Trino, DuckDB, Polars, pandas, BigQuery exports and Snowflake unloads all read and write it, and the object-store table formats built on top — Iceberg, Delta Lake, Hudi — are directories of Parquet files plus a metadata log. It replaced CSV for anything large, because a CSV must be read and reparsed end to end for every query while a Parquet reader consults the footer, discards the row groups whose statistics rule them out, and decompresses only the columns asked for. Where it is still the wrong choice is small data and human interchange: a two-hundred-row lookup table is easier to keep as CSV, and record-at-a-time streaming ingestion belongs to Avro, because a Parquet file only becomes valid at the moment its footer is written.
Open Parquet files on this site
Every one of these runs in your browser — the file never leaves your machine.
Related formats
Formats that are easy to confuse with Parquet — compare their leading bytes.
| Format | First bytes | Extension |
|---|---|---|
| SQLite database | 53 51 4C 69 74 65 | .sqlite |