Text (UTF-8 with BOM) file format
Plain text encoded as UTF-8 with a byte-order mark.
- MIME type
- text/plain
- Extensions
- .txt
- Category
- Text & markup
Magic-number signature
| 0x00000000 | EFBBBF | ··· | UTF-8 BOM |
About the Text (UTF-8 with BOM) format
A byte-order mark is the character U+FEFF placed at the very start of a file. In UTF-16 it earns its name, because FF FE and FE FF are distinguishable and the reader genuinely needs to be told which way round the file is. In UTF-8 there is no byte order to mark: the three bytes EF BB BF exist purely as a signal that the file is UTF-8 at all, and the Unicode standard neither requires nor recommends them. Windows editors add them anyway, because the alternative on Windows was having the file read as the system code page, and the result is a permanent low-level friction between Windows and Unix tooling that has outlasted every attempt to settle it.
Header layout, byte by byte
| Offset | Length | Field | Meaning |
|---|---|---|---|
| 0 | 3 | UTF-8 byte-order mark | EF BB BF, the UTF-8 encoding of U+FEFF, a character whose Unicode name is zero width no-break space. Nothing may come before it: a mark that is not at offset 0 is not a mark at all, just an invisible character sitting in the middle of the text. |
| 0 | 1 | Lead byte EF | 1110 1111 in binary. The three high bits are what mark the start of a three-byte sequence, which is the whole reason the mark costs three bytes in UTF-8 and only two in UTF-16 - the character itself is unchanged, only its encoding is longer. |
| 1 | 2 | Continuation bytes BB BF | 10 111011 and 10 111111. Strip the two high bits from each, append what is left to the lead byte's four payload bits, and you get 1111 1110 1111 1111 - which is U+FEFF. Doing this by hand once is the fastest way to stop treating the mark as an arbitrary magic number. |
| 3 | variable | Start of the text | Everything the file is actually for begins here. A shebang line, an XML declaration, a JSON opening brace, a CSV header row, a PHP opening tag: all of them are now three bytes late, and every single problem this format causes is a consequence of that one fact. |
| 3 | 3 | A second byte-order mark | EF BB BF EF BB BF is common enough to be worth checking for deliberately. It happens when an editor that adds marks saves a file that already had one, and when files are joined together, in which case the second file's mark ends up buried in the middle of the result where nothing will ever show it to you. |
Telling it apart from its neighbours
| Easily confused with | How to tell |
|---|---|
| XML | Not alternatives at all - the mark is a prefix and the file underneath is still whatever it was. What changes is that the format's own recognition has to skip three bytes first. XML is one of the few standards that addresses this head on: it permits a byte-order mark in front of the declaration and permits nothing else, so those three bytes are legal and a single space is not. |
| UTF-16 text | Two bytes rather than three: FF FE for little-endian, FE FF for big-endian. You rarely need the mark to tell them apart, because UTF-16 text is unmistakable in a hex dump - every ASCII character carries a 00 beside it, so an English document appears as alternating letters and zero bytes down the whole right-hand column. |
| UTF-32 text | 00 00 FE FF for big-endian and FF FE 00 00 for little-endian. The little-endian one is genuinely ambiguous: those same four bytes are a UTF-16 little-endian mark followed by a NUL character, and the only way to choose between the readings is to look at what comes after them. |
| Windows-1252 text | The same bytes read under the wrong assumption. When a UTF-8 mark is decoded as CP1252 or Latin-1 it appears as three visible characters at the top of the document - an i with a diaeresis, a right-pointing double angle quote and an inverted question mark. Seeing those means the file is fine and the reader is wrong, which is the opposite of what it looks like. |
How these files break
- A mark in front of a script. The kernel looks for 23 21 at offset 0 to find the interpreter, finds EF BB BF instead, and the script fails with a message about a bad interpreter or an exec format error that mentions neither the mark nor the first line. The file looks perfect in every editor.
- A mark in a PHP include. The three bytes sit outside the opening tag, so they count as output, and the next call that tries to set a header fails with the headers-already-sent error, blaming a file that appears to have nothing above its first line.
- A mark in a CSV. The first column's name becomes the mark plus its text, so a lookup by header name misses while the file looks flawless on screen. Excel writes the mark deliberately - it is how Excel recognises a UTF-8 CSV - so this arrives most often in files that were exported exactly as intended.
- A mark in front of JSON. The specification says an implementation must not add one and may ignore one; the JavaScript engines behind Node and the browsers do neither, and throw an unexpected-token error naming a character that no editor will display and no cursor will land on.
- A mark in the middle. Joining files with cat, or with any code that appends text, leaves the second file's mark embedded in the result as a zero-width character. It survives diffs, it is invisible in editors, and it turns a string comparison that should pass into one that fails for no visible reason.
Where the format stands today
The mark is not going away and it is not becoming the norm either. Windows Notepad, the older PowerShell output cmdlets, Visual Studio and Excel all still write one, and almost everything born on Unix still treats a file that has one as slightly broken. The rule that has settled out in practice is to write UTF-8 without a mark everywhere, with one standing exception: a CSV that Excel will open, where the mark is the only reliable way to stop Excel guessing the system code page and mangling every accented name in the file. Editors expose the choice as two separate encodings, usually named UTF-8 and UTF-8 with BOM, and a project that mixes them will eventually produce a diff where a line looks changed and is not. The point most often missed is that the mark is content, not metadata: Git stores it, hashes cover it, byte counts include it, and a file with a mark and a file without one are different files by every measure any tool applies.
Open Text (UTF-8 with BOM) files on this site
Every one of these runs in your browser — the file never leaves your machine.
Related formats
Formats that are easy to confuse with Text (UTF-8 with BOM) — compare their leading bytes.