What a byte order mark is
A BOM is a few bytes at the very start of a text file that say which Unicode encoding it uses: EF BB BF for UTF-8, FF FE for UTF-16 little-endian (Windows' "Unicode"), FE FF for UTF-16 big-endian, and FF FE 00 00 or 00 00 FE FF for UTF-32. Decoded, it is the character U+FEFF. UTF-16 and UTF-32 need it to tell the byte order; UTF-8 does not, and most tools that are not Windows programs expect UTF-8 without it.
The .NET code that writes one
Recorded on .NET 10 by writing the text "é" and reading the first bytes back:
| Code | Writes a BOM | |
|---|---|---|
File.WriteAllText(path, text) | No | UTF-8 without a BOM |
File.WriteAllText(path, text, Encoding.UTF8) | Yes | Encoding.UTF8 includes its preamble EF BB BF |
File.WriteAllLines(path, lines, Encoding.UTF8) | Yes | the same |
File.WriteAllText(path, text, new UTF8Encoding(false)) | No | the way to say UTF-8 without a BOM |
new StreamWriter(path) | No | default: UTF-8 without a BOM |
new StreamWriter(path, false, Encoding.UTF8) | Yes | with a BOM |
File.WriteAllBytes(path, Encoding.UTF8.GetBytes(text)) | No | GetBytes never adds the preamble |
XmlWriter.Create(path) | Yes | default XmlWriterSettings use UTF-8 with a BOM |
JsonSerializer.Serialize(stream, value) | No | UTF-8 without a BOM |
using System.Text;
string path = Path.GetTempFileName();
File.WriteAllText(path, "caf\u00e9"); // no BOM
Console.WriteLine(Convert.ToHexString(File.ReadAllBytes(path))); // prints 636166C3A9
File.WriteAllText(path, "caf\u00e9", Encoding.UTF8); // Encoding.UTF8 writes a BOM
Console.WriteLine(Convert.ToHexString(File.ReadAllBytes(path))); // prints EFBBBF636166C3A9
File.WriteAllText(path, "caf\u00e9", new UTF8Encoding(false)); // UTF-8 without a BOM
Console.WriteLine(Convert.ToHexString(File.ReadAllBytes(path))); // prints 636166C3A9Encoding.UTF8 is the trap: it is "UTF-8 with a BOM" whenever something asks the encoding for its preamble, which File and StreamWriter do. Encoding.Default on .NET (not .NET Framework) is UTF-8 without a BOM.
"'0xEF' is an invalid start of a value"
This System.Text.Json error means the JSON starts with a UTF-8 BOM. Encoding.UTF8.GetString does not remove it (the string starts with U+FEFF, and Trim() keeps it too), and JsonDocument.Parse and JsonSerializer.Deserialize reject both the bytes and that string. Reading through a Stream works, because the serializer skips a BOM there, and so do File.ReadAllText and StreamReader. Otherwise skip the three bytes yourself:
using System.Text;
using System.Text.Json;
byte[] json = [0xEF, 0xBB, 0xBF, (byte)'{', (byte)'}'];
// Encoding.UTF8.GetString keeps the BOM as U+FEFF, and the JSON parser rejects it.
string text = Encoding.UTF8.GetString(json);
Console.WriteLine(text.Length); // prints 3
try { JsonDocument.Parse(text); }
catch (JsonException e) { Console.WriteLine(e.Message.Split('.')[0]); } // prints '0xEF' is an invalid start of a value
// Skip it (or read through a StreamReader or File.ReadAllText, which remove it).
string clean = Encoding.UTF8.GetString(json.AsSpan(Encoding.UTF8.Preamble.Length));
Console.WriteLine(JsonDocument.Parse(clean).RootElement.ValueKind); // prints ObjectRemove a BOM in C#
using System.Text;
string path = Path.GetTempFileName();
File.WriteAllText(path, "{\"id\":1}", Encoding.UTF8); // a JSON file with a BOM
// Remove a UTF-8 BOM from a file, if it has one.
byte[] bytes = File.ReadAllBytes(path);
if (bytes.AsSpan().StartsWith(Encoding.UTF8.Preamble)) File.WriteAllBytes(path, bytes[3..]);
Console.WriteLine(Convert.ToHexString(File.ReadAllBytes(path))); // prints 7B226964223A317DWhen you want a BOM
Excel opens a UTF-8 CSV file without a BOM in the machine's legacy code page, so accented letters come out garbled. For CSV files meant for Excel, write UTF-8 with a BOM (Encoding.UTF8 or new UTF8Encoding(true)), or use the "Download with UTF-8 BOM" button above.
FAQ
What is the UTF-8 BOM?
The three bytes EF BB BF at the start of a file, the UTF-8 encoding of U+FEFF. It marks the file as UTF-8. It is optional, and many parsers (JSON, shell scripts, some CSV readers) treat it as garbage.
How do I remove the BOM from a file?
Drop the file above and download it without the BOM. In C#, check whether the bytes start with Encoding.UTF8.Preamble and write the rest; in editors, save as "UTF-8" rather than "UTF-8 with BOM".
Why does Encoding.UTF8 write a BOM?
Encoding.UTF8 is a UTF8Encoding that emits its preamble. File.WriteAllText and StreamWriter write the preamble of the encoding you pass. Pass new UTF8Encoding(false), or no encoding at all, for UTF-8 without a BOM.
Why does my JSON fail with '0xEF' is an invalid start of a value?
The JSON starts with a UTF-8 BOM. Remove it from the file, skip the first three bytes, or deserialize from a Stream, which skips it.
Do files leave my browser?
No. They are read and rewritten in the page.