What is UTF-8 Character Encoding? A Real Bug That Finally Made It Click

Date

February 18, 2026

The Bug That Started This

I was adding an emoji picker to an existing message input feature. Nothing fancy — users type messages, optionally add an emoji, and the messages get stored. There was also an existing feature that let users download all messages as a CSV file, which they’d open in Excel.

The feature worked. Or so I thought.

When I tested it end-to-end, I noticed something strange in the downloaded CSV: some emojis displayed correctly in Excel, and some showed up as garbled nonsense — strange boxes, question marks, or completely wrong characters.

Same code. Same download function. Some emojis fine, some broken.

I had no idea where to start.

The Fix — and the Questions It Raised

The fix eventually came down to two small additions to the CSV download logic:

const BOM = encodeDownloadFileInUTF8 ? ["\uFEFF"] : [];

const csvBlob = new Blob([...BOM, data || blob], {
  type: `text/csv;${encodeDownloadFileInUTF8 ? "charset=utf-8;" : ""}`,
});

Two things were added:

A mysterious "\uFEFF" prepended to the file — called a BOM
charset=utf-8 added to the MIME type

Both were added. The emojis rendered correctly in Excel.

But I refused to just move on without understanding why. What is a BOM? What does charset=utf-8 actually tell the browser? And why were only some emojis broken in the first place?

Answering those questions meant going all the way back to basics — to how computers represent text at all.

How Does a Computer Store a Letter?

Here’s something that seems obvious once you hear it, but nobody spells out clearly:

Computers only understand numbers. Everything in memory — images, videos, code, text — is ultimately stored as binary digits: ones and zeros.

A number like 65 is easy. In binary it’s 01000001. Done.

But what about the letter "A"? You can’t directly convert a character to binary the way you can a number. A character isn’t a number — it’s a symbol.

So how does it work?

The answer is: we agreed on a map.

What Is a Character Set? ASCII and Unicode Explained

A character set (or charset) is essentially a lookup table. It maps every character — letters, digits, punctuation, symbols — to a unique number. Once a character has a number, that number can be converted to binary, and the machine can store it.

The two most important character sets:

ASCII was one of the earliest and simplest. It maps 128 characters to numbers 0–127. The letter "A" is 65. "B" is 66. A space is 32. It covers the basic English alphabet, digits, and common punctuation — nothing more.

Unicode is the modern, universal standard. Instead of 128 characters, it maps over 140,000 of them — every letter from every human language, mathematical symbols, and yes, emojis. The 😊 emoji, for example, is Unicode code point U+1F60A. In JavaScript, you can verify this:

"A".charCodeAt(0);   // 65  — the number Unicode (and ASCII) maps "A" to
"😊".codePointAt(0); // 128522 — the number Unicode maps this emoji to

So the character set answers: “What number does this character correspond to?”

But there’s a second question: “How do we actually store that number in bytes?”

That’s where encoding comes in.

What Is UTF-8 and How Does It Work?

UTF-8 is not a character set. It’s an encoding — a set of rules for turning Unicode code point numbers into actual bytes that get written to memory or a file.

Here’s the key insight that makes UTF-8 clever:

Simple characters (basic English letters, digits) that have small code point numbers get stored in 1 byte
Characters from other languages and accented letters use 2 or 3 bytes
Emojis and rare symbols use 4 bytes

This is what “variable-width encoding” means. UTF-8 uses as few bytes as necessary for each character.

"A"  → code point 65     → UTF-8: 1 byte  → 01000001
"é"  → code point 233    → UTF-8: 2 bytes → 11000011 10101001
"中" → code point 20013  → UTF-8: 3 bytes → 11100100 10111000 10001101
"😊" → code point 128522 → UTF-8: 4 bytes → 11110000 10011111 10011000 10001010

This is why UTF-8 became the dominant encoding for the web. It’s backward-compatible with ASCII (the first 128 characters encode identically), space-efficient for English text, and capable of representing every character in existence.

When you write <meta charset="UTF-8"> at the top of your HTML, you’re telling the browser: “When you read this file’s bytes, decode them using UTF-8 rules.”

Without that declaration, the browser guesses. And when it guesses wrong, characters break.

Why Were Only Some Emojis Breaking in the CSV?

Now that the foundation is set, we can diagnose exactly what was happening.

The CSV was being generated and downloaded without explicitly declaring its encoding. Excel, when it opens a CSV, also has to guess the encoding. And Excel’s default assumption — especially on Windows — is not UTF-8. It often defaults to a legacy encoding like Windows-1252, which only handles 256 characters.

Here’s why only some emojis broke:

Simple emojis that happened to fall within the range Excel’s legacy encoding could interpret showed up fine — by accident
Emojis with 4-byte UTF-8 sequences that fell completely outside the legacy encoding’s range came out as garbage

It wasn’t a random bug. It was entirely predictable once you understand encoding.

The Fix, Explained

Two things were added to the code. They look similar in purpose but operate at completely different layers — and only one of them actually fixed the bug.

What Is a BOM (Byte Order Mark) and Why Does It Fix Excel?

const BOM = encodeDownloadFileInUTF8 ? ["\uFEFF"] : [];
const csvBlob = new Blob([...BOM, data || blob], { ... });

BOM stands for Byte Order Mark. It’s a special invisible character (U+FEFF) prepended to the very start of a file.

To understand what the BOM actually is, you need a tiny bit of backstory.

When a number takes up multiple bytes in memory, there are two valid ways to arrange those bytes — most significant byte first (big-endian) or least significant byte first (little-endian). Both are legitimate. But if the reader doesn’t know which order was used, they’ll misread the number entirely.

UTF-16 and UTF-32 store characters in multi-byte chunks, so byte order matters for them. The character U+FEFF was placed at the very start of files so that readers could detect which byte order was used — if they read it as FE FF it’s big-endian, if they read it as FF FE it’s little-endian. That’s where the name Byte Order Mark comes from.

UTF-8 doesn’t actually need this. It processes one byte at a time and builds up characters using its own rules, so byte order is never ambiguous. The BOM has no technical purpose in UTF-8.

But it got repurposed as a convention. When \uFEFF is encoded in UTF-8, it produces a specific three-byte sequence at the start of the file: EF BB BF. Excel on Windows recognises these exact bytes as a signal meaning “this file is UTF-8.” It has nothing to do with byte order anymore — Excel just looks for those three bytes, and if it finds them, it stops guessing and decodes the whole file as UTF-8.

The BOM is what actually fixed the bug. It survives the journey from browser to disk to Excel because it lives inside the file itself — not in any header or metadata that gets stripped away.

Does `charset=utf-8` in the MIME Type Fix the Problem?

type: `text/csv;charset=utf-8;`

This tells the browser — and any application that reads MIME types — that the content is encoded in UTF-8.

But here’s the honest reality: for the specific bug we were fixing, this alone would not have helped.

When a user downloads a CSV and opens it by double-clicking, Excel on Windows doesn’t see the MIME type. That information exists at the HTTP/browser layer and gets lost the moment the file is saved to disk. Excel never gets a chance to read it.

So why add it at all? Because not everything that reads your file is Excel:

A Python script processing the CSV programmatically will read the MIME type and decode correctly
An API consuming the file will respect it
A browser rendering the CSV directly will use it
It documents intent — any tool that does respect MIME types will handle the file correctly

Think of it this way:

	`charset=utf-8` in MIME	BOM `\uFEFF`
Lives in	HTTP / MIME headers	The file’s actual bytes
Survives saving to disk	❌ No	✅ Yes
Excel on Windows reads it	❌ Not reliably	✅ Yes
Programmatic tools read it	✅ Yes	Mostly ignored

The MIME type tells tools that ask what encoding this is. The BOM tells tools — like Excel — that don’t bother asking.

For this bug, if you had to pick just one: the BOM was the fix. The charset=utf-8 was the right thing to add alongside it — but it was good practice, not the cure.

The Mental Model, Simplified

Character  →  [Character Set]  →  Code Point Number  →  [Encoding]  →  Bytes in memory/file

   "A"     →    Unicode        →        65            →    UTF-8     →   01000001
   "😊"    →    Unicode        →      128522          →    UTF-8     →   4 bytes

Character Set (Unicode): gives every symbol a unique number
Encoding (UTF-8): defines how to turn that number into actual bytes
BOM (\uFEFF): a signal baked into the file’s bytes — survives to disk, read by Excel
charset=utf-8: a signal in the MIME/HTTP layer — read by browsers and programmatic tools, but lost when the file is saved locally

What to Take Away

The next time you see <meta charset="UTF-8"> or charset=utf-8 in a content type header, you’ll know exactly what it’s doing. It’s not boilerplate. It’s a contract — you’re telling every tool in the chain: “Here’s the encoding I used to write these bytes. Please use the same rules to read them.”

When that contract is missing or mismatched, text breaks. Sometimes obviously, sometimes only for edge cases like 4-byte emoji sequences that a legacy encoding never anticipated.

A small invisible character at the start of a file fixed my bug. Now I’ll never forget what it means.

Frequently Asked Questions

What is UTF-8 encoding? UTF-8 is a variable-width character encoding system that converts Unicode code points into bytes. It uses 1 byte for basic ASCII characters and up to 4 bytes for emojis and rare symbols. It is the most widely used encoding on the web.

What is the difference between a character set and an encoding? A character set (like Unicode or ASCII) maps characters to unique numbers called code points. An encoding (like UTF-8) defines the rules for converting those code point numbers into actual bytes stored in memory or files. UTF-8 is an encoding of the Unicode character set.

Why does <meta charset="UTF-8"> need to be in every HTML file? Without it, the browser has to guess how to interpret the file’s bytes. If the guess is wrong, characters render as garbage — especially accented letters, non-Latin scripts, and emojis. Declaring charset=utf-8 tells the browser exactly which rules to use.

What is a Byte Order Mark (BOM) and when do you need it? A BOM (\uFEFF) is a special invisible character prepended to a file’s bytes. In UTF-8 files, it encodes as EF BB BF and signals to applications like Excel on Windows that the file is UTF-8 encoded. It is particularly important when generating CSVs that users will open in Excel, since Excel does not reliably read the MIME type charset declaration after a file is saved to disk.

Why do only some emojis break in Excel CSVs? Excel on Windows defaults to a legacy encoding (like Windows-1252) when opening CSVs without explicit encoding signals. Emojis that happen to fall within that legacy encoding’s range display by accident. Emojis requiring 4-byte UTF-8 sequences — outside the legacy encoding’s range entirely — come out as garbled characters.

Co-written with AI.