Text copied from a website, PDF, or messenger can look normal yet fail a search, survive duplicate removal, or exceed an expected character count. One possible cause is that the text contains characters other than the ordinary space you can see and type with the spacebar.
Common examples include the no-break space (NBSP, U+00A0), zero width space (U+200B), and smart quotation marks (U+2018, U+2019, U+201C, and U+201D). This article measures controlled strings containing each type to show how character count, UTF-8 size, search, and exact duplicate checks change.
The practical conclusion is simple: visual similarity does not mean identical data. These characters also have legitimate uses, so clean only what conflicts with the destination of the text.
Character codes, not appearance, determine exact comparison
A program that splits or searches only for ordinary U+0020 spaces may not treat it as the same text.
It adds a Unicode code point and UTF-8 bytes even when it creates no visible width.
They suit readable documents but may need conversion for code, data, or a search expression that requires straight quotes.
Six visually similar strings were measured under the same rules
- 1
Create an ordinary-space and NBSP version of ‘견적서 검토 완료’, a no-space and zero-width-space version of ‘견적서검토완료’, and straight-quote and smart-quote versions of ‘문서 확인 완료’.
- 2
Count Unicode code points by expanding each string and calculate UTF-8 bytes with TextEncoder, which is also available in modern browsers.
- 3
Compare each special-character string with its baseline character by character and split the space samples using only ordinary U+0020.
- 4
Place each visually matching pair in a Set, which keeps only values that are exactly equal, and count whether one or two entries remain.
- 5
Replace NBSP with ordinary space, remove the test U+200B characters, and convert smart quotes to straight quotes; then compare each cleaned result with its baseline again.
Measured result: similar appearance did not produce equal data
The measurements were run on September 1, 2026. Length means Unicode code points and size means bytes after UTF-8 encoding. ‘Exact match’ tests whether every internal character equals the baseline rather than whether the strings look similar to a reader.
The NBSP sample retained nine code points but used two more UTF-8 bytes than the ordinary-space sample and failed exact comparison. Two zero width spaces increased the no-space sample from 7 to 9 code points and from 21 to 27 bytes. Curly double quotes kept the same 10 code points as straight quotes but used four more bytes and remained a different string.
| Test string | Key internal characters | Code points | UTF-8 | Exact baseline match |
|---|---|---|---|---|
| 견적서 검토 완료 | Two ordinary U+0020 spaces | 9 | 23 bytes | Yes |
| 견적서 검토 완료 | Two U+00A0 NBSP characters | 9 | 25 bytes | No |
| 견적서검토완료 | No inserted character | 7 | 21 bytes | Yes |
| 견적서 + 검토 + 완료 | Two U+200B zero width spaces | 9 | 27 bytes | No |
| 문서 "확인" 완료 | Straight U+0022 quotes | 10 | 22 bytes | Yes |
| 문서 “확인” 완료 | Smart U+201C and U+201D quotes | 10 | 26 bytes | No |
NBSP is a space designed to prevent a line break
NBSP stands for no-break space. It normally occupies the same visible width as an ordinary U+0020 space but prevents a line from breaking at that position. The Unicode Standard describes U+00A0 as the nonbreaking counterpart of U+0020 with the same width and different line-breaking behavior.
It can be copied from a web page where a number and unit or a name should stay together. In this test, splitting on ordinary spaces produced three pieces for the baseline and only one for the NBSP string. Searching for the ordinary-space phrase did not find the NBSP version either.
- Appearance: commonly indistinguishable from an ordinary space
- Search and splitting: can differ when software expects only U+0020
- Cleaning decision: convert to ordinary space for data entry or deduplication when nonbreaking layout is unnecessary
Zero width space is an invisible word boundary
U+200B ZERO WIDTH SPACE marks a word boundary or possible line break without a visible width. The Unicode Line Breaking Algorithm describes it as an additional invisible break opportunity where an ordinary space cannot be used.
The plain ‘견적서검토완료’ and a version containing U+200B between its three parts can appear identical. In the measurement, the latter held two extra code points and six extra UTF-8 bytes, and it did not match a search or duplicate check against the uninterrupted baseline.
- A field exceeds a character limit with no visible explanation
- Apparently identical rows remain after duplicate removal
- An account name, product code, or search term does not match after copying
Smart quotes are correct for documents but risky in strict data
Document editors can replace straight quotes with opening and closing shapes such as ‘ ’ and “ ”. Unicode assigns straight U+0022 and smart U+201C/U+201D quotation marks different code points.
Smart quotes can improve typography in a report or publication. JSON, code, command lines, some CSV workflows, and literal search expressions may require straight quotes instead. Convert them according to the destination rather than treating every smart quote as an error.
Duplicate checks compare values rather than what the eye sees
Putting the ordinary-space and NBSP strings into one exact-value set produced two entries, not one. The no-space and zero-width-space strings, and the straight-quote and smart-quote strings, also remained separate pairs. Exact comparison follows the sequence of character codes.
After replacing NBSP with ordinary space, removing the test U+200B characters, and converting curly double quotes to straight quotes, all three special samples exactly matched their baselines. This was a controlled case with known substitutions. Real documents can also contain tabs, ideographic spaces, direction controls, and combining characters.
Do not delete every invisible character automatically
NBSP and zero width space exist for line breaking and language processing, while smart quotes are valid punctuation. Cleaning can help a plain data list, search term, command, or code sample, but it can change a publication, multilingual document, or signed text.
A safer sequence is to keep the source, reproduce the problem, identify the suspected character, apply only the necessary conversion, and compare the result. Then read names, languages, units, quotation marks, and line breaks rather than trusting only a lower character count.
- Search or deduplication list: unify unwanted spaces and inspect zero-width characters
- Code, JSON, and commands: preserve the exact quote and whitespace rules of the syntax
- Reading and publishing: decide whether smart quotes and nonbreaking spaces should remain
- Important source: keep a copy and compare before and after cleaning
Scope and limits of this experiment
The measurements are reproducible for the listed strings under UTF-8. Different words, counts, or encodings such as UTF-16 produce different byte totals. Fonts and layout can also change how NBSP and smart quotes appear.
There is no universally safe ‘remove all invisible characters’ operation. Some characters are required for language and layout. The evidence shows that an unexpected character can change exact search, splitting, deduplication, and size—not that every special character is harmful.
Official documentation used for this analysis
Inspect character codes when visually identical text behaves differently
NBSP, zero width space, and smart quotes are valid Unicode characters, but they are not the same data as ordinary spaces or straight quotes. The controlled test found measurable differences in exact equality, ordinary-space splitting, search, duplicate detection, and UTF-8 size.
When copied text behaves unexpectedly, preserve the source, clean only the characters that conflict with its destination, and compare again. Accurate cleanup preserves useful document punctuation and language behavior instead of removing it mechanically.