Three small text jobs, each with a detail that decides whether the result is right or quietly wrong.
Why duplicates survive deduplication
A tool comparing lines exactly will treat two entries as different if anything at all differs, including things you cannot see.
Run through the usual causes before concluding the tool is broken.
- Trailing whitespace. "apple" and "apple " are different strings. This is the leading cause.
- Case. "Apple" and "apple" differ unless comparison is case-insensitive.
- Line endings. Windows CRLF against Unix LF, when lists come from different sources.
- Unicode normalisation. An accented character can be one code point or a letter plus a combining mark. Identical on screen, different in memory.
- Non-breaking spaces, which arrive from web pages and Word and look exactly like ordinary spaces.
So the order of operations is: trim each line, normalise the case if appropriate, normalise Unicode, then deduplicate. Cleaning first and comparing second turns an unreliable result into a correct one.
Order, and scale
Two decisions worth making deliberately. Whether the original order matters, and how large the list is.
Sorting and deduplicating together is fast and destroys the order. Preserving order requires remembering what has been seen, which is slower and usually what you want for anything human-facing like a mailing list.
| Size | Approach |
|---|---|
| Up to a few thousand | Any browser tool or a spreadsheet |
| Tens of thousands | A browser tool still handles it; spreadsheets slow |
| Hundreds of thousands | Command line: sort -u, or awk to keep order |
| Millions | A database, or a streaming script |
Excel's Remove Duplicates is convenient and has a trap: it compares the displayed value, so numbers formatted differently can be treated as distinct, and it silently ignores leading apostrophes. For a list that matters, clean the text first. Duplicate Line Remover works in the browser, which matters when the list is customer email addresses.
Where lorem ipsum comes from
It is mangled Latin from Cicero — De Finibus Bonorum et Malorum, written in 45 BC, from a passage about pleasure and pain. The familiar text begins mid-word, because it was cut from "dolorem".
Printers have used it as specimen text since at least the sixteenth century, and it entered modern design through Letraset sheets and then desktop publishing.
The purpose is real: placeholder text stops people reading the copy when they are supposed to be assessing the layout. Show a client a draft with plausible English in it and the conversation turns to the wording every time.
And where it misleads
The same property that makes it useful makes it dangerous late in a project. Lorem ipsum hides exactly the problems that real content creates.
Latin has different word lengths and letter frequencies from English, so line breaks fall differently and the texture of a paragraph is not the same. More importantly, it is uniform — every paragraph is a comfortable length, no heading is awkwardly long, no name overflows its container, and no list has a single item.
- Use it early, while assessing layout and hierarchy.
- Switch to realistic content before sign-off, including the awkward cases.
- Test the extremes: the longest plausible heading, the shortest paragraph, an empty state.
- Never let it reach production. It does, regularly, and it is embarrassing in a way that is hard to explain.
Lorem Ipsum Generator produces however much you need. Treat it as scaffolding: essential during construction, and a problem if it is still there when the building opens.
The Markdown that earns its keep
Markdown is small enough to learn in a sitting, and most of what people use is a handful of constructs.
Headings with hash marks, emphasis with asterisks, lists with dashes or numbers, links in square brackets followed by parentheses, and code in backticks. That covers the overwhelming majority of real use.
- Headings:
# Onethrough###### Six. Leave a space after the hashes. - Emphasis:
*italic*,**bold**. - Lists:
- item, or1. item. Indent nested items consistently. - Links:
[text](url). Images are the same with a leading!. - Code: backticks inline, or three backticks for a block with a language name.
- Quotes: a leading
>.
Tables and task lists are worth knowing and are extensions rather than core Markdown — they come from GitHub Flavored Markdown, so a strict CommonMark renderer may not support them. In practice most renderers do.
The line break problem
This is the one thing that surprises everybody, and it is by design rather than a quirk.
A single newline in Markdown does not produce a line break. Two lines of text with one newline between them render as a single paragraph, because Markdown treats consecutive lines as continuing prose — which is what you want for text wrapped in an editor.
To force a break, end the line with two spaces, or leave a blank line to start a new paragraph. Two trailing spaces are invisible, which is why the fix seems arbitrary and why many editors strip them on save. Where a genuine line break matters — an address block, a poem — a blank line between paragraphs or an explicit <br> is more robust. Markdown Editor previews as you type, which makes this obvious immediately rather than after publishing.
Frequently asked questions
Why did my duplicate remover miss obvious duplicates?
Because the entries differ invisibly — trailing whitespace is the usual cause, followed by inconsistent case, mixed line endings, non-breaking spaces from web pages, and accented characters stored as either one code point or a letter plus a combining mark.
How do I deduplicate while keeping the original order?
Use a method that remembers what it has seen rather than one that sorts. Sorting is faster and destroys the order, which matters for anything human-facing like a mailing list.
What is the best way to deduplicate a very large list?
Up to tens of thousands of lines, a browser tool handles it. Beyond that, the command line — sort -u when order does not matter, or awk to preserve it. Past a few million lines, use a database or a streaming script.
Where does lorem ipsum come from?
Mangled Latin from Cicero's De Finibus Bonorum et Malorum, written in 45 BC. The familiar text starts mid-word because it was cut from "dolorem". Printers have used it as specimen text since at least the sixteenth century.
Why not use real text instead of lorem ipsum?
Early on, placeholder text stops people critiquing the copy when they should be assessing the layout. Late on it hides the problems real content causes — long headings, overflowing names, single-item lists — so switch to realistic content before sign-off.
Why does my Markdown line break not work?
A single newline does not create a break — Markdown treats consecutive lines as one paragraph so that editor wrapping does not affect output. End the line with two trailing spaces, or leave a blank line for a new paragraph.
Are Markdown tables standard?
They come from GitHub Flavored Markdown rather than core CommonMark, along with task lists. Most renderers support them, but a strict CommonMark implementation may not.