Duplicate Line Remover

How to use Duplicate Line Remover

1. Paste your list — one item per line — into the input box.
2. Set the options: trim whitespace, ignore case, keep first or last, and sort.
3. Copy the deduplicated list, or switch to the frequency view to see occurrence counts.

What counts as a duplicate?

By default, an exact byte-for-byte match. apple and Apple are different lines, and apple with a trailing space is different again.

That default is deliberately strict, because the two most common real-world lists — email addresses and URLs — have case rules that are not intuitive. The domain part of an email is case-insensitive, but the local part technically is not. URL paths are case-sensitive on Linux servers and case-insensitive on Windows ones.

Turn on "trim whitespace" and "ignore case" together and the tool normalizes each line before comparing, while still outputting the original text of whichever copy it keeps. This is almost always what you want for a mailing list.

Should I keep the first or the last occurrence?

It depends on whether your list is ordered by time.

Keep-first preserves the order of first appearance and is right for lists where earlier means more authoritative — a deduplicated import queue, or a priority-ordered set of rules.

Keep-last is right for append-only logs where each new line supersedes the previous one. Configuration files, CSV exports where a record was corrected and re-appended, and event streams all read this way.

Both modes preserve the original order of the surviving lines. Turning on sort discards that order in favor of alphabetical, which is useful for producing a canonical list you can diff against another one later.

How do I count how many times each line appears?

Switch to frequency view. Instead of a deduplicated list, you get each unique line with its occurrence count, sorted by count descending.

This turns the tool into a quick tally for anything you can get onto separate lines: the most common referrer in a log file, the most repeated error message, the distribution of values in a spreadsheet column, or the top search queries in a Search Console export. Paste the column, switch views, and read the top of the list.

What is the performance ceiling?

Deduplication uses a JavaScript Set, which is a hash table with average O(1) insertion and lookup. Total runtime is O(n) in the number of lines.

In practice a one-million-line list — roughly 25 MB of email addresses — deduplicates in well under a second on a mid-range laptop. The bottleneck is rendering the result to the page rather than the deduplication itself, which is why very large outputs render into a plain textarea instead of a styled list.

A server-based tool would spend more time uploading that 25 MB than this one spends processing it.

Why does this matter for privacy?

Look at what people actually deduplicate: customer email lists, lead exports from a CRM, phone numbers, order IDs, employee rosters.

Under GDPR, pasting a list of EU customer emails into a website that uploads it to a server is a transfer of personal data to a third-party processor. You need a lawful basis and, in most cases, a data processing agreement with that site — which free tools do not offer.

Processing the list in your own browser is not a transfer at all, because the data never reaches another controller. For anyone handling EU, UK or California resident data, that distinction is the difference between a routine task and a compliance problem.

Frequently asked questions