Remove Duplicate Lines

Paste a list and get it back with the repeats taken out, in the original order unless you ask for it sorted. It tells you exactly how many lines it removed, so you can check the result is what you expected.

Your text

Ignore case Treats Apple and apple as the same line. Ignore surrounding whitespace Treats a line with trailing spaces as identical to one without. Worth leaving on - invisible whitespace is the usual reason a duplicate is missed. Remove empty lines Sort the result alphabetically Keep the last occurrence instead of the first Show only the duplicates Inverts the tool: instead of removing repeats, it lists the lines that appeared more than once.

Result

How the matching works

A line counts as a duplicate when it is character-for-character identical to one that came before it. The two options above relax that comparison - one folds upper and lower case together, the other ignores spaces and tabs at either end of the line.

Trailing whitespace is worth being careful about, because it is invisible. Two lines that look identical on screen but differ by one trailing space are not duplicates to any exact comparison, and this is far and away the most common reason a deduplication appears not to have worked. That is why the whitespace option is on by default.

Order is preserved by default: the first time a line appears it is kept, and every later copy is dropped, so a list you had already arranged stays arranged. Sorting is available when you want it, and the keep the last occurrence option matters when your lines are log entries or records where the most recent copy is the one you want.

Questions

Does it keep the original order of the lines?

Yes, by default. The first occurrence of each line stays where it was and later copies are removed. Sorting alphabetically is optional, and there is also an option to keep the last occurrence rather than the first.

Why were two identical-looking lines not treated as duplicates?

Almost always because of invisible whitespace or a difference in capitalisation. Switch on ignore surrounding whitespace and ignore case, and they will match.

Can I see which lines were duplicated instead of removing them?

Yes. Switch on show only the duplicates and the tool inverts: rather than returning a cleaned list, it returns just the lines that appeared more than once.

Is there a limit on how much text I can paste?

There is no fixed limit. The work happens in your browser, so the practical ceiling is your device's memory. Lists of tens of thousands of lines are processed effectively instantly.

Is my list uploaded?

No. Everything runs in your browser. There is no endpoint on this site that receives the text.

Text broken into short lines instead? Use the line break remover.

Somewhere to keep the tidied version

Snoq is a Windows notes app with a rich text editor, tabs and full-text search, storing everything in an AES-256 encrypted database on your own disk. No account, no cloud, no telemetry, and free.

Download Snoq for Windows