Remove Duplicate Lines

Exactly how a duplicate is detected

The text is cut into lines at each line break and read from top to bottom. Each line is compared with every line already kept; if an identical one was seen, the line is dropped, otherwise it is added to the output. That is why the first occurrence survives and later copies disappear, and why the output keeps the original order unless you sort it.

With Trim whitespace on (the default), spaces and tabs at both ends of a line are removed before comparing and before output, so 'apple' and ' apple ' match and the kept line is the trimmed version. With Ignore case on, lines are compared lowercased, but the line that is kept retains its original capitalisation.

A worked example with blank lines

Paste nine lines: apple, apple, banana, Apple, ' apple ' with spaces around it, two empty lines, banana, and a final empty line. With the default settings the result is four lines: apple, banana, Apple and one empty line, and the label reads '5 duplicate lines removed'. Blank lines are deduplicated like any other line, so only the first blank survives. Switch on Ignore case and the result shrinks to apple, banana and one blank line, with 6 removed.

How Sort A→Z orders the lines

Sorting uses the browser's locale-aware comparison, not a plain byte order. Letters are grouped regardless of case, so 'Apple', 'apple' and 'ápple' end up next to each other, and punctuation such as an underscore sorts before digits and letters. Numbers are compared as text, which puts '10' before '9'; for a numeric list, sort it in a spreadsheet instead.

Sorting is applied after duplicates are removed, and only to the output. Your pasted text is never rearranged.

Lines that look identical but are not

• Different spacing inside a line: 'apple pie' with one space and with two spaces are different lines. Trimming only affects the ends.
• Tabs versus spaces inside the line, which are invisible in the box but compared literally.
• Accented letters typed two different ways: a precomposed 'é' and an 'e' followed by a combining accent look the same but do not match.
• A trailing carriage return from Windows line endings when Trim whitespace is off. Leave trimming on and it is removed.
• Items separated by commas on one line: the tool only works on line breaks, so put each item on its own line first.

Where it saves the most time

• Email and mailing lists exported from two systems and pasted together, where Ignore case catches addresses typed with different capitalisation.
• Keyword lists for ads or SEO research merged from several exports.
• Log files and error output where the same line repeats hundreds of times and you want each distinct message once.
• A column copied from a spreadsheet to find the distinct values, with Sort A→Z to produce a tidy lookup list.