Remove Duplicate Lines
Paste a list and get back each line only once, with options for letter case, extra spaces, empty lines and sorting.
🔒 Runs entirely in your browser. Nothing you enter is sent to a server.
Lists collect duplicates. Email addresses exported from two systems, keywords gathered from several tools, product codes copied from a few spreadsheets. This tool strips out the repeats and gives you each line once, with sensible options for the small differences that usually get in the way.
How to use the duplicate line remover
- Paste your list into the left box, one item per line.
- Choose how to compare lines:
- Letter case: ignore it so “Apple” and “apple” match, or match exactly.
- Spaces: ignore spaces and tabs at the start and end of each line, or match them exactly.
- Decide whether to remove empty lines.
- Keep the original order, or sort the result A–Z or Z–A.
- Copy the result, download it, or click Use output as input to keep working on it.
The status line tells you how many duplicates and empty lines were removed and how many unique lines are left.
How it works
The tool reads your text line by line and keeps a record of every line it has already seen:
- Optionally trim the line and convert it to lowercase. This is the comparison key.
- If the key has been seen before, skip the line and count it as a duplicate.
- Otherwise, remember the key and keep the line exactly as it was.
Looking up a key in this record takes the same time no matter how many lines came before, so even very long lists are processed almost instantly.
for each line:
key = options(line)
if key in seen: removed += 1
else: seen.add(key); output.append(line)Worked example
Paste this nine-line list, which includes an empty line and “banana” with a trailing space:
apple
Banana
cherry
apple
banana
cherry
date
AppleWith the default settings (ignore case, ignore spaces at the ends, remove empty lines):
- “apple”, “Banana”, “cherry” and “date” are kept the first time they appear.
- The second “apple”, “banana “, the second “cherry” and “Apple” are removed as duplicates: 4 duplicates.
- The empty line is removed: 1 empty line.
- Result: 4 unique lines: apple, Banana, cherry, date.
Switch to Match case exactly and “Banana”, “banana” and “Apple” are now all different, so you get 6 unique lines and only 2 duplicates are removed.
When to use which option
| You are cleaning | Suggested settings |
|---|---|
| Email addresses | Ignore case, ignore spaces |
| Keywords or tags | Ignore case, ignore spaces, sort A–Z |
| Passwords or codes | Match case exactly |
| Code or config lines | Match case, match spaces exactly |
| Names | Ignore spaces, but consider matching case |
Tips and pitfalls
- Clean up first if needed. If items are separated by commas rather than new lines, replace the commas with line breaks first.
- Sorting is natural. “item 2” sorts before “item 10”, the way people expect, rather than strictly character by character.
- Look out for look-alike characters. Text copied from documents may contain curly quotes, non-breaking spaces or long dashes that make two lines differ.
- Keep a copy of the original. Deduplication cannot be undone once you overwrite your source, so paste the result somewhere new.
- Your data stays private. The whole process runs in your browser, so you can safely clean customer lists and other sensitive data.
Frequently asked questions
Which copy of a duplicate is kept?
The first one. Each line is kept the first time it appears, and later copies are removed. With "keep original order", the result stays in the order of those first appearances.
What does "ignore case" do?
It treats lines that differ only in capital letters as the same, so "Apple", "apple" and "APPLE" count as one. The version that appears first is the one that is kept, with its original capitalisation.
Why are some lines that look identical not removed?
Usually because of hidden differences such as a space at the end of a line, a tab, or a different kind of apostrophe or dash. Choose "Ignore spaces at the ends" to deal with trailing spaces. Characters that only look alike, such as a straight ' and a curly ’, are still different.
Is there a limit to how many lines I can paste?
There is no fixed limit. The tool runs in your browser and handles lists of hundreds of thousands of lines in a moment on a typical computer.
Can I remove duplicates from a CSV or spreadsheet column?
Yes. Copy the column from your spreadsheet, paste it here, and paste the result back. For a full CSV file, whole rows are compared, so a row only counts as a duplicate if every field matches.