Remove duplicate lines
Delete repeated lines from a list, keeping the first or last occurrence, with case-sensitivity and trimming options.
0 / 2,000,000 characters · Updates as you type.
Paste a list (email addresses, keywords, URLs, log lines, product codes) and every repeated line is removed while the original order of the survivors is kept. The facts panel reports how many lines went in, how many came out and how many duplicates were dropped.
Choose whether "Apple" and "apple" count as the same line, whether spaces at the start and end of a line should be ignored (a common source of near-duplicates when copying from spreadsheets), and whether to keep the first or the last occurrence. Keeping the last is useful for logs and exports where later lines override earlier ones. Blank lines are treated like any other line unless you choose to leave them in place.
The tool works line by line on the exact text, so "example.com" and "https://example.com" stay separate. To sort the result, use the sort-lines tool afterwards.
How to use it
- Paste your lines into the box, one item per line.
- Set case sensitivity, trimming and whether to keep the first or last occurrence.
- Check the counts, then copy or download the de-duplicated list.
Frequently asked questions
Does it keep the original order?
Yes. Surviving lines stay in the order they appeared. Keeping the last occurrence keeps each line at the position of its last appearance.
How does case-insensitive matching work for other languages?
Lines are compared after locale-aware lowercasing, so É and é match. Arabic has no letter case, so the option has no effect on Arabic text.
Can it remove duplicates within a line, like repeated words?
No, it compares whole lines. Use find and replace with a regular expression for repeated words.
How many lines can it handle?
Up to 2 million characters, typically hundreds of thousands of lines, in a fraction of a second, because it uses a hash set rather than comparing every pair of lines.
Related tools
- Sort lines of textSort lines A to Z or Z to A, in natural numeric order, by length, reverse them, or shuffle them randomly.
- Remove extra spaces and blank linesTrim lines, collapse repeated spaces, remove or collapse blank lines, convert tabs and spaces, and strip invisible characters.
- Compare text (diff checker)Compare two texts line by line or word by word and see additions and deletions side by side or inline.
- Find and replace textFind and replace text, as plain text or with regular expressions (capture groups, flags), with a count of replacements.
- Word counter and character counterCount words, characters (with and without spaces), sentences and paragraphs, estimate reading and speaking time, and see keyword density.