Two lines that look identical can be technically different
'Duplicate' sounds like an obvious, binary judgment, but text comparison has quiet gotchas that decide whether two visually identical-looking lines actually match. A trailing space after 'apple' that you can't see in a text box makes it a different string from 'apple' with no trailing space, character for character. 'Apple' and 'apple' differ only in case, but a strict string comparison sees them as completely unrelated sequences of characters. Deduplication isn't really about spotting visual repeats — it's about defining exactly what 'the same' means before comparing, and that definition is exactly what the case-sensitivity and whitespace-trimming toggles control here.
How the comparison settings change what counts as a duplicate
| Setting | 'Apple' vs 'apple' | ' Apple' vs 'Apple' (leading space) | Recommended for |
|---|---|---|---|
| Case-insensitive, trim off (defaults) | Duplicate | Not duplicate | Casual text where case and stray spaces don't matter |
| Case-insensitive, trim on | Duplicate | Duplicate | Messy pasted lists, copy-pasted data with inconsistent spacing |
| Case-sensitive, trim off | Not duplicate | Not duplicate | Code, identifiers, anything where case and exact spacing matter |
| Case-sensitive, trim on | Not duplicate | Duplicate | Case-sensitive data with inconsistent leading/trailing spaces |
Deduping your list
Paste your lines into the input area.
Toggle case-sensitive matching if you need to treat 'Apple' and 'apple' as different lines.
Toggle trim whitespace to ignore leading and trailing spaces when comparing.
Review the unique output, the removed count, and the unique count, then copy the result.
Why trimming only affects comparison, not what's kept
This is a subtle but important detail: turning on trim whitespace changes how two lines are compared for equality, but it doesn't strip whitespace from the actual output. If ' apple' (with a leading space) appears before 'apple' in your list, and trimming is on, the second line gets dropped as a duplicate — but the surviving first line keeps its leading space exactly as you pasted it, since trimming only affects the comparison step, not the kept line's content.
Why order preservation matters for real data
Deduplication keeps the first occurrence of each line and drops later repeats, preserving your original ordering rather than alphabetizing or otherwise reorganizing the list. This matters whenever line order carries information: a chronologically pasted log where you want the earliest instance of each repeated entry, a prioritized list where the first mention represents the intended ranking, or simply a list you don't want scrambled just because it needed cleaning. If your list also needs sorting after deduplication — alphabetical, numeric, or otherwise — that's a distinct operation handled by Sort Lines, applied after deduping rather than instead of it.
What happens to blank lines by default
Blank lines are removed by default, which matters for lists that accumulate stray empty lines from copy-pasting — a spreadsheet export, a list assembled from multiple sources, or text with inconsistent line breaks. If blank lines carry structural meaning in your specific list (separating sections, marking paragraph breaks), switch on 'Keep blank lines' before deduping, since the default behavior would otherwise silently collapse every blank line in the input down to at most one kept instance, or remove them entirely depending on how many exist.
Common mistakes
Leaving case-sensitivity off when deduping something where case genuinely matters — variable names, product SKUs, or codes where 'ABC123' and 'abc123' are meant to be distinct entries.
Forgetting that trim-whitespace only affects the comparison, not the output — expecting the kept lines to come out stripped of stray spaces when they won't be.
Deduping a list where blank lines are meaningful separators, without first enabling 'Keep blank lines' to preserve that structure.
Assuming deduplication also sorts the result — order is preserved from the original input, based on first occurrence, not reorganized alphabetically.
Real use cases
Cleaning a mailing list or email export that accumulated repeated entries from multiple imports.
Deduplicating a list of URLs or keywords pulled from several sources before further processing.
Merging two exported lists of names or IDs and removing entries that appear in both.
Cleaning up a pasted log or dataset with accidental repeated rows before analysis.
Frequently asked questions
Q: What order do the unique lines appear in?
A: First occurrence is kept; subsequent duplicates are dropped, and the original ordering is preserved throughout.
Q: Does case sensitivity matter?
A: By default no — 'Apple' and 'apple' are treated as duplicates. Turn on case-sensitive mode to treat them as different lines.
Q: What does 'Trim whitespace' do?
A: When on, leading and trailing spaces are ignored during comparison, but the original line — spaces and all — is still what's kept in the output.
Q: Are blank lines removed?
A: By default, yes. Turn on 'Keep blank lines' if you want them preserved as part of the output.
Q: Is there a line limit?
A: No hard limit — processing is local and handles tens of thousands of lines comfortably.
Q: I deduped my list but two lines that look identical are both still there — why?
A: Check for invisible differences: trailing spaces, tabs instead of spaces, or a case difference with case-sensitive mode enabled. Turning on trim whitespace and turning off case-sensitivity will catch most of these hidden mismatches.
Dedupe your list now
Clean up your lines with Remove Duplicate Lines. Need the result sorted afterward? Try Sort Lines. Replacing specific text across your list first? Use Find & Replace, or compare two versions of a list with Text Diff.
