Starlight Tools

Duplicate Line Remover

Remove repeated rows from lists and text while preserving order. This free tool needs no upload or signup: everything is processed locally in your browser.

Remove or find repeated lines

One complete item per line

Your source text is never changed when you process it.

0 characters · 0 lines Enter text, load the sample, or import a .txt file.

Import a local .txt file up to 10 MB. The file is read only by your browser and is not uploaded.

Copy or download the processed result

Output retains the chosen source line exactly; comparison settings do not rewrite it.

0 characters · 0 lines Nothing to copy or download yet.
Removal mode
Advanced matching, blank-line, and sorting options
Matching mode
Blank lines
Output order

Exact match is the default. Ignore case and trim surrounding whitespace change comparison keys only, not retained text. Sorting is applied after duplicate and blank-line handling, so it deliberately changes output order.

Live duplicate analysis

Counts update with the input and current matching and blank-line rules. They are not announced on every keystroke.

Total lines0
Nonblank lines0
Distinct lines0
Duplicate occurrences0
Duplicate groups0
Potential reduction0%

Add input to see repeated values and occurrence counts.

Local processing: text and imported files stay in your browser.

No storage or account: this page does not save your content or require signup.

Original text retained: matching changes comparisons, not capitalization or whitespace in the chosen line.

Large lists: performance depends on browser memory; very large inputs may briefly reduce responsiveness.

How to remove duplicate lines

Paste or import line-based text, choose a removal mode, then select matching, blank-line, and output-order rules. The live analysis shows what matches before you process. Select Process lines, verify the separate output, and then copy or download it.

How matching works

Each physical line is one complete value. Exact matching compares every character and is case-sensitive. Ignore case and trim surrounding whitespace can group visually similar lines, but they affect only the internal comparison key. The first or last retained source line is returned unchanged. Keep last follows the order in which the retained last occurrences appeared.

Sorting and blank lines

Blank means an empty line or a line containing only whitespace. You can remove all blanks, keep a single blank, or preserve every blank. Sorting happens after duplicate and blank-line processing. A–Z and Z–A use your browser's text collation; numeric sort reads a number at the start of each line and places nonnumeric lines afterward.

Duplicate line examples

Ignore case: keywords

Input
Blue Widgets
blue widgets
widget guide
Keep first
Blue Widgets
widget guide

Trim for comparison: URLs

Input
https://example.com
  https://example.com  
https://example.org
Keep first
https://example.com
https://example.org

Keep last: SKUs

Input
SKU-204
SKU-105
SKU-204
Keep last
SKU-105
SKU-204

Keep one blank: logs

Input
INFO started


WARN retry
Output
INFO started

WARN retry

Practical line-matching guidance

  • Complete lines: the tool compares newline-separated values, not individual words within a line.
  • Order: Keep first and Keep last are stable when Preserve original order is selected; sorting intentionally changes that sequence.
  • Unicode: accented letters, emoji, and other Unicode text are supported. Visually identical characters with different Unicode encodings may still compare differently.
  • Line endings: Windows, macOS, and Unix line endings are accepted; output uses standard line-feed separators.
  • CSV limits: complete physical rows can be compared, but quoted CSV cells containing newlines are not parsed as single records.
  • Different tasks: duplicate-line removal does not remove repeated words inside a line or identify fuzzy, misspelled, or semantically similar near-duplicates.

Frequently asked questions

Is my text uploaded or stored?

No. Processing happens locally in your browser. This page does not upload or store the text you enter, and no account is required.

Does the tool preserve line order?

Yes, when Preserve original order is selected. Keep first retains the order of first appearances; Keep last retains the order of the last appearances. Any sorting option deliberately changes the output order after duplicate handling.

What is the difference between keeping the first and last occurrence?

Keep first returns the first line in each matching group. Keep last returns the last line in each group. The retained line keeps its original text and capitalization because matching options change comparison keys, not output text.

How are blank lines handled?

Choose Remove all blank lines, Keep one blank line, or Preserve blank lines. Lines containing only spaces or tabs count as blank. Preserved blank lines remain separate and are not deduplicated.

Is duplicate matching case-sensitive?

Exact matching is case-sensitive by default, so Apple and apple are different. Turn on Ignore case to place them in the same duplicate group while retaining the original capitalization of the chosen line.

Does trimming whitespace alter the output?

No. Trim surrounding whitespace affects comparison keys only. Leading and trailing spaces remain in the retained output line.

Can I use this with CSV rows?

You can paste CSV text and compare complete physical lines, but this is not a CSV parser. Quoted fields containing line breaks span multiple physical lines and may not be handled as one record.

Can the tool process large inputs?

It can process large lists, but the practical limit depends on your device, browser, line length, and available memory. Very large files may briefly make the page less responsive, so split them into smaller parts if needed.

Does it find near-duplicates or spelling variations?

No. It compares complete lines using only the selected case and surrounding-whitespace rules. It does not use fuzzy matching, correct spelling, or detect semantically similar lines.

Does it work on mobile devices?

Yes. The Input and Output areas stack on narrow screens, actions use touch-friendly controls, and advanced matching and sorting options can be expanded when needed.

Which browser limits apply to file import and clipboard actions?

Text file import uses your browser's local file reader and is limited here to .txt files up to 10 MB. Clipboard copy or paste may require permission and a secure browser context.

Explore more tools