Lesson 10 of 13 · 22 min

Removing duplicates and keeping one row

By the end of this lesson you can number the rows inside each duplicate group with ROW_NUMBER, keep exactly one row per group by a stated survivor rule, map duplicates to their survivor, and delete the extra rows from a copy of a table.

DISTINCT cannot deduplicate customers: the two registrations of one person differ in customer_id, signup_date and the spelling of the email, so they are distinct rows. What you need is a survivor rule: within each group of rows that describe the same thing, which one do you keep?

Common rules are the earliest record (it owns the longest history), the most recently updated record (it has the freshest contact details) or the most complete record. Whatever you choose, write it down and make it deterministic: if two rows tie on the rule, a tie-breaker (usually the id) decides, so the query gives the same answer every time.

Included with a Formula Quest membership

This lesson, its 4 graded exercises and every other Members lesson in SQL Quest come with a Formula Quest membership that includes SQL (Single Quest with SQL selected, Quest Trio including SQL, or Quest Unlimited), or with SQL access from your organization.

Free to everyone: all of SQL Foundations and Retrieving and Filtering Data, the first lesson of every track, the playground, the dataset library and the SQL reference. See free lessons