Finding Missing Values In A MySQL Table

19th September 2008 - 2 minutes read time

Note: This post is over a year old and so the information contained here might be out of date. If you do spot something please leave a comment and we will endeavour to correct.

If you have a table of incremental values it can be hard to find out which ones are missing. The only solution might be to write a script to get all the data from the database and see which ones are missing. However, there is a way of doing this without using a script.

Using a standard select query like this:

SELECT * FROM TABLE;

Gets the following data:

We can see that values are missing, but which ones? The following query will show us where the gaps are in the data of the table.

SELECT t1.id+1 AS Missing
FROM TABLE AS t1
LEFT JOIN TABLE  AS t2 ON t1.id+1 = t2.id
WHERE t2.id IS NULL
ORDER BY t1.id;

Produces the following result.

However, this only tell us where the gaps are, not how long they are. To get the range of where the gaps from and to we need to do something a little more complex.

SELECT
t1.id+1 AS 'Missing From',
MIN(t2.id) - 1 AS 'To'
FROM TABLE AS t1, TABLE AS t2
WHERE t1.id < t2.id
GROUP BY t1.id
HAVING t1.id < MIN(t2.id) - 1;

This query gives the following result.

Missing From	To
2	2
4	9
11	22

Using this dataset we can figure out where the gaps in the data are and perhaps do something with them.

MySQL

Comments

Permalink

Nice Work! My MySql keeps pulling up an error on the last SQL statement. There error is: #1054 - Unknown column 't1.id' in 'having clause' Versions are: phpMyAdmin - 2.11.5.2 MySQL client version: 5.0.45 Any ideas?

Permalink

Odd this... It works in MySQL query browser, so I thought it must be an issue with phpMyAdmin. And it was! For some reason it has a problem with the table.column reference in the having command. I found the solution was to add quotes around the name, like this:

HAVING 't1.ID' }
 
Let us know if that works!

Permalink

Great example! Thanks!

Permalink

Very nice. I tweaked your example a little to find records that are missing by date

CREATE TABLE IF NOT EXISTS `sequence` (
`id` BIGINT NOT NULL AUTO_INCREMENT ,
`date` DATETIME NOT NULL ,
PRIMARY KEY (`id`)
)
ENGINE = InnoDB;

# missing 10/2, 10/12, and 10/13

INSERT INTO sequence (date) values ('2011-10-01 00:00:00');
INSERT INTO sequence (date) values ('2011-10-03 00:00:00');
INSERT INTO sequence (date) values ('2011-10-04 00:00:00');
INSERT INTO sequence (date) values ('2011-10-05 00:00:00');
INSERT INTO sequence (date) values ('2011-10-06 00:00:00');
INSERT INTO sequence (date) values ('2011-10-07 00:00:00');
INSERT INTO sequence (date) values ('2011-10-08 00:00:00');
INSERT INTO sequence (date) values ('2011-10-09 00:00:00');
INSERT INTO sequence (date) values ('2011-10-10 00:00:00');
INSERT INTO sequence (date) values ('2011-10-11 00:00:00');
INSERT INTO sequence (date) values ('2011-10-14 00:00:00');
INSERT INTO sequence (date) values ('2011-10-15 00:00:00');
INSERT INTO sequence (date) values ('2011-10-16 00:00:00');
INSERT INTO sequence (date) values ('2011-10-17 00:00:00');
INSERT INTO sequence (date) values ('2011-10-18 00:00:00');
INSERT INTO sequence (date) values ('2011-10-19 00:00:00');
INSERT INTO sequence (date) values ('2011-10-20 00:00:00');

mysql> SELECT
-> DATE_ADD(t1.date, INTERVAL 1 DAY) AS 'Missing From',
-> DATE_ADD( MIN(t2.date), INTERVAL -1 DAY) AS 'To'
-> FROM sequence AS t1, sequence AS t2
-> WHERE t1.date GROUP BY t1.date
-> HAVING t1.date +---------------------+---------------------+
| Missing From | To |
+---------------------+---------------------+
| 2011-10-02 00:00:00 | 2011-10-02 00:00:00 |
| 2011-10-12 00:00:00 | 2011-10-13 00:00:00 |
+---------------------+---------------------+
2 rows in set (0.00 sec)

Permalink

That's awesome! I'm sure I'll have a use for that one day.

Thanks for your input Andy :)

Add new comment

Related Content

Case Insensitive Like Searches In MySQL

I needed to create a query that did a case insensitive search using the LIKE command in MySQL and I quickly realised that in order to do this I would need to alter both the parameter and the table data to be the same case. This can be done by using the MySQL UPPER() command on the table data and the strtoupper() PHP function on the input data.

Connecting To A Non Standard MySQL Socket In PHP

Connecting to a MySQL database in PHP is usually not a difficult thing to do, in fact it is one of the first things that many tutorials will go though. However, there are certain curcumstances that require more information than the standard host, password and username details. A good example of this is when connecting to a local MySQL server through a different (or at least non standard) socket. The normal place for the MySQL socket to be on a Linux install is /var/run/mysqld/mysqld.sock, but some hosts might change this.

Count Number Of Characters In A String With MySQL

Today I needed to grab some data from a table where there was more than one occurrence of a string within another string. Basically, I needed to find all URL's from a table that were more than 3 levels deep (i.e. with 3 slashes), but realised there wasn't a function to do this in MySQL. I found an alternative method, but it got me thinking on how that might be possible.

MySQL Order Table By Character Length

As part of debugging a bit of code I needed to know the longest possible field lengths that a record contains. You might need to know this if you are performing a database migration. The following query returns a field, along with the length of the string, and orders the results by the number of characters in that string.

Backup MySQL Database PHP Script

There are quite a few scripts available on the Internet that allow you to dump data from a database into a format that can be used to replicate that database structure elsewhere. The following function is my take on this commonly occurring script.

Starting And Stopping MySQL Using Windows .bat Files

I use my PC for a lot of different things, and I don't necessarily need to have MySQL server running all the time, especially when I want to play a game. So I wondered if there was a simple way in which I could start and stop the server using a simple .bat file.

I had a look at the MySQL website and found a page that details how to start MySQL from the command line. This page suggested that I use the following command.