mysqlguy.net

Using Events to manage Table Partitioning by Date: wrap-up

Submitted by jay on February 20, 2008 - 10:30am

As a follow up to the series of posts I've been making, I wanted to post what I ended up with.  Thanks to everyone who posted comments, your help was extremely useful. To recap, I have a working pair of events to add and remove partitions to this table:


create table log (
    logged datetime not null,
    id int not null auto_increment,
    text varchar(256),
    PRIMARY KEY ( logged, id )
)
PARTITION BY RANGE( TO_DAYS( logged ) ) (
    PARTITION p20080206 VALUES LESS THAN (733444),
    PARTITION p20080207 VALUES LESS THAN (733445),
    PARTITION p20080208 VALUES LESS THAN (733446)
);

The partitions are keyed off of the TO_DAYS function on the date/time of the log entries. Every day I want to create a new partition for tomorrow. It's important that we create this a day in advance so new entries at 12:00AM that day do not get a "partition not found" error. Our event to do so looks like this:


DELIMITER $
CREATE EVENT log_addpartition ON SCHEDULE EVERY 1 DAY
DO
  BEGIN
    DECLARE new_partition CHAR(32) DEFAULT
      CONCAT ('p', DATE_FORMAT(DATE_ADD(CURDATE(), INTERVAL 1 DAY), '%Y%m%d'));
    DECLARE max_day INTEGER DEFAULT TO_DAYS(NOW()) +2;

    SET @s = 
      CONCAT('ALTER TABLE log ADD PARTITION (PARTITION ', new_partition,
      ' VALUES LESS THAN (', max_day, '))');
    PREPARE stmt FROM @s;
    EXECUTE stmt;
    DEALLOCATE PREPARE stmt;
  END $
DELIMITER ;
UPDATE (2/27/2008):  I found a bug in the above event.  The 'max_day' should be "TO_DAYS(NOW()) + 2" (and not +1) because the LESS THAN on the new partition is NOT inclusive.  Sorry for any confusion this may have caused.

To remove old partitions (in this example, those more than 1 week old), we run this event:

DELIMITER $
CREATE EVENT log_removepartition ON SCHEDULE EVERY 1 DAY
DO
  BEGIN
    DECLARE old_partitions CHAR(64) DEFAULT '';
    SELECT CONCAT( 'ALTER TABLE log DROP PARTITION ', 
        GROUP_CONCAT( PARTITION_NAME ))
    INTO @s
    FROM information_schema.PARTITIONS
    WHERE   TABLE_SCHEMA=schema() AND
            TABLE_NAME='log' AND
            CREATE_TIME < DATE_SUB( CURDATE(), INTERVAL 1 WEEK )
    GROUP BY TABLE_NAME;

    PREPARE stmt FROM @s;
    EXECUTE stmt;
    DEALLOCATE PREPARE stmt;
  END $
DELIMITER ;

I can check the status of the events in the Information Schema:


mysql> select * from INFORMATION_SCHEMA.EVENTS\G
*************************** 1. row ***************************
       EVENT_CATALOG: NULL
        EVENT_SCHEMA: test
          EVENT_NAME: log_addpartition
             DEFINER: root@localhost
           TIME_ZONE: SYSTEM
          EVENT_BODY: SQL
    EVENT_DEFINITION: BEGIN
    DECLARE new_partition CHAR(32) DEFAULT
      CONCAT ('p', DATE_FORMAT(DATE_ADD(CURDATE(), INTERVAL 1 DAY), '%Y%m%d'));
    DECLARE max_day INTEGER DEFAULT TO_DAYS(NOW()) +1;


    SET @s =
      CONCAT('ALTER TABLE log ADD PARTITION (PARTITION ', new_partition,
      ' VALUES LESS THAN (', max_day, '))');
    PREPARE stmt FROM @s;
    EXECUTE stmt;
    DEALLOCATE PREPARE stmt;
  END
          EVENT_TYPE: RECURRING
          EXECUTE_AT: NULL
      INTERVAL_VALUE: 1
      INTERVAL_FIELD: DAY
            SQL_MODE:
              STARTS: 2008-02-20 07:05:05
                ENDS: NULL
              STATUS: ENABLED
       ON_COMPLETION: NOT PRESERVE
             CREATED: 2008-02-20 07:05:05
        LAST_ALTERED: 2008-02-20 07:05:05
       LAST_EXECUTED: 2008-02-20 07:05:05
       EVENT_COMMENT:
          ORIGINATOR: 1118021742
CHARACTER_SET_CLIENT: latin1
COLLATION_CONNECTION: latin1_swedish_ci
  DATABASE_COLLATION: latin1_swedish_ci
*************************** 2. row ***************************
       EVENT_CATALOG: NULL
        EVENT_SCHEMA: test
          EVENT_NAME: log_removepartition
             DEFINER: root@localhost
           TIME_ZONE: SYSTEM
          EVENT_BODY: SQL
    EVENT_DEFINITION: BEGIN
    DECLARE old_partitions CHAR(64) DEFAULT '';

    SELECT CONCAT( 'ALTER TABLE log DROP PARTITION ',         GROUP_CONCAT( PARTITION_NAME ))
    INTO @s
    FROM information_schema.PARTITIONS
    WHERE   TABLE_SCHEMA=schema() AND
            TABLE_NAME='log' AND
            CREATE_TIME < DATE_SUB( CURDATE(), INTERVAL 1 WEEK )
    GROUP BY TABLE_NAME;

    PREPARE stmt FROM @s;
    EXECUTE stmt;
    DEALLOCATE PREPARE stmt;
  END
          EVENT_TYPE: RECURRING
          EXECUTE_AT: NULL
      INTERVAL_VALUE: 1
      INTERVAL_FIELD: DAY
            SQL_MODE:
              STARTS: 2008-02-20 07:06:48
                ENDS: NULL
              STATUS: ENABLED
       ON_COMPLETION: NOT PRESERVE
             CREATED: 2008-02-20 07:06:48
        LAST_ALTERED: 2008-02-20 07:06:48
       LAST_EXECUTED: 2008-02-20 07:06:48
       EVENT_COMMENT:
          ORIGINATOR: 1118021742
CHARACTER_SET_CLIENT: latin1
COLLATION_CONNECTION: latin1_swedish_ci
  DATABASE_COLLATION: latin1_swedish_ci
2 rows in set (0.01 sec)

And finally, what does our table actually look like?


select DATE_FORMAT(logged, '%m/%d/%Y'), count(*) from log group by TO_DAYS(logged);
+---------------------------------+----------+
| DATE_FORMAT(logged, '%m/%d/%Y') | count(*) |
+---------------------------------+----------+
| 02/13/2008                      |     1103 | 
| 02/14/2008                      |     2158 | 
| 02/15/2008                      |     2878 | 
| 02/16/2008                      |     2880 | 
| 02/17/2008                      |     2880 | 
| 02/18/2008                      |     2878 | 
| 02/19/2008                      |     2880 | 
| 02/20/2008                      |      892 | 
+---------------------------------+----------+
8 rows in set (0.02 sec)

Note we have 8 days in this table because in our 'log_removepartition' event because we search for all CREATE_TIMEs that are greater than 1 week ago (not greater than and equal to).

Trackback URL for this post:

http://mysqlguy.net/trackback/13

Hi, thanks for this! I'm

Hi, thanks for this!

I'm trying to use it and noticed a couple of things:

1. There's no max value set, so I got an error the next day.

To remedy that, I did:

# 'lo' is a datetime field

alter table logs
PARTITION BY RANGE( TO_DAYS( lo ) ) (
PARTITION p20091107 VALUES LESS THAN (to_days('2009-11-07')),
PARTITION p20091108 VALUES LESS THAN (to_days('2009-11-08')),
PARTITION p20091109 VALUES LESS THAN (to_days('2009-11-09')),
PARTITION pmax VALUES LESS THAN MAXVALUE
);

 

Then I did:

DELIMITER $
CREATE EVENT logs_add_partition ON SCHEDULE EVERY 1 DAY STARTS TIMESTAMP(CURRENT_DATE+1,'00:00:01')
DO
BEGIN
DECLARE new_partition CHAR(32) DEFAULT
CONCAT ('p', DATE_FORMAT(DATE_ADD(CURDATE(), INTERVAL 1 DAY), '%Y%m%d'));
DECLARE max_day INTEGER DEFAULT TO_DAYS(NOW()) +2;

SET @s =
CONCAT('ALTER TABLE logs ADD PARTITION (PARTITION ', new_partition,
' VALUES LESS THAN (', max_day, '))');
PREPARE stmt FROM @s;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
END $
DELIMITER ;

DELIMITER $
CREATE EVENT logs_del_partition ON SCHEDULE EVERY 1 DAY STARTS TIMESTAMP(CURRENT_DATE+1,'00:00:02')
DO
BEGIN
DECLARE old_partitions CHAR(64) DEFAULT '';
SELECT CONCAT( 'ALTER TABLE logs DROP PARTITION ',
GROUP_CONCAT( PARTITION_NAME ))
INTO @s
FROM information_schema.PARTITIONS
WHERE TABLE_SCHEMA=schema() AND
TABLE_NAME='logs' AND
CREATE_TIME < DATE_SUB( CURDATE(), INTERVAL 1 MONTH )
GROUP BY TABLE_NAME;

PREPARE stmt FROM @s;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
END $
 

# The only thing I added here was the STARTS TIMESTAMP(CURRENT_DATE+1,'00:00:01')

# to tell it to run ever day after midnight.

I'm not sure this will work...it hasn't run yet.

Do you foresee an issue with having to set the MAX?

 

The remove partition event

The remove partition event doesn't work if you're having partitions with hashes.. For example, you partition by date to be able to remove old data, but the partitions itselves are partitions with hashes to speed up even more. What happens is that the remove partition events tries to execute:

ALTER TABLE mobiledata DROP PARTITION p20100318,p20100318,p20100318,p20100318,p20100318,p20100318,p20100318,p20100318,p20100318,p20100318

 

That's illegal, and I assume because we're having the same partition name several times.

So I added a simple "DISTINCT" to:

GROUP_CONCAT( DISTINCT PARTITION_NAME ))
 

and it works.

Warning!

Comment abuse is not tolerated on this site, besides all the comments are moderated, so don't bother posting comments that are not on topic, only for increasing the SEO of your site, or are outright spam.  If you've got something intelligent to contribute, by all means, post a link to your blog.  

About Me

Jay Janssen
Yahoo!, Inc.
jayj at yahoo dash inc dot com
MySQL
High Availability
Global Load Balancing
Failover
View Jay Janssen on Twitter  View Jay Janssen's LinkedIn profile View Jay Janssen's Facebook profile