mysqlguy.net

Using Events to manage Table Partitioning by Date: wrap-up

Submitted by jay on February 20, 2008 - 10:30am

As a follow up to the series of posts I've been making, I wanted to post what I ended up with.  Thanks to everyone who posted comments, your help was extremely useful. To recap, I have a working pair of events to add and remove partitions to this table:


create table log (
    logged datetime not null,
    id int not null auto_increment,
    text varchar(256),
    PRIMARY KEY ( logged, id )
)
PARTITION BY RANGE( TO_DAYS( logged ) ) (
    PARTITION p20080206 VALUES LESS THAN (733444),
    PARTITION p20080207 VALUES LESS THAN (733445),
    PARTITION p20080208 VALUES LESS THAN (733446)
);

The partitions are keyed off of the TO_DAYS function on the date/time of the log entries. Every day I want to create a new partition for tomorrow. It's important that we create this a day in advance so new entries at 12:00AM that day do not get a "partition not found" error. Our event to do so looks like this:


DELIMITER $
CREATE EVENT log_addpartition ON SCHEDULE EVERY 1 DAY
DO
  BEGIN
    DECLARE new_partition CHAR(32) DEFAULT
      CONCAT ('p', DATE_FORMAT(DATE_ADD(CURDATE(), INTERVAL 1 DAY), '%Y%m%d'));
    DECLARE max_day INTEGER DEFAULT TO_DAYS(NOW()) +2;

    SET @s = 
      CONCAT('ALTER TABLE log ADD PARTITION (PARTITION ', new_partition,
      ' VALUES LESS THAN (', max_day, '))');
    PREPARE stmt FROM @s;
    EXECUTE stmt;
    DEALLOCATE PREPARE stmt;
  END $
DELIMITER ;
UPDATE (2/27/2008):  I found a bug in the above event.  The 'max_day' should be "TO_DAYS(NOW()) + 2" (and not +1) because the LESS THAN on the new partition is NOT inclusive.  Sorry for any confusion this may have caused.

To remove old partitions (in this example, those more than 1 week old), we run this event:

DELIMITER $
CREATE EVENT log_removepartition ON SCHEDULE EVERY 1 DAY
DO
  BEGIN
    DECLARE old_partitions CHAR(64) DEFAULT '';
    SELECT CONCAT( 'ALTER TABLE log DROP PARTITION ', 
        GROUP_CONCAT( PARTITION_NAME ))
    INTO @s
    FROM information_schema.PARTITIONS
    WHERE   TABLE_SCHEMA=schema() AND
            TABLE_NAME='log' AND
            CREATE_TIME < DATE_SUB( CURDATE(), INTERVAL 1 WEEK )
    GROUP BY TABLE_NAME;

    PREPARE stmt FROM @s;
    EXECUTE stmt;
    DEALLOCATE PREPARE stmt;
  END $
DELIMITER ;

I can check the status of the events in the Information Schema:


mysql> select * from INFORMATION_SCHEMA.EVENTS\G
*************************** 1. row ***************************
       EVENT_CATALOG: NULL
        EVENT_SCHEMA: test
          EVENT_NAME: log_addpartition
             DEFINER: root@localhost
           TIME_ZONE: SYSTEM
          EVENT_BODY: SQL
    EVENT_DEFINITION: BEGIN
    DECLARE new_partition CHAR(32) DEFAULT
      CONCAT ('p', DATE_FORMAT(DATE_ADD(CURDATE(), INTERVAL 1 DAY), '%Y%m%d'));
    DECLARE max_day INTEGER DEFAULT TO_DAYS(NOW()) +1;


    SET @s =
      CONCAT('ALTER TABLE log ADD PARTITION (PARTITION ', new_partition,
      ' VALUES LESS THAN (', max_day, '))');
    PREPARE stmt FROM @s;
    EXECUTE stmt;
    DEALLOCATE PREPARE stmt;
  END
          EVENT_TYPE: RECURRING
          EXECUTE_AT: NULL
      INTERVAL_VALUE: 1
      INTERVAL_FIELD: DAY
            SQL_MODE:
              STARTS: 2008-02-20 07:05:05
                ENDS: NULL
              STATUS: ENABLED
       ON_COMPLETION: NOT PRESERVE
             CREATED: 2008-02-20 07:05:05
        LAST_ALTERED: 2008-02-20 07:05:05
       LAST_EXECUTED: 2008-02-20 07:05:05
       EVENT_COMMENT:
          ORIGINATOR: 1118021742
CHARACTER_SET_CLIENT: latin1
COLLATION_CONNECTION: latin1_swedish_ci
  DATABASE_COLLATION: latin1_swedish_ci
*************************** 2. row ***************************
       EVENT_CATALOG: NULL
        EVENT_SCHEMA: test
          EVENT_NAME: log_removepartition
             DEFINER: root@localhost
           TIME_ZONE: SYSTEM
          EVENT_BODY: SQL
    EVENT_DEFINITION: BEGIN
    DECLARE old_partitions CHAR(64) DEFAULT '';

    SELECT CONCAT( 'ALTER TABLE log DROP PARTITION ',         GROUP_CONCAT( PARTITION_NAME ))
    INTO @s
    FROM information_schema.PARTITIONS
    WHERE   TABLE_SCHEMA=schema() AND
            TABLE_NAME='log' AND
            CREATE_TIME < DATE_SUB( CURDATE(), INTERVAL 1 WEEK )
    GROUP BY TABLE_NAME;

    PREPARE stmt FROM @s;
    EXECUTE stmt;
    DEALLOCATE PREPARE stmt;
  END
          EVENT_TYPE: RECURRING
          EXECUTE_AT: NULL
      INTERVAL_VALUE: 1
      INTERVAL_FIELD: DAY
            SQL_MODE:
              STARTS: 2008-02-20 07:06:48
                ENDS: NULL
              STATUS: ENABLED
       ON_COMPLETION: NOT PRESERVE
             CREATED: 2008-02-20 07:06:48
        LAST_ALTERED: 2008-02-20 07:06:48
       LAST_EXECUTED: 2008-02-20 07:06:48
       EVENT_COMMENT:
          ORIGINATOR: 1118021742
CHARACTER_SET_CLIENT: latin1
COLLATION_CONNECTION: latin1_swedish_ci
  DATABASE_COLLATION: latin1_swedish_ci
2 rows in set (0.01 sec)

And finally, what does our table actually look like?


select DATE_FORMAT(logged, '%m/%d/%Y'), count(*) from log group by TO_DAYS(logged);
+---------------------------------+----------+
| DATE_FORMAT(logged, '%m/%d/%Y') | count(*) |
+---------------------------------+----------+
| 02/13/2008                      |     1103 | 
| 02/14/2008                      |     2158 | 
| 02/15/2008                      |     2878 | 
| 02/16/2008                      |     2880 | 
| 02/17/2008                      |     2880 | 
| 02/18/2008                      |     2878 | 
| 02/19/2008                      |     2880 | 
| 02/20/2008                      |      892 | 
+---------------------------------+----------+
8 rows in set (0.02 sec)

Note we have 8 days in this table because in our 'log_removepartition' event because we search for all CREATE_TIMEs that are greater than 1 week ago (not greater than and equal to).

Trackback URL for this post:

http://mysqlguy.net/trackback/13

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.

More information about formatting options

Warning!

Comment abuse is not tolerated on this site, besides all the comments are moderated, so don't bother posting comments that are not on topic, only for increasing the SEO of your site, or are outright spam.  If you've got something intelligent to contribute, by all means, post a link to your blog.  

MySQL

Yahoo

Recent comments

About Me

Jay Janssen
Yahoo!, Inc.
jayj at yahoo dash inc dot com

MySQL
High Availability
Global Load Balancing
Failover

View Jay Janssen's LinkedIn profileView Jay Janssen's Facebook profile

User login

Friends

Links