Comments on: Round Robin Data Storage in MySQL

By: Martin

Martin — Mon, 18 Aug 2008 17:21:32 +0000

Agree to Lucas… With REPLACE the DBMS has to check the keys and, if a matching value is found, it will execute a DELETE statement and then the according INSERT statement.

By: leo

leo — Sat, 19 Jul 2008 11:14:56 +0000

Well it’s mainly not about scaling. It’s about storage. With this you reduce the amount of data, which will prevent you from scaling for some time. Sure on the same time you also lose data, so talking about data mining, you will not have data to digg anymore.

If you intrested in spliting databases with MySQL you might be interested in DB Slayer, MySQL Proxy or in MySQL’s own Partitioning feature. Although if it comes to the mining itself you might still write your own code.

By: James

James — Sun, 13 Jul 2008 12:23:09 +0000

Doubtful this would scale well. Particularly for data mining queries later on.

I’m looking into scaling strategies right now and the one I’m thinking of is literally spitting data across dated databases or tables. Depending on your storage requirements having data split across multiple databases would allow for manageable clusters of servers each with dedicated processing power for data mining. The cost is typically financial (minimal with something like EC2) and code (locally working out which database (and possibly connection) and table to write to and later read back from.

By: Lathrop Preston

Lathrop Preston — Tue, 01 Jul 2008 15:04:49 +0000

I think in the application being discussed here for archive data all you really need would be maybe a weekly or monthly roll-up count. So that could still be done somehow.

By: Lukas

Lukas — Mon, 23 Jun 2008 16:34:41 +0000

You definately want to use INSERT ON DUPLICATE KEY UPDATE instead of REPLACE. With REPLACE you will dn an insert and then a delete is issued in case of a constraint violation. With on duplicate key update you will do the insert, it will detect the constraint violation and will do an update instead. Since the row size will remain the same, there is no need for any page shuffeling to be done. With the REPLACE you are poking wholes into your data pages, which will require frequent OPTIMIZE TABLES in order to ensure that you are getting as much data as possible in a single IO cycle.

I have an example of a fixed length queue implemented using on DUPLICATE KEY UPDATE in one of my recent talks:
http://pooteeweet.org/files/phpquebec08/sql_un_patterns.pdf

By: Round Robin Data Storage in MySQL - Tilllate Techblog - To take away

Round Robin Data Storage in MySQL - Tilllate Techblog - To take away — Mon, 23 Jun 2008 10:42:55 +0000

[...] Robin Data Storage in MySQL – Tilllate Techblog juin.23, 2008 in Actualités via Planet PHP par (author [...]

By: leo

leo — Mon, 23 Jun 2008 07:20:33 +0000

We also tried stuff with “REPLACE INTO”. At the begining the table is empty (or at least there’s no record for a certain user). So the “SELECT” will not return a record and it does fail

By: tomatenschaf

tomatenschaf — Sun, 22 Jun 2008 22:21:54 +0000

I think, the following should also work:

REPLACE INTO member_gold_visitor_rrd (uid, last_update, visitor_uid, sequence_id) SELECT 5, NOW(), 1400254, ((`sequence_id`+1) % 5))
FROM member_gold_visitor_rrd
WHERE uid=5
ORDER BY last_update DESC
LIMIT 1

or:

REPLACE INTO member_gold_visitor_rrd (uid, last_update, visitor_uid, sequence_id) SELECT 5, NOW(), 1400254, ((max(`sequence_id`) % 5)+1)
FROM member_gold_visitor_rrd
WHERE uid=5

(not testet, but I would try these)

By: Eran Galperin

Eran Galperin — Sun, 22 Jun 2008 21:42:40 +0000

The problem is that in most cases you’d want to preserve historical data for statistical purposes. So I don’t see a way to avoid a CRON

By: Dennis J.

Dennis J. — Sun, 22 Jun 2008 20:57:24 +0000

Forget that. I misinterpreted the ORDER BY.