Comments on: Why paging needs a lot of performance

By: Leo Büttiker

Leo Büttiker — Sun, 13 Apr 2008 11:03:50 +0000

Well that’s true Mark. But it’s a question if you realy need this pages then, if they’re not vistited by bots they might be also not visited by any humans (’cause usualy they do not found time to click trough 1000 pages)

By: Mark

Mark — Mon, 31 Mar 2008 08:19:28 +0000

You *could* also fiddle with the robots.txt file, to ward off the problems with an over-eager googlebot and his dastardly cousins

By: Leo Büttiker

Leo Büttiker — Fri, 25 Jan 2008 07:05:51 +0000

Doing to many counts is evil, allways! And with no prober caching strategie you run into this very easy, no mather wich db you use. Well MySQL’s SQL_CALC_FOUND_ROWS make *erhm* running easier.

By: cms

cms — Tue, 15 Jan 2008 10:31:56 +0000

You write - count is evil.. it’s not true.. yeah, maybe in mysql and funny SQL_CALC_FOUND_ROWS but in postgresql you don’t found it

By: Leo Büttiker

Leo Büttiker — Mon, 14 Jan 2008 15:11:41 +0000

@Quinton: ORDER BY/GROUP BY with LIMIT is slow because it can’t take profit from the LIMIT offset.
@Freddy: Thanks I get it now. I have to try it out.
@Bertrand: Yes, querying a database is slow! The point of this article is that ORDER BY/GROUP BY with LIMIT my causes very slow queries. You see this faster in queries with fulltextsearch, but it’s the same for fields with indexes, but not in that dramatic way. As long as you have no problems with doing paging, great! But when you run into performance issues you probably have to rethink how unusable my tips are.

By: Bertrand Mansion

Bertrand Mansion — Mon, 14 Jan 2008 08:02:44 +0000

Sure. And using a database will cost some performance as well…

Optimization has to be done on the database level, some databases are better than others in this regard. I don’t buy your arguments. Obviously you are mixing problems related to your fulltext search with problems related to paging, ordering and indexes. And the solutions you provide are all but usable in real web applications.

In the end, I wonder what is the point of this article. Querying a database is slow?

By: Freddy

Freddy — Thu, 10 Jan 2008 19:25:27 +0000

I presume gb.id is the unique ID for every gb-entry.
If you do “SELECT gb.*, [...] WHERE gb.uid_to=’22152′
AND visible=’1′ LIMIT 0,10;” for page 1 you might probably do something like “SELECT gb.*, [...] WHERE gb.uid_to=’22152′
AND visible=’1′ LIMIT 2000,10;” or “[...] LIMIT 10 OFFSET 2000″ für page 200 which performs VERY bad in MYSQL (especially if you use ORDER BY gb.id which you should to assure the right order).
Its much faster to do a “SELECT gb.id FROM [] ORDER BY gb.id DESC LIMIT 2000,1″ first (if and only if there is a key on “id”) and do “SELECT gb.*, [...] WHERE [...] AND id < [value from query 1] ORDER BY id DESC LIMIT 10;”. Query 1 only uses the index and is very very fast. Query 2 is the expensiv one doesn’t care anymore about the gb-entrys from the first 200 pages (because their ID is higher).

I hope its more clear now.

By: Pourquoi la pagination coûte autant de performances

Pourquoi la pagination coûte autant de performances — Thu, 10 Jan 2008 09:35:09 +0000

[...] Why paging needs a lot of performance (0 visite) [...]

By: Quinton

Quinton — Tue, 08 Jan 2008 18:32:50 +0000

I’ve just recently experienced this very issue.

SQL_CALC_FOUND_ROWS is going to be problematic when paginating large datasets. Reason being MySQL is required to evaluate all the rows in the table even though you may limit results returned to say 10. See the mysqlperformanceblog.

In this case I agree use COUNT(*) as separate query for determining the total no. of rows matched by a LIMITed query. At least on subsequent pages the query which performs the COUNT(*) can be query-cached.

But you havent figured out the real reason why ORDER BY/GROUP BY clauses affect performance so severly.

You need compound indexes. If you really really dont need an ORDER BY use ORDER BY NULL to negate the implicit ORDER BY created when performing a GROUP BY

By: Leo Büttiker

Leo Büttiker — Tue, 08 Jan 2008 07:17:32 +0000

@Trophaeum: Cool idea!
@Freddy: Yes, indexes are very helpfull. Specialy when you can use them for the order as well. Sometimes deterministic is not so important, think about search results or lists that are mainly for googlebot indexing. I’m not 100 percent sure what you want to do with spliting queries, “ORDER BY id” and LIMIT will only work on very simple queries (and unfortunatly we don’t have anymore simple queries