For now over three years we are working with unit tests. I’d like to share some of those experiences.

As we have two frameworks in place for our website, I can compare two different strategies for unit tests.

In our homemade legacy framework we were using SimpleTest. Mainly because I read PHP|Architect’s Guide to PHP Design Patterns and Simpletest was Jason’s framework of choice.

We have a cron job running which runs all tests every hour. The results of the tests are being shown on a page. The results are also displayed in Nagios.

(more…)

Last week, I was lucky to go to QCon 2008 in London to meet with all the big names in the software industry. This was a great opportunity to see what everyone’s working on and to get a feeling for the newest trends in software development. Here’s what my gut says about the world of software development after QCon 2008 (part 1):

(more…)

Trevi is not only a fountain in Italy, it’s our new application framework as well. We migrated our first pages to this new platform and brought them online three weeks ago. But let me explain the story of Trevi.

Another Framework?

There’re already thousands of web frameworks out there so I would sink into the ground if we really wrote another one. But serving pages for 2 million unique clients, it would also not have been a solution to just go to a shop and take the beautiful looking, nice boxed xyz framework from a big company. So my co-worker (and Trevi project lead) Maarten started a year ago to evaluate a framework that fits our needs best.
(more…)

On the view of your database the worst thing you can do in your web app is paging. Paging is horrible in the view of performance. To explain let me make a little example:

SELECT SQL_CALC_FOUND_ROWS gb.*,
 u.username,
        u.uid,
        u.geschlecht,
        u.mitfoto,
        [... some more fields...]
FROM member_gold_guestbook gb
LEFT JOIN users u ON u.uid=gb.uid_from
[... some more left joins...]
WHERE gb.uid_to=’22152′
AND visible=’1′
LIMIT 0,10;

That’s not that bad at all, but when you go to page 300 your database server will hate you for this. The database server has not only to calculate the 10 items you want to show but also all 3000 previous items.

Sure you may argue nobody will ever go to page 300. Somebody will not, but “googlebot” and his evil brothers will. And the bad thing is that you can do nothing against it, as long as you need paging. There are just a few tricks that may reduce your server load a bit.

(more…)

In many companies, management decides to setup a web proxy server to be able to control/disable the non-busines-related content. YouTube could probably be blamed for many hours spent doing non-business stuff.

Well, I use YouTube myself. Sometimes when taking a break and sometimes to listen to music that goes with the video, and you don’t have to watch it at all, but it’s probably the fastest way find and listen to a certain song if it’s available. Well, this time, YouTube will be a critical factor to finding the elegant solution.

(more…)

Consider you have a set of data that is changing dynamically for each page request and you need to cache that data the fastest way possible. You can’t cache dynamic and unpredictable data as a whole, can you? Hence, we would put each data entry into cache separately to be able to fetch it separately and dynamically. But this means bombing your cache infrastructure with with requests.

Caching Text Elements

Let’s get more concrete. To translate tilllate.com into all different languages, we use text elements (like gettext). For storage we are using MySQL and thus each text element is a row in the translation table. While this storage is very easy to maintain, it is quite silly to use in production, where you have ~100 text elements per page and peaks of 1500 page requests per second with a resulting 150′000 MySQL queries per second. Don’t even ask, we don’t do it. But even for a highly scalable memcached infrastructure, a 150′000 requests per second just isn’t easy to digest.

Let’s talk about a better solution. It consists of three concepts: Two-Tiered Caching, Incremental Caching and Cache Versioning.
(more…)

After a long day of meetings and other tedious manager work the perfect way to relax is to code. The best is a mini-projects where you see your results after an hour or so. I call these tasks “Plausch-Projekte” (”plah-oosh project” =”fun projects”).

This week my plah-oosh projects were two metric tools for Ganglia. Besides Nagios Ganglia is the main monitoring tool for our cluster. We monitor something like 20 metrics like load, memory, disk usage, network activity.

Ciprian and Stefan recently built a script to monitor apache (bytes/sec, hits/sec, idle processes etc.) via the /server-status interface. Based on their work I hacked two scripts:

ganglia_mysql_metrics.php monitors multiple mysql parameters like queries/sec, slow queries/sec, threads connected:

ganglia_squid_metrics.php reports regularly about squid metrics: Requests/sec, service time, available file descriptors:

The scripts are quick and dirty code. Procedural. Not well documented. Does only read the mcast_port from the config file and ignores the rest. But it might be a good base to be used on your cluster too. Just call them every minute via the crontab.

With 125 million page impressions a month and highly dynamic content, caching is essential for tilllate.com. At tilllate, we have worked with several different caching techniques. Before we used caching, we just pre-generated the data: A nightly cron job populates a database table or generates a file containing the expensive data. Usually expensive queries, like the Most viewed pictures.

Then we are using Cache_Lite a lot. For example our homepage: If you look at the source code of our homepage you will notice the string <!--cache id a:4:{i:0;s:1:-->. This means that the page is coming out of the cache. There are a few disadvantages of Cache_Lite:
(more…)

Writing testable code is tricky. On reason is because you have to test all of your software components individually and isolated from each other. Imagine that you’re testing your new and cool DB abstraction layer. It’s using:

• Some sort of error log
• A config component to fetch DB credentials
• Your caching layer to temporarely save stuff
• I almost forgot: It’s using PDO

Loose Coupling

Having everything tied up like this, you’re going to have a hard time testing it. After all, you don’t want to test the connected components, just your new and cool query class. That’s where Dependency Injection kicks in - along with some other patterns with cool names. The goal is to break your piece of software down to a bunch loosely coupled components and tie them together dynamically. As soon as your software is getting the Lego look and feel, you know that you’re on the right track. This is where you can use Dependency Injection for the logging component:

Does it feel Lego already? Then go on and do the same with the other 3 dependencies.
(more…)

If you’re programming php for a while you know PHP has some fancy type converting stuff. Or do you know an other programing language that can do that?

The trick is simple, if you do a mathematical operation on a string, PHP tries to convert it in a number. It does the number parsing of the string from the left to the right, as long it’s a number and stop afterwards. So converting of “12 number 34″ will you just get you 12. If the string do not start with a number PHP converts it to a 0. So be careful about 100/"bags 10".

(more…)