A few weeks ago Mark Drew wrote a series of blog entries on the topic of performance. In fairness to Mark, he does state more than once that his primary goal was to compare the relative performance of creating structs versus CFCs versus Java objects. However, it's the comparison of various CFML server engines in two of his blog entries that caught many people's eyes, including mine, and that's what I'd like to comment on here.

As I commented on one of Mark's blog entries, the type of "loop 10,000 times" style of micro-benchmarks he used in his testing are seriously flawed. It's very difficult to draw any broad conclusions or make meaningful generalizations from the results of such tests. Among the problems with these types of tests are:

• Micro-benchmarks can make differences that are small and insignificant seem large and important, to an arbitrary degree. Let's assume the difference between two operations is one-tenth of a millisecond; with 10 iterations the difference is one millisecond--insignificant; with 100 iterations the difference is 10 milliseconds--still no big deal; but with 10,000 iterations the difference is 1,000 milliseconds--whoa, that's a big difference! Want to make the difference seem even larger? Just run more iterations. The question is: how many of these operations do you realistically expect to perform during execution of a single request?

• Micro-benchmarks don't reflect real-world applications. They artificially isolate individual operations and don't give you any context about the relative importance of these operations versus what else might be happening when you execute a request. For example, accessing an external resource--such as a database, or web service, or LDAP server, or mail server, etc.--is going to hundreds or thousands of times slower (or more!) than executing any tag or function that doesn't access an external resource.

• Micro-benchmarks don't reflect the real-world execution environment. Most developers use single-processor computers (though this has been changing recently), and the micro-benchmarks are usually executed within a single request. In the real-world, most web servers are multi-processor machines, and the most important question is: how does it behave under load? That is, what does the performance look like when executing multiple requests simultaneously?

The bottom line is that micro-benchmarks can cause you to focus on trivial or insignificant items, and can produce misleading results that don't translate into the real-world. Micro-benchmarks are like parking your car in the driveway and revving the engine while it's in neutral to see how fast it'll go. Sure, you can measure the maximum RPMs, which will give some indication of how fast the car might go, but it doesn't tell you what's really going to happen when you put it into gear and head out onto the highway.

In order to illustrate these points, we created some tests that we think are more meaningful (but which still have some serious limitations, as I'll discuss below), to see what results we'd get. Basically what we did was modify one of Mark's original tests to remove the nested loops and instead execute the CreateObject function a fixed number of times within a loop:

For completeness, here's what Person.cfc looks like:

Then we used a commercial load testing tool (Microsoft Application Center Test) configured to run three simultaneous clients with no wait time between requests so there were always three requests executing simultaneously. We ran successive test sessions at increments of 100--first 100 CreateObject calls, then 200, then 300, etc., up to 1000--and measured the average response times for one-minute test durations.

The test server has dual-CPU 3.0GHz Intel Xeon processors with hyperthreading, 1.0GB of RAM, and is running Windows Server 2003 and IIS 6.0 (that is, it's a pretty typical modern web server). We ran the tests with CFMX 7.0.2, BlueDragon Server JX 6.2.1 and 7.0 beta2, BlueDragon.NET 6.2.1 and 7.0 beta2, and just for fun, last night's developer build of BD JX and BD.NET 7.0 (we've done some performance enhancements in the the month or so since BD 7.0 beta2 was built and I was curious to see how it did).

Here are the results, which are quite different from Mark's micro-benchmarks (click to get a larger image):

What can we conclude from these results? For one thing, if you relied on Mark's micro-benchmarks to conclude that BlueDragon is slower than CFMX for CreateObject calls, you'd be wrong. However, I'd caution against generalizing too broadly from these results, because these tests contain some (but not all) of the same flaws as the micro-benchmarks.

For example, while you may have an application that creates 100, or 500, or even 1000 CFCs via CreateObject, are they really going to be all of the same type? Probably not. And the Person.cfc from Mark's tests is pretty simple--what happens if you're creating CFCs that contains dozens of functions that contain lengthy or complex logic of their own instead of just two simple setters? How will that affect performance?

Further, I only had three simultaneous clients running (though with no wait times, this is a higher load that you might think at first). What happens if this is higher? or lower? or if we add wait times?

My main point is that the only meaningful performance testing is testing of your application in your production environment under traffic conditions that you reasonably expect to see in the real world. Be skeptical of any artificial performance benchmarks--especially micro-benchmarks--and always look at them with a critical eye with the questions: "How generalizable are these results?" and "How do these results relate to the real world?".

There's only one way for you to know which CFML server is the fastest for you and your application: test them yourself.

One final point. We do all of our performance testing of BlueDragon on multi-CPU servers using external load testers to generate various levels of traffic. Any claims we make regarding BlueDragon performance are based on these types of tests, and testing of real-world customer applications, and not on micro-benchmarks nor any testing that relies on the server measuring itself (as is typically done with micro-benchmarks).

Joel Spolsky just posted an interesting article on Language Wars. I tend to agree with his conclusion:

"...the bottom line is that there are three and a half platforms (C#, Java, PHP, and a half Python) that are all equally likely to make you successful, an infinity of platforms where you're pretty much guaranteed to fail spectacularly when it's too late to change anything (Lisp, ISAPI DLLs written in C, Perl), and a handful of platforms where The Jury Is Not In, So Why Take The Risk When Your Job Is On The Line? (Ruby on Rails)."

Of course, I'd want to add CFML to the list of programming languages that "are all equally likely to make you successful." But, he's talking about "platforms", not "programming languages" (though he does seem to confuse the two somewhat), and the two leading CFML programming language implementations are on two of the three-and-a-half platforms he mentions--Java (Adobe CFMX and BlueDragon) and .NET (BlueDragon). Choosing CFML allows you to also choose either Java or .NET as your platform.

This comment about why you shouldn't use Ruby also caught my eye: "...it's known to be slow, so if you become The Next MySpace, you'll be buying 5 times as many boxes as the .NET guy down the hall." Ironically, that's almost exactly what really happened to MySpace, who started with ColdFusion and then quickly switched to .NET and BlueDragon to improve performance and reduce the number of servers they'd have to buy.

There's a very interesting article on TheServerSide.com ("Your Enterprise Java Community") about the reasons why Java is succumbing to .NET within one organization ("a federal government contractor outside Washington, DC"). For someone who has "been certified in and built a career on Java", the author lays out a pretty good case for .NET versus Java/J2EE. Of course, since this is a Java/J2EE-based community web site, there are plenty of counter-arguments presented in the comments, as well as people who are supportive of the author's position. A good read.

As reported by CNET, in 2005 sales of Windows-based servers surpassed UNIX to take over as the sales leader for the first time. Some highlights from the article:

Computer makers sold $17.7 billion worth of Windows servers worldwide in 2005 compared with $17.5 billion in Unix servers.

Linux took third place, with $5.3 billion worth of servers sold in 2005.

Note that these revenue numbers are for hardware sales of servers running a particular operating systems, and not software sales for the operating systems themselves.

Here's an interesting BusinessWeek article about the relative decline of Java, and rise of Microsoft .NET and LAMP (Linux, Apache, MySQL, PHP). While the headline is a little sensational (which is why I borrowed it), because it's unlikely that Java is going away any time soon, it does highlight the fact that there's not one technology that's going to dominate -- we're living in a world where there are three major software technology "stacks", particularly for web development. Here are some statistics from the article:

• In North America, the percentage of developers who use Java as one of their principal programming languages declined to 47.9 in Evans' fall survey, vs. 51.4% in the fall of 2002. The same surveys show that while Java use is climbing in Asia, it's on the decline in Europe.

• Microsoft .NET usage increased to 54.1% from 40.3% in the same period in North America, and exceeded Java use in Europe and Asia.

• The use of PHP in North America grew to 36.1% this fall, from 26% in the fall of 2003. It grew almost as quickly in Europe and Asia.

I found this quote from John Loiacono, executive vice-president of Sun's software division, particularly interesting, since it's effectively an admission that .NET has caught up with Java:

Sun acknowledges that .NET is picking up steam in corporations. "There will be more than one language out there," says John Loiacono. "Plus, it's Microsoft. They'll be a player." Still, he believes Java and .NET are essentially in a dead heat and denies that .NET has pulled out into the lead.

In a world where there are three major software technology stacks, why would you take the risk of limiting yourself to using only one of those three stacks? If you only use CFMX for your CFML-based web application development, you're stuck with only using Java. If the above statistics are true, then at some point in your career (probably soon) you're going to run into a client, or customer, or employer, who's committed to one of the other three stacks--or who's using more that one stack, which is quite common in large corporations.

By using BlueDragon for your CFML-based web application development, you have the choice of either Java or Microsoft .NET (sorry, we can't help you with PHP, but we do offer a free version of BlueDragon that runs on the Linux, Apache, MySQL stack). In fact, BlueDragon is the *only* technology that allows you to build a single web application--in CFML--and then deploy that application natively on either Java or Microsoft .NET (or both!).

It's almost 2006 -- don't stay stuck in the 90's.