To give you an idea of the size of the project, some metrics for you, we have 1064 CFCs, 329 .cfm pages, 161 configured Transfer Objects, 40 MachII frameworks instances (we started this before modules), and 201 Tables. This actually results in about 22,000 .class files being generated by ColdFusion behind the scenes.
What we began to experience is extremely long start up times when the server was first started up. For example with no class files saved on the server, we have a start time of around 20 minutes before the application was responsive, and if multiple people hit the site at once, we were looking at around 45 minutes before the application would perform usably.
With class files saved, this dropped down to 9 minutes, but we still ran into huge difficulties with multiple users hitting the site at once. On top of that, whenever we had to upload a change, clearing the template cache so the change would propagate brought the server to it's knees, so we were forced to restart CF with every deployment of code.
So after much haranguing, and talking to a whole slew of people, both Adobe, and non Adobe (you all know who you are, and thank you very much for the time and effort you all put into helping out on this), it eventually got narrowed down to a bottleneck in Java 6 as described here (as also reported by Sean Corfield )
So for those of you not so familiar with Java, what does all that actually mean? Well, let's look at what ColdFusion does behind the scenes when you run a .cfm or .cfc page:
- Checks to see if it has the Java Classes already in memory to do what you have requested - if not,
- CF Reads in the file you are executing
- Parses the CFML
- Converts the parsed CFML data structure, and converts it into Java code
- Compiled that Java code down to actual Java .class files
- Loads the resulting .class file into memory
- Executes the Java code contained in the .class file as necessary.
So probably nothing too revolutionary in there in terms of our understanding of the ColdFusion process, however, there is a big bottle neck in Java 6 where the loading into memory Java classes is really really really slow (maybe I need another really there), so step 1.5 on the above processes takes a very long time to get through - and the problem only gets worse when there are large numbers of very small classes - which in almost any ColdFusion application, and ours in particular, there are.
There are several places in which this issue can cause serious problems -
- At System Start Up
As discussed previously.
Obviously during development, files tend to change quite regularly. During development of our application, it would not be strange for me to be spending several minutes, to the tens of minutes waiting for a small change in either a .cfm or .cfc to come through. This simply slows down the amount of work that you are able to do in any given time frame.
- At Run Time
Since ColdFusion has a finite limit on the number of cached templates (which are just the .class files mentioned earlier stored in memory), it is quite likely that at some point during an application life cycle, part of what is stored in the template cache will get purged so as to make room for other Java classes that have become active. We hit the same issue as we hit on step 1.5 as above, as the ColdFusion server slows down as it pulls in the required Java classes into memory. This can result in random slowdowns in the application, which are hard to reproduce.
This becomes a larger issue on shared host systems, in which a trusted template cache is impossible, and it is quite likely the ColdFusion server is constantly moving the generated Java code in and out of it's template cache, as multiple systems require different ColdFusion code to be executed.
It wasn't until we had one of those head-slap moments when a co-worker turned around to me and said 'well.. why don't we try Java 1.5?'?
Once installed, suddenly everything started working like we wanted it to, performance wise.
Since everyone loves a pretty graph, here are some metrics on server start up, taking a Selenium script through a series of steps through one part of our application, so you can see the considerable difference between the two Java Versions
Not only has this made our production systems run incredibly fast, it also means that we are able to upload code to the production server, clear the template cache, clear the application cache, and we are good to go, there is no need to restart ColdFusion for changes.
This also means I can develop without having to take a coffee break in between code changes, which has upped my productivity as well, in fact, I can now develop happily in Machii with the config mode set to constantly reload, and performance is no problem at all.
As far as I am aware, this issue does not exist in the development snapshots of Java 7, and apparently a fix is in the pipeline for 1.6, but Sun hasn't been forthcoming about the date.
Until that time, I would suggest moving your ColdFusion 8 servers over to Java 1.5, and enjoy the speed improvements!!!