Huge Performance Gain in Moving ColdFusion 8 to Java 1.5

We have a reasonably large application that we have been building for the past 2 years or so, with MachII, ColdSpring and Transfer, and we moved over to ColdFusion 8, pretty much as soon as the Beta came out, and have been developing with it ever since.

To give you an idea of the size of the project, some metrics for you, we have 1064 CFCs, 329 .cfm pages, 161 configured Transfer Objects, 40 MachII
frameworks instances (we started this before modules), and 201 Tables. 
This actually results in about 22,000 .class files being generated by
ColdFusion behind the scenes.

What we began to experience is
extremely long start up times when the server was first started up. For
example with no class files saved on the server, we have a start time
of around 20 minutes before the application was responsive, and if
multiple people hit the site at once, we were looking at around 45
minutes before the application would perform usably.

With class
files saved, this dropped down to 9 minutes, but we still ran into huge
difficulties with multiple users hitting the site at once.  On top of
that, whenever we had to upload a change, clearing the template cache
so the change would propagate brought the server to it's knees, so we
were forced to restart CF with every deployment of code.

So
after much haranguing, and talking to a whole slew of people, both
Adobe, and non Adobe (you all know who you are, and thank you very
much for the time and effort you all put into helping out on this), it
eventually got narrowed down to a bottleneck in Java 6 as described here (as also reported by Sean Corfield )

So
for those of you not so familiar with Java, what does all that actually
mean?  Well, let's look at what ColdFusion does behind the scenes when
you run a .cfm or .cfc page:

  1. Checks to see if it has the Java Classes already in memory to do what you have requested – if not,
    1. CF Reads in the file you are executing
    2. Parses the CFML
    3. Converts the parsed CFML data structure, and converts it into Java code
    4. Compiled that Java code down to actual Java .class files
    5. Loads the resulting .class file into memory
  2. Executes the Java code contained in the .class file as necessary.

So
probably nothing too revolutionary in there in terms of our
understanding of the ColdFusion process, however, there is a big bottle
neck in Java 6 where the loading into memory Java classes is really really really slow (maybe I need another really there), so step 1.5 on the above processes takes a very long time to get through – and the problem only gets worse
when there are large numbers of very small classes – which in almost
any ColdFusion application, and ours in particular, there are.

There are several places in which this issue can cause serious problems -

  1. At System Start Up
    As discussed previously.

  2. Development.
    Obviously during development, files tend to change quite regularly. 
    During development of our application, it would not be strange for me
    to be spending several minutes, to the tens of minutes waiting for a small change in either a .cfm or .cfc to come through. This simply slows down the amount of work that you are able to do in any given time frame.

  3. At Run Time
    Since ColdFusion has a finite limit on the number of cached templates
    (which are just the .class files mentioned earlier stored in memory),
    it is quite likely that at some point during an application life cycle,
    part of what is stored in the template cache will get purged so as to
    make room for other Java classes that have become active.  We hit the
    same issue as we hit on step 1.5 as above, as the ColdFusion server
    slows down as it pulls in the required Java classes into memory.  This
    can result in random slowdowns in the application, which are hard to
    reproduce.

    This becomes a larger issue on shared host systems, in which a trusted
    template cache is impossible, and it is quite likely the ColdFusion
    server is constantly moving the generated Java code in and out of it's
    template cache, as multiple systems require different ColdFusion code
    to be executed.

It wasn't until we had one of those head-slap moments when a co-worker
turned around to me and said 'well.. why don't we try Java 1.5?'?

Once installed, suddenly everything started working like we wanted it to, performance wise.

Since everyone loves a pretty graph, here are some metrics on server
start up, taking a Selenium script through a series of steps through
one part of our application, so you can see the considerable difference
between the two Java Versions

Average TIme Taken

Total Time Taken

Not only has this made our production systems run incredibly fast, it
also means that we are able to upload code to the production server,
clear the template cache, clear the application cache, and we are good
to go, there is no need to restart ColdFusion for changes.

This also means I can develop without having to take a coffee break in
between code changes, which has upped my productivity as well, in fact,
I can now develop happily in Machii with the config mode set to
constantly reload, and performance is no problem at all.

As far as I am aware, this issue does not exist in the development
snapshots of Java 7, and apparently a fix is in the pipeline for 1.6,
but Sun hasn't been forthcoming about the date.

Until that time, I would suggest moving your ColdFusion 8 servers over to Java 1.5, and enjoy the speed improvements!!!

Leave a Comment

Comments

  • Michael Sharman | December 6, 2007

    Hi Mark,

    Understand that you get improvements on startup (or any CF files which haven’t been compiled to .class files) but what about after that. I mean the ‘general’ running of CF on 1.6 vs 1.5, would you see slower performance on 1.5?

  • Will B. | December 6, 2007

    I’m good at CF, but no so much on configuration management. Is there a tutorial or something somewhere that explains how to do this switching from J7 to 1.5? (Sounds like going backwards…! Someone explained a while back to me about weird version numbering, though.)

  • Mark | December 7, 2007

    @Michael
    The general run time of 1.5 vs 1.6 once classes were loaded into memory was almost identical. But, to reiterate, you don’t get the random slowdowns on 1.5 where classes are loaded back into memory at run time.

    @Will
    http://www.forta.com/blog/index.cfm?mode=entry&entry=A61BAA93-3048-80A9-EF7306C155FD29F4

  • Stefan | May 18, 2008

    Just wanted a little update about this issue. Are you guys still running 1.5 or is the problems fixed in newer versions of Java 1.6? I am running a CF system that has more 100 hits/second at peaks and about 50 hits/second in average and have so far been running the system for a long time without any glitches. But the traffic goes up and it is becoming incresingly difficult to get the system up after adding new code or similar. The site is almost exlusively cfc’s and cached objects, so when it crashes, it crashes bad…

    Thanks

  • Mark | May 19, 2008

    @Stefan – still running 1.5. The bug fix required is meant to be coming in 1.6 update 10.

  • Brian G | June 15, 2008

    1.6u10 is in beta now: http://java.sun.com/javase/downloads/index.jsp

  • John Scott | July 10, 2008

    We are experiencing the same problem with our MachII application that we want to migrate to CFMX6.1 to CF8. Looks like Java 1.6u10 is still in beta so we will wait until the release version arrives before we try it. Reverting to Java 1.5 has solved the problem for us too and we are presently using that.

  • Mark | September 10, 2008

    We’re having the same random slowness issues running CF 8.01 on Linux (Cent OS)/JBOSS/Apache with Java version 1.5.0_16. Any chance backing down to 1.5.0_14 would help us at all? Anyone else out there running CF on JBOSS? Any possibility that this would cause us any of these problems?

  • Brian | September 10, 2008

    Mark – I would instead advise you to move up to 1.6.0_10 which fixes this problem. u10 has been in beta for a loooong time and is very stable. Check my blog: http://www.ghidinelli.com/2008/08/18/java16-u10-gives-big-boost-to-modelglue-transfer-coldspring-performance

  • Pat Branley | December 18, 2008

    Mark, whats the latest on this issue ? does 1.6.0_10 solve it or is 1.5 still a good option ?

  • Mark | December 18, 2008

    @Pat -
    I would recommend moving up to the latest JVM, I believe its update 11 now. It should be far more performant that 1.5, and solves the issue with the original 1.6 JVM.

  • Aurelien | April 2, 2009

    Hello Mark -

    I’ve followed this advice by upgrading my JVM to the 1.6.0_11.

    Until this point (reproduced on two servers) all cfform type Flash doesn’t work. CFchart doesn’t appear either.

    I’ve rooled back to the original jvm, and the problem is still there.

    Any idea ?

    Thanks,
    Aurelien

  • Mark | April 3, 2009

    @Aurelien, I haven’t heard that one before. What error message are you getting?

    It may be worth running CF via a command console, so you can see if there is any debug errors there.