Finding all those sneaky log files in Coldfusion

At some point you will have to troubleshoot server problems. Trust me - it will happen. It's no good sticking your head in the sand or your fingers in your ears (or both - tricky, but doable). Your first question will likely be, "Where do I find out information about what is happening on the server." Fortunately there are a great many log files in Coldfusion. Unfortunately they are not easy to find, and they contain different formats of information. There are 3 "groups" of file based logs on a Coldfusion MX installation. These groups are mostly the same for CFMX 7 as they are for CFMX 6 and CFMX 6.1 (a few exceptions). Here's a rundown:

1. /CFusionMX/logs - Coldfusion Logs

Actually, most CF Folks already know about this location. It contains the following logs:

• Application Log - This is a record of nearly all the errors that are thrown by the server. I say nearly because errors thrown by onSessionEnd() in the application.cfc file are not recorded here for some unknown reason (see this entry). Errors generated by Flash using Flash remoting are not recorded here either. The errors are in a single line comma delimited text qualified format. They look like this:

  "Error","jrpp-17","06/15/05","14:14:56","ap_v1","Variable undefined... "
  "Error","jrpp-17","06/15/05","14:15:48",,"File not found: c:\..."
  "Error","jrpp-14","06/15/05","14:16:37","ap_v1","Error Executing..."
  "Error","jrpp-22","06/15/05","14:19:41","ap_v1","Element undefined...
  

  The jrpp-N identifier is the processid or thread number. The date and time followed by the name of the application (if germane) and the error message. The file path is also included, which is why you should take pains to protect this file from prying eyes.
• Exception Log - This log is a more complete version of the application log. It includes Flash remoting errors on a CFMX 6 server (but still no "onsessionEnd()" errors on a cfmx 7 server). This file includes the stack trace of the error in question. Using the date, time and the jrpp (thread or process id) you can get more specific detail regarding errors in the application log. I use this log when the error thrown is too generic to be of use. Sometimes the Java stack trace will provide additional clues. this log resembles the following:

  "Error","jrpp-2134","06/08/05","10:38:00",,"File not found..."
  coldfusion.runtime.TemplateNotFoundException: File not found: /members.cfm
  at coldfusion.filter.PathFilter.invoke(PathFilter.java:83)
  at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:50)
  at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:52)
  ....
  

• Server Log - This log contains the start up routine detailing the start of all the services related to CF. It's really only useful when you are troubleshooting specific services (like the graph server for example). The one exception is that this is the location that CF logs slow pages. If you set the parameter to log pages that are slower than N number of seconds you will see warning messages here that say "blah.cfm" took longer than N seconds.
• Mail Log - This log contains errors that are reported from sending mail. You will see things like "invalid address" posted here. You can adjust the level of mail logging in the CF Administrator.
• MailSent Log - If you set CF to log all sent mail you will see the messages logged here. The messages are logged with the TO, FROM and SUBJECT.
• Flash Log - This log (as far as I can tell) contains errors generated by server side flash compiling, like the chart engine or flash forms (as of CFMX 7). in CFMX it also contains errors generated by RMA (flash remoting), but those are in a separate log in CFMX 7. If your information is different or more specific, please correct me on this one.
• Flash-remoting (cfmx 7) - This log looks very much like the exception log except it contains errors that are generated by flash remoting.
• Eventgateway Log (cfmx 7) - If you have the enterprise version of CMFX 7 you will see this log. It contains errors and exceptions fired by the event gateway. If you are on standard version you will see a message stating that your license is invalid - it means the license for the gateway.
• Additional Gateway Logs (cfmx 7) - If you are running gateways you will see a log for each one listed here. You may also see a gateway specific exception log.

2. /CFusionMX/runtime/logs - Jrun Logs

This only applies if JRUN is your underlying Java Ap server (as in the standard edition or Enterprise w/ JRUN). In this folder you will see at least 3 logs. They are named for the name of the "server" or "servers" located in the /runtime/servers/ folder. The naming convention is "*servername*-err.log (or -out.log or -event.log etc). The server name is actually the name of a folder inside of the "servers" folder that contains configuration files for the server in a SERVER-INF subfolder. This is a byproduct of convoluted Java folder management, and one of the frustrating things about Java (where to find everything). So, for example, if there were 2 servers in the /runtime/servers/ folder called "server1" and "server2" you could expect to see "server1-out.log" and "server2-out.log" in the /runtime/logs folder.

2 hints for those of you running standard version CF. CFMX 6.x uses a server titled "default" so you will see "default-err.log", while CFMX 7 uses a server titled (more appropriately) "coldfusion" so you will see "coldfusion-err.log". The information placed in these logs comes from the JRUN server. It may or may not reflect errors in the coldfusion log, but it is usually lower level information, useful in troubleshooting the JVM and server level performance issues

• *servename*-err.log - You will find socket connection errors, compiler errors and other non-specific CF errors in this log. If you had a server crash, it's a good place to look for symptoms. The format is not text qualified.
• *servername*-event.log - You can actually edit the jrun.xml file to "split out" the various events into separate files. These events may or may not be error events. This log contains startup and initialize events, Warnings and other server level info.
• *servername*-out.log - This is the Java "standard output" log. Basically, anything that needs a home but doesn't tie to a specific log goes here. That makes the information useful, but the format is unpredictable. There are CF errors logged here as well - perhaps because the "out" logger isn't aware of the CF logging operations. I'm not sure.
• *servername*-metrics.log - To see this log you must specifically enable metrics (see this excellent how-to by Brandon Purcell). There is a whole group of metrics you can log. Much of the information is similar to what you might find using the performance monitor or CF Stat - things like threads, idle time and memory usage. It's a good way to monitor if you are experience outages at regular intervals.

3. /CFusionMX/db/slserver52/logging

This one is more obscure and not often looked at. This is logging by the sequelink ODBC server. If you are using the ODBC socket driver or any Access DB's on your CF installation you might find useful information here. On a CFMX 7 (or CFMX 6.1 with ver. 5.4 drivers) the path is /CFusionMX7/db/slserver54/logging. You will see 2 types of files here - log files and "dmp" files (dumps). The log files have errors generated by the sequelink server and the dmp files are created when there is an exception resulting in a crash. I have yet to find the .dmp files useful. Usually there is stack trace and an access violation - not much more.

Obviously, in addition to these there are loggers you can set up and customize on your own. There are also the OS logs (event viewer for example) and the web server logs. After knowing where all the information is located the real issue becomes sifting through it. On that note I'll give this piece of advice. If you are troubleshooting a crash, go to each log and extract the information that ranges from 20 minutes before the crash to the recovery (reboot or whatever). Don't draw any conclusions or concoct any theories until you know everything that was going on at that point in time on the server. Then, create scenarios that you think are possible causes and try to figure out a way to test them or (worst case) trap them the next time they occur.

One of the biggest errors I see in troubleshooting is folks who start with 1 piece of information and as soon as they have a guess they go with it. This produces shallow solutions that often do not take all the variables into account. When you are dealing with a crash or performance sometimes having patience is the hardest thing to manage - but it can be your greatest asset.