In trying to diagnose an Out of Memory error in a large Java application, a co-worker of mine (Chad Handrich) came across the article Analyzing a Memory Leak in Java Applications using VisualVM and used the tips in it to find a memory leak that had been causing OOM errors for quite some time.
He also found that another helpful tool is the “Performance Monitor” app that comes with Windows. To use it, create a new “User Defined Data Collector Set”. Choose the option “Create Manually”, then “Performance Counter”, then you can select “Process”, and choose all the byte-count style counters if analyzing memory usage. And choose the “java.exe” instance from the list. Once it’s created, right click the collector you made and select “Start”. It will begin polling and saving the metrics you specified.