The trigger: we observed JVM crashes with java.lang.OutOfMemoryError.
The goal: find a tool that can analyze large (>10 GB) heap dump files created on OOM Error, analyze them in automated way and give a binary answer whether a memory leak was detected or not.
There are several tools on the market which claim to be able to analyze the heap dump files: Shark, VisualVM CLI, JProfiler CLI / YourKit CLI, IBM Heap Analyzer, JHAT, Eclipse Memory Analyzer Tool. An additional alternative is to write own analysis tool.
The winner is Eclipse Memory Analyzer Tool.
Details
Let’s take a look at the options we have.
Shark (Android)
This tool is designed specifically for Android and it is not suitable for:
Backend services
Microservices
JVM heap dumps
Large heaps (GBs)
VisualVM CLI
Limited automation
Small heaps only
JProfiler CLI / YourKit CLI
Commercial
Very strong automation
Excellent diff engine
I work with YourKit tool frequently doing manual analysis (like heap utilization or memory leaks). It is very useful and powerful, but so far I have used it on living java processes.
This tools cannot analyze .hprof files. It can dump, create and analyze snapshot files. I did not find any tool how to convert .hprof to a snapshot.
So if we want to release automated analysis tool we have to rely on customers having JProfiler / YourKit.
IBM Heap Analyzer (HA)
Excellent for OpenJ9 (high-performance Java Virtual Machine (JVM) originally developed by IBM). Since I am using a different JVM, I decided not to test this solution
CLI automation
JHAT
It is not supported anymore.
Eclipse Memory Analyzer Tool (MAT)
MAT can check for specific class causing leak and will give true/false result for this class.
MAT can compare two hprof files.
MAT is the winner of my tool selection. It successfully passed several of my test cases. I have not used it in production environment so far.
Writing own analysis tool.
This is extremely complex and time consuming. Below some aspects of it (I compare it to the MAT).
JVM Heap Dumps Are Massive Graphs. A .hprof file contains:
Millions of objects
Billions of references
Deep object hierarchies
Mixed primitive arrays, strings, classes, threads, GC roots
Example: a 1 GB server heap can have 20–50 million objects.
To detect leaks, we must analyze all objects’ reachability. That is literally a giant directed graph.
Challenge: We need a memory-efficient graph representation — otherwise our analyzer will run out of memory before it finishes analyzing the heap.
Heap dump format is tricky as .hprof files are binary files. There are:
Multiple formats depending on VM
Object ID sizes differ per JVM (32-bit vs 64-bit)
Classes, fields, arrays, primitives are encoded differently
References vs primitive values require different parsing
So we need a robust, low-level parser. One small mistake = parser crashes on a 20 GB heap.
We need GC root resolution
Leaks are only defined relative to GC roots (things that are always reachable).
Static fields
Threads
JNI references
Classloaders
We must implement:
A system that finds all roots
Correctly identifies strong vs weak references
Tracks reachable vs unreachable objects
Otherwise, we will misreport leaks.
Dominator tree computation is non-trivial
To find retained memory (what a leak “holds”):
We must build a dominator tree of the heap graph. This involves graph algorithms on millions of nodes. Naive implementations can use hundreds of GB of RAM for large heaps. Eclipse MAT does this efficiently using memory-mapped indexes, compression, and optimized algorithms.
Rewriting it yourself = months of careful work.
We need retained size calculations
To detect leaks:
We must sum memory recursively from dominator tree. Count arrays, object headers, references correctly. Handle shared subgraphs properly
Mistakes here → false positives/negatives.
We need scalability + stability
Our tool must handle:
1 GB → small test dumps (easy)
40–100 GB production dumps (hard)
Running in headless, CI/CD environments
Parsing without crashing
Most homegrown parsers fail silently on production dumps.
We need automation / reporting / error handling. Even if we parse the heap correctly:
We must generate reports programmatically
Identify “Problem Suspects”
Handle malformed dumps
Support multiple JVM types
All this adds more complexity.
Industry estimate
Rewriting MAT from scratch is approximately:
| Step | Time estimate |
| HPROF parser | 2–3 months (some hprof parsers can be found on GitHub) |
| Graph builder + dominator tree | 2–3 months |
| Retained size calculation | 1 month |
| Leak detection heuristics | 1 month |
| Testing & validation | 2–3 months |
| Total | ~8–12 months |
And that’s for a team of engineers experienced in JVM internals.
Summary
Reasons why it is “extremely complex”:
JVM heaps are massive and intricate graphs
.hprof binary format is complicated and JVM-dependent
Must correctly identify GC roots
Must compute dominator trees efficiently
Must calculate retained memory precisely
Must scale to production heaps
Must generate usable, reliable reports
Bottom line: We would be basically rewriting Eclipse MAT, which has been refined over 15+ years.
Instead of writing our own parser:
We can use MAT Core + CLI automation
Focus on automating leak detection logic
We get all graph algorithms, retained size, dominators, and reports for free
Our work becomes hours of automation and Java code, not months of low-level engineering.
Notice: despite years of development, MAT is still not 100% correct:
Off-Heap Blindness: It cannot track native memory leaks (like DirectByteBuffers or JNI allocations) because they don’t live in the .hprof file.
Unreachable Objects: Automated dumps can capture “dead” objects not yet swept by GC, sometimes leading to false positives depending on the garbage collector algorithm used (e.g., ZGC or Shenandoah).
Complex Cyclic Dependencies: If objects reference each other in massive circular chains, MAT’s Dominator Tree algorithm can struggle to isolate a single root suspect.
🚀 What’s Next?
If you are interested to see code example, where I build a fully headless Java orchestrator using ProcessBuilder to parse .hprof file automatically please read: Automating JVM Heap Dump Analysis – A Complete Java Implementation.
Leave a Reply