Flame graphs 76 gas credit card login


Flame graphs are a visualization of profiled software, allowing the most frequent code-paths to be identified quickly and accurately. They can be generated using my open source programs on github.com/brendangregg/FlameGraph, which create interactive SVGs. See the Updates section for other implementations. Recently I’ve been helping with d3-flame-graph.

The x-axis shows the stack profile population, sorted alphabetically (it is not the passage of time), and the y-axis shows stack depth, counting from zero at the bottom. Each rectangle represents a stack frame. The wider a frame is is, the more often it was present in the stacks. The top edge shows what is on-CPU, and beneath it is its ancestry. The colors are usually not significant, picked randomly to differentiate frames.

Icicle charts are flame graphs upside down. Some people prefer it that way. My flamegraph.pl creates them using –inverted. I prefer the standard "flame" layout, where the y-axis is counting stack depth upwards from zero at the bottom. With icicle charts, the y-axis has zero at the top, and counts downwards, which I find odd (I’m used to line charts with a 0,0 origin in the bottom left). I’m also used to scanning them top-down to look for plateaus. But I don’t really feel strongly about this.

Flame charts were first added by Google Chrome’s WebKit Web Inspector ( bug). While inspired by flame graphs, flame charts put the passage of time on the x-axis instead of the alphabet. This means that time-based patterns can studied. Flame graphs reorder the x-axis samples alphabetically, which maximizes frame merging, and better shows the big picture of the profile. Multi-threaded applications can’t be shown sensibly by a single flame chart, whereas they can with a flame graphs (a problem flame charts didn’t need to deal with, since it was initially used for single-threaded JavaScript analysis). Both visualizations are useful, and tools should make both available if possible (eg, TraceCompass does). Some analysis tools have implemented flame charts and mistakingly called them flame graphs.

I invented flame graphs when working on a MySQL performance issue and needed to understand CPU usage quickly and in depth. The regular profilers/tracers had produced walls of text, so I was exploring visualizations. I first traced CPU function calls and visualized it using Neelakanth Nadgir’s time-ordered visualization for callstacks, which itself was inspired by Roch Bourbonnais’s CallStackAnalyzer and Jan Boerhout’s vftrace. These look similar to flame graphs, but have the passage of time on the x-axis. But there were two problems: the overhead of function tracing was too high, perturbing the target, and the final visualization was too dense to read when spanning multiple seconds. I switched to timed sampling (profiling) to solve the overhead problem, but since the function flow is no longer known (sampling has gaps) I ditched time on the x-axis and reordered samples to maximize frame merging. It worked, the final visualization was much more readable. Neelakanth and Roch’s visualizations used completely random colors to differentiate frames. I thought it looked nicer to narrow the color palette, and picked just warm colors initially as it explained why the CPUs were "hot" (busy). Since it resembled flames, it quickly became known as flame graphs.

I described more detail of the original performance problem that led to flame graphs in my ACMQ/CACM article (link above). The flame graph visualization is really an adjacency diagram with an inverted icicle layout, which I used to visualize profiled stack traces. Updates

• Google Chrome’s performance analysis tool, WebKit Web Inspector, introduced " Flame Charts", inspired by flame graphs. These are a similar visualization, but the x-axis is time, instead of the alphabet. Check out the screenshot, which includes a mouse-over popup that links to the source code. (Update: there is a bug to add flame graphs to Chrome, in addition to flame charts: Chromium 452624.)

• I wrote a document summarizing four techniques for generating Memory Leak (and Growth) Flame Graphs, which visualize stacks with byte counts, instead of the traditional CPU sample Flame Graphs. I also colored them green to indicate that they are a different type.

• Paul Irish and Umar Hansa posted an awesome demo of using FlameCharts to investigate time in V8, which includes zooming in and clicking on functions to go to code, and Addy Osmani posted a longer video tutorial. While these aren’t Flame Graphs, they show developments in a related visualization: a time-series version that retains sequence and ordering.

• I wrote a post showing how the Google lightweight-java-profiler can be used to create Java Flame Graphs. This is only Java code time, and not other CPU consumers. See my later work with Java (-XX:+PreserveFramePointer) for a different and more complete profile.

• In my SCALE13x talk (2015), I previewed mixed-mode Java flame graphs using Linux perf_events ("perf"). This uses a JVM frame pointer patch that has become the -XX:+PreserveFramePointer option in both JDK9 ( JDK-8068945) and JDK8u60 ( JDK-8072465). For the first time, we can see all CPU consumers in one visualization! See slides 40 to 57 for more details, and my post about it (when I write it!).

• Cor-Paul Bezemer has been investigating flame graph differentials with his Flamegraphdiff software, which shows the difference from A to B using three flame graphs simultaneously. This was also the subject of a SANER2015 paper ($), and talk.

• Zoltan Majo fixed JDK-8068945 for Java 9 and JDK-8072465 for Java 8 update 60 build 19 (or later, download as early access here). This adds -XX:+PreserveFramePointer, which allows Linux perf_events to sample full stacks for making flame graphs. This began with a prototype patch I developed and submitted: ( A hotspot patch for stack profiling). Great to see this functionality make it into the JVM!

• NodeSource have flamegraphs in their NSolid Node.js analysis product. The graphics look very nice, and they also have treemaps and sunbursts (both of which I think are usually less effective than flame graphs, but I don’t mind having the option).

• Evan Klitzke posted pyflame: Uber enginering’s ptracing profiler for python, as well as the source on github. Python is tricky to profile as it’s interpreted only, so basic frame pointer-based stack walking only identifies interpreter frames.

• Myself and my Netflix colleague Martin Spier published FlameScope, a new performance analysis tool that allows profiles to be visualized as subsecond-offset heat maps, and then time ranges to be selected and then visualized as a flame graph. It’s also on github Netflix/flamescope.