| <!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" |
| "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" |
| [<!ENTITY % poky SYSTEM "../poky.ent"> %poky; ] > |
| |
| <chapter id='profile-manual-usage'> |
| |
| <title>Basic Usage (with examples) for each of the Yocto Tracing Tools</title> |
| |
| <para> |
| This chapter presents basic usage examples for each of the tracing |
| tools. |
| </para> |
| |
| <section id='profile-manual-perf'> |
| <title>perf</title> |
| |
| <para> |
| The 'perf' tool is the profiling and tracing tool that comes |
| bundled with the Linux kernel. |
| </para> |
| |
| <para> |
| Don't let the fact that it's part of the kernel fool you into thinking |
| that it's only for tracing and profiling the kernel - you can indeed |
| use it to trace and profile just the kernel, but you can also use it |
| to profile specific applications separately (with or without kernel |
| context), and you can also use it to trace and profile the kernel |
| and all applications on the system simultaneously to gain a system-wide |
| view of what's going on. |
| </para> |
| |
| <para> |
| In many ways, perf aims to be a superset of all the tracing and profiling |
| tools available in Linux today, including all the other tools covered |
| in this HOWTO. The past couple of years have seen perf subsume a lot |
| of the functionality of those other tools and, at the same time, those |
| other tools have removed large portions of their previous functionality |
| and replaced it with calls to the equivalent functionality now |
| implemented by the perf subsystem. Extrapolation suggests that at |
| some point those other tools will simply become completely redundant |
| and go away; until then, we'll cover those other tools in these pages |
| and in many cases show how the same things can be accomplished in |
| perf and the other tools when it seems useful to do so. |
| </para> |
| |
| <para> |
| The coverage below details some of the most common ways you'll likely |
| want to apply the tool; full documentation can be found either within |
| the tool itself or in the man pages at |
| <ulink url='http://linux.die.net/man/1/perf'>perf(1)</ulink>. |
| </para> |
| |
| <section id='perf-setup'> |
| <title>Setup</title> |
| |
| <para> |
| For this section, we'll assume you've already performed the basic |
| setup outlined in the General Setup section. |
| </para> |
| |
| <para> |
| In particular, you'll get the most mileage out of perf if you |
| profile an image built with INHIBIT_PACKAGE_STRIP = "1" in your |
| local.conf. |
| </para> |
| |
| <para> |
| perf runs on the target system for the most part. You can archive |
| profile data and copy it to the host for analysis, but for the |
| rest of this document we assume you've ssh'ed to the host and |
| will be running the perf commands on the target. |
| </para> |
| </section> |
| |
| <section id='perf-basic-usage'> |
| <title>Basic Usage</title> |
| |
| <para> |
| The perf tool is pretty much self-documenting. To remind yourself |
| of the available commands, simply type 'perf', which will show you |
| basic usage along with the available perf subcommands: |
| <literallayout class='monospaced'> |
| root@crownbay:~# perf |
| |
| usage: perf [--version] [--help] COMMAND [ARGS] |
| |
| The most commonly used perf commands are: |
| annotate Read perf.data (created by perf record) and display annotated code |
| archive Create archive with object files with build-ids found in perf.data file |
| bench General framework for benchmark suites |
| buildid-cache Manage build-id cache. |
| buildid-list List the buildids in a perf.data file |
| diff Read two perf.data files and display the differential profile |
| evlist List the event names in a perf.data file |
| inject Filter to augment the events stream with additional information |
| kmem Tool to trace/measure kernel memory(slab) properties |
| kvm Tool to trace/measure kvm guest os |
| list List all symbolic event types |
| lock Analyze lock events |
| probe Define new dynamic tracepoints |
| record Run a command and record its profile into perf.data |
| report Read perf.data (created by perf record) and display the profile |
| sched Tool to trace/measure scheduler properties (latencies) |
| script Read perf.data (created by perf record) and display trace output |
| stat Run a command and gather performance counter statistics |
| test Runs sanity tests. |
| timechart Tool to visualize total system behavior during a workload |
| top System profiling tool. |
| |
| See 'perf help COMMAND' for more information on a specific command. |
| </literallayout> |
| </para> |
| |
| <section id='using-perf-to-do-basic-profiling'> |
| <title>Using perf to do Basic Profiling</title> |
| |
| <para> |
| As a simple test case, we'll profile the 'wget' of a fairly large |
| file, which is a minimally interesting case because it has both |
| file and network I/O aspects, and at least in the case of standard |
| Yocto images, it's implemented as part of busybox, so the methods |
| we use to analyze it can be used in a very similar way to the whole |
| host of supported busybox applets in Yocto. |
| <literallayout class='monospaced'> |
| root@crownbay:~# rm linux-2.6.19.2.tar.bz2; \ |
| wget <ulink url='http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2'>http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2</ulink> |
| </literallayout> |
| The quickest and easiest way to get some basic overall data about |
| what's going on for a particular workload is to profile it using |
| 'perf stat'. 'perf stat' basically profiles using a few default |
| counters and displays the summed counts at the end of the run: |
| <literallayout class='monospaced'> |
| root@crownbay:~# perf stat wget <ulink url='http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2'>http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2</ulink> |
| Connecting to downloads.yoctoproject.org (140.211.169.59:80) |
| linux-2.6.19.2.tar.b 100% |***************************************************| 41727k 0:00:00 ETA |
| |
| Performance counter stats for 'wget <ulink url='http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2'>http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2</ulink>': |
| |
| 4597.223902 task-clock # 0.077 CPUs utilized |
| 23568 context-switches # 0.005 M/sec |
| 68 CPU-migrations # 0.015 K/sec |
| 241 page-faults # 0.052 K/sec |
| 3045817293 cycles # 0.663 GHz |
| <not supported> stalled-cycles-frontend |
| <not supported> stalled-cycles-backend |
| 858909167 instructions # 0.28 insns per cycle |
| 165441165 branches # 35.987 M/sec |
| 19550329 branch-misses # 11.82% of all branches |
| |
| 59.836627620 seconds time elapsed |
| </literallayout> |
| Many times such a simple-minded test doesn't yield much of |
| interest, but sometimes it does (see Real-world Yocto bug |
| (slow loop-mounted write speed)). |
| </para> |
| |
| <para> |
| Also, note that 'perf stat' isn't restricted to a fixed set of |
| counters - basically any event listed in the output of 'perf list' |
| can be tallied by 'perf stat'. For example, suppose we wanted to |
| see a summary of all the events related to kernel memory |
| allocation/freeing along with cache hits and misses: |
| <literallayout class='monospaced'> |
| root@crownbay:~# perf stat -e kmem:* -e cache-references -e cache-misses wget <ulink url='http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2'>http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2</ulink> |
| Connecting to downloads.yoctoproject.org (140.211.169.59:80) |
| linux-2.6.19.2.tar.b 100% |***************************************************| 41727k 0:00:00 ETA |
| |
| Performance counter stats for 'wget <ulink url='http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2'>http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2</ulink>': |
| |
| 5566 kmem:kmalloc |
| 125517 kmem:kmem_cache_alloc |
| 0 kmem:kmalloc_node |
| 0 kmem:kmem_cache_alloc_node |
| 34401 kmem:kfree |
| 69920 kmem:kmem_cache_free |
| 133 kmem:mm_page_free |
| 41 kmem:mm_page_free_batched |
| 11502 kmem:mm_page_alloc |
| 11375 kmem:mm_page_alloc_zone_locked |
| 0 kmem:mm_page_pcpu_drain |
| 0 kmem:mm_page_alloc_extfrag |
| 66848602 cache-references |
| 2917740 cache-misses # 4.365 % of all cache refs |
| |
| 44.831023415 seconds time elapsed |
| </literallayout> |
| So 'perf stat' gives us a nice easy way to get a quick overview of |
| what might be happening for a set of events, but normally we'd |
| need a little more detail in order to understand what's going on |
| in a way that we can act on in a useful way. |
| </para> |
| |
| <para> |
| To dive down into a next level of detail, we can use 'perf |
| record'/'perf report' which will collect profiling data and |
| present it to use using an interactive text-based UI (or |
| simply as text if we specify --stdio to 'perf report'). |
| </para> |
| |
| <para> |
| As our first attempt at profiling this workload, we'll simply |
| run 'perf record', handing it the workload we want to profile |
| (everything after 'perf record' and any perf options we hand |
| it - here none - will be executed in a new shell). perf collects |
| samples until the process exits and records them in a file named |
| 'perf.data' in the current working directory. |
| <literallayout class='monospaced'> |
| root@crownbay:~# perf record wget <ulink url='http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2'>http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2</ulink> |
| |
| Connecting to downloads.yoctoproject.org (140.211.169.59:80) |
| linux-2.6.19.2.tar.b 100% |************************************************| 41727k 0:00:00 ETA |
| [ perf record: Woken up 1 times to write data ] |
| [ perf record: Captured and wrote 0.176 MB perf.data (~7700 samples) ] |
| </literallayout> |
| To see the results in a 'text-based UI' (tui), simply run |
| 'perf report', which will read the perf.data file in the current |
| working directory and display the results in an interactive UI: |
| <literallayout class='monospaced'> |
| root@crownbay:~# perf report |
| </literallayout> |
| </para> |
| |
| <para> |
| <imagedata fileref="figures/perf-wget-flat-stripped.png" width="6in" depth="7in" align="center" scalefit="1" /> |
| </para> |
| |
| <para> |
| The above screenshot displays a 'flat' profile, one entry for |
| each 'bucket' corresponding to the functions that were profiled |
| during the profiling run, ordered from the most popular to the |
| least (perf has options to sort in various orders and keys as |
| well as display entries only above a certain threshold and so |
| on - see the perf documentation for details). Note that this |
| includes both userspace functions (entries containing a [.]) and |
| kernel functions accounted to the process (entries containing |
| a [k]). (perf has command-line modifiers that can be used to |
| restrict the profiling to kernel or userspace, among others). |
| </para> |
| |
| <para> |
| Notice also that the above report shows an entry for 'busybox', |
| which is the executable that implements 'wget' in Yocto, but that |
| instead of a useful function name in that entry, it displays |
| a not-so-friendly hex value instead. The steps below will show |
| how to fix that problem. |
| </para> |
| |
| <para> |
| Before we do that, however, let's try running a different profile, |
| one which shows something a little more interesting. The only |
| difference between the new profile and the previous one is that |
| we'll add the -g option, which will record not just the address |
| of a sampled function, but the entire callchain to the sampled |
| function as well: |
| <literallayout class='monospaced'> |
| root@crownbay:~# perf record -g wget <ulink url='http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2'>http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2</ulink> |
| Connecting to downloads.yoctoproject.org (140.211.169.59:80) |
| linux-2.6.19.2.tar.b 100% |************************************************| 41727k 0:00:00 ETA |
| [ perf record: Woken up 3 times to write data ] |
| [ perf record: Captured and wrote 0.652 MB perf.data (~28476 samples) ] |
| |
| |
| root@crownbay:~# perf report |
| </literallayout> |
| </para> |
| |
| <para> |
| <imagedata fileref="figures/perf-wget-g-copy-to-user-expanded-stripped.png" width="6in" depth="7in" align="center" scalefit="1" /> |
| </para> |
| |
| <para> |
| Using the callgraph view, we can actually see not only which |
| functions took the most time, but we can also see a summary of |
| how those functions were called and learn something about how the |
| program interacts with the kernel in the process. |
| </para> |
| |
| <para> |
| Notice that each entry in the above screenshot now contains a '+' |
| on the left-hand side. This means that we can expand the entry and |
| drill down into the callchains that feed into that entry. |
| Pressing 'enter' on any one of them will expand the callchain |
| (you can also press 'E' to expand them all at the same time or 'C' |
| to collapse them all). |
| </para> |
| |
| <para> |
| In the screenshot above, we've toggled the __copy_to_user_ll() |
| entry and several subnodes all the way down. This lets us see |
| which callchains contributed to the profiled __copy_to_user_ll() |
| function which contributed 1.77% to the total profile. |
| </para> |
| |
| <para> |
| As a bit of background explanation for these callchains, think |
| about what happens at a high level when you run wget to get a file |
| out on the network. Basically what happens is that the data comes |
| into the kernel via the network connection (socket) and is passed |
| to the userspace program 'wget' (which is actually a part of |
| busybox, but that's not important for now), which takes the buffers |
| the kernel passes to it and writes it to a disk file to save it. |
| </para> |
| |
| <para> |
| The part of this process that we're looking at in the above call |
| stacks is the part where the kernel passes the data it's read from |
| the socket down to wget i.e. a copy-to-user. |
| </para> |
| |
| <para> |
| Notice also that here there's also a case where the hex value |
| is displayed in the callstack, here in the expanded |
| sys_clock_gettime() function. Later we'll see it resolve to a |
| userspace function call in busybox. |
| </para> |
| |
| <para> |
| <imagedata fileref="figures/perf-wget-g-copy-from-user-expanded-stripped.png" width="6in" depth="7in" align="center" scalefit="1" /> |
| </para> |
| |
| <para> |
| The above screenshot shows the other half of the journey for the |
| data - from the wget program's userspace buffers to disk. To get |
| the buffers to disk, the wget program issues a write(2), which |
| does a copy-from-user to the kernel, which then takes care via |
| some circuitous path (probably also present somewhere in the |
| profile data), to get it safely to disk. |
| </para> |
| |
| <para> |
| Now that we've seen the basic layout of the profile data and the |
| basics of how to extract useful information out of it, let's get |
| back to the task at hand and see if we can get some basic idea |
| about where the time is spent in the program we're profiling, |
| wget. Remember that wget is actually implemented as an applet |
| in busybox, so while the process name is 'wget', the executable |
| we're actually interested in is busybox. So let's expand the |
| first entry containing busybox: |
| </para> |
| |
| <para> |
| <imagedata fileref="figures/perf-wget-busybox-expanded-stripped.png" width="6in" depth="7in" align="center" scalefit="1" /> |
| </para> |
| |
| <para> |
| Again, before we expanded we saw that the function was labeled |
| with a hex value instead of a symbol as with most of the kernel |
| entries. Expanding the busybox entry doesn't make it any better. |
| </para> |
| |
| <para> |
| The problem is that perf can't find the symbol information for the |
| busybox binary, which is actually stripped out by the Yocto build |
| system. |
| </para> |
| |
| <para> |
| One way around that is to put the following in your local.conf |
| when you build the image: |
| <literallayout class='monospaced'> |
| INHIBIT_PACKAGE_STRIP = "1" |
| </literallayout> |
| However, we already have an image with the binaries stripped, |
| so what can we do to get perf to resolve the symbols? Basically |
| we need to install the debuginfo for the busybox package. |
| </para> |
| |
| <para> |
| To generate the debug info for the packages in the image, we can |
| add dbg-pkgs to EXTRA_IMAGE_FEATURES in local.conf. For example: |
| <literallayout class='monospaced'> |
| EXTRA_IMAGE_FEATURES = "debug-tweaks tools-profile dbg-pkgs" |
| </literallayout> |
| Additionally, in order to generate the type of debuginfo that |
| perf understands, we also need to add the following to local.conf: |
| <literallayout class='monospaced'> |
| PACKAGE_DEBUG_SPLIT_STYLE = 'debug-file-directory' |
| </literallayout> |
| Once we've done that, we can install the debuginfo for busybox. |
| The debug packages once built can be found in |
| build/tmp/deploy/rpm/* on the host system. Find the |
| busybox-dbg-...rpm file and copy it to the target. For example: |
| <literallayout class='monospaced'> |
| [trz@empanada core2]$ scp /home/trz/yocto/crownbay-tracing-dbg/build/tmp/deploy/rpm/core2_32/busybox-dbg-1.20.2-r2.core2_32.rpm root@192.168.1.31: |
| root@192.168.1.31's password: |
| busybox-dbg-1.20.2-r2.core2_32.rpm 100% 1826KB 1.8MB/s 00:01 |
| </literallayout> |
| Now install the debug rpm on the target: |
| <literallayout class='monospaced'> |
| root@crownbay:~# rpm -i busybox-dbg-1.20.2-r2.core2_32.rpm |
| </literallayout> |
| Now that the debuginfo is installed, we see that the busybox |
| entries now display their functions symbolically: |
| </para> |
| |
| <para> |
| <imagedata fileref="figures/perf-wget-busybox-debuginfo.png" width="6in" depth="7in" align="center" scalefit="1" /> |
| </para> |
| |
| <para> |
| If we expand one of the entries and press 'enter' on a leaf node, |
| we're presented with a menu of actions we can take to get more |
| information related to that entry: |
| </para> |
| |
| <para> |
| <imagedata fileref="figures/perf-wget-busybox-dso-zoom-menu.png" width="6in" depth="2in" align="center" scalefit="1" /> |
| </para> |
| |
| <para> |
| One of these actions allows us to show a view that displays a |
| busybox-centric view of the profiled functions (in this case we've |
| also expanded all the nodes using the 'E' key): |
| </para> |
| |
| <para> |
| <imagedata fileref="figures/perf-wget-busybox-dso-zoom.png" width="6in" depth="7in" align="center" scalefit="1" /> |
| </para> |
| |
| <para> |
| Finally, we can see that now that the busybox debuginfo is |
| installed, the previously unresolved symbol in the |
| sys_clock_gettime() entry mentioned previously is now resolved, |
| and shows that the sys_clock_gettime system call that was the |
| source of 6.75% of the copy-to-user overhead was initiated by |
| the handle_input() busybox function: |
| </para> |
| |
| <para> |
| <imagedata fileref="figures/perf-wget-g-copy-to-user-expanded-debuginfo.png" width="6in" depth="7in" align="center" scalefit="1" /> |
| </para> |
| |
| <para> |
| At the lowest level of detail, we can dive down to the assembly |
| level and see which instructions caused the most overhead in a |
| function. Pressing 'enter' on the 'udhcpc_main' function, we're |
| again presented with a menu: |
| </para> |
| |
| <para> |
| <imagedata fileref="figures/perf-wget-busybox-annotate-menu.png" width="6in" depth="2in" align="center" scalefit="1" /> |
| </para> |
| |
| <para> |
| Selecting 'Annotate udhcpc_main', we get a detailed listing of |
| percentages by instruction for the udhcpc_main function. From the |
| display, we can see that over 50% of the time spent in this |
| function is taken up by a couple tests and the move of a |
| constant (1) to a register: |
| </para> |
| |
| <para> |
| <imagedata fileref="figures/perf-wget-busybox-annotate-udhcpc.png" width="6in" depth="7in" align="center" scalefit="1" /> |
| </para> |
| |
| <para> |
| As a segue into tracing, let's try another profile using a |
| different counter, something other than the default 'cycles'. |
| </para> |
| |
| <para> |
| The tracing and profiling infrastructure in Linux has become |
| unified in a way that allows us to use the same tool with a |
| completely different set of counters, not just the standard |
| hardware counters that traditional tools have had to restrict |
| themselves to (of course the traditional tools can also make use |
| of the expanded possibilities now available to them, and in some |
| cases have, as mentioned previously). |
| </para> |
| |
| <para> |
| We can get a list of the available events that can be used to |
| profile a workload via 'perf list': |
| <literallayout class='monospaced'> |
| root@crownbay:~# perf list |
| |
| List of pre-defined events (to be used in -e): |
| cpu-cycles OR cycles [Hardware event] |
| stalled-cycles-frontend OR idle-cycles-frontend [Hardware event] |
| stalled-cycles-backend OR idle-cycles-backend [Hardware event] |
| instructions [Hardware event] |
| cache-references [Hardware event] |
| cache-misses [Hardware event] |
| branch-instructions OR branches [Hardware event] |
| branch-misses [Hardware event] |
| bus-cycles [Hardware event] |
| ref-cycles [Hardware event] |
| |
| cpu-clock [Software event] |
| task-clock [Software event] |
| page-faults OR faults [Software event] |
| minor-faults [Software event] |
| major-faults [Software event] |
| context-switches OR cs [Software event] |
| cpu-migrations OR migrations [Software event] |
| alignment-faults [Software event] |
| emulation-faults [Software event] |
| |
| L1-dcache-loads [Hardware cache event] |
| L1-dcache-load-misses [Hardware cache event] |
| L1-dcache-prefetch-misses [Hardware cache event] |
| L1-icache-loads [Hardware cache event] |
| L1-icache-load-misses [Hardware cache event] |
| . |
| . |
| . |
| rNNN [Raw hardware event descriptor] |
| cpu/t1=v1[,t2=v2,t3 ...]/modifier [Raw hardware event descriptor] |
| (see 'perf list --help' on how to encode it) |
| |
| mem:<addr>[:access] [Hardware breakpoint] |
| |
| sunrpc:rpc_call_status [Tracepoint event] |
| sunrpc:rpc_bind_status [Tracepoint event] |
| sunrpc:rpc_connect_status [Tracepoint event] |
| sunrpc:rpc_task_begin [Tracepoint event] |
| skb:kfree_skb [Tracepoint event] |
| skb:consume_skb [Tracepoint event] |
| skb:skb_copy_datagram_iovec [Tracepoint event] |
| net:net_dev_xmit [Tracepoint event] |
| net:net_dev_queue [Tracepoint event] |
| net:netif_receive_skb [Tracepoint event] |
| net:netif_rx [Tracepoint event] |
| napi:napi_poll [Tracepoint event] |
| sock:sock_rcvqueue_full [Tracepoint event] |
| sock:sock_exceed_buf_limit [Tracepoint event] |
| udp:udp_fail_queue_rcv_skb [Tracepoint event] |
| hda:hda_send_cmd [Tracepoint event] |
| hda:hda_get_response [Tracepoint event] |
| hda:hda_bus_reset [Tracepoint event] |
| scsi:scsi_dispatch_cmd_start [Tracepoint event] |
| scsi:scsi_dispatch_cmd_error [Tracepoint event] |
| scsi:scsi_eh_wakeup [Tracepoint event] |
| drm:drm_vblank_event [Tracepoint event] |
| drm:drm_vblank_event_queued [Tracepoint event] |
| drm:drm_vblank_event_delivered [Tracepoint event] |
| random:mix_pool_bytes [Tracepoint event] |
| random:mix_pool_bytes_nolock [Tracepoint event] |
| random:credit_entropy_bits [Tracepoint event] |
| gpio:gpio_direction [Tracepoint event] |
| gpio:gpio_value [Tracepoint event] |
| block:block_rq_abort [Tracepoint event] |
| block:block_rq_requeue [Tracepoint event] |
| block:block_rq_issue [Tracepoint event] |
| block:block_bio_bounce [Tracepoint event] |
| block:block_bio_complete [Tracepoint event] |
| block:block_bio_backmerge [Tracepoint event] |
| . |
| . |
| writeback:writeback_wake_thread [Tracepoint event] |
| writeback:writeback_wake_forker_thread [Tracepoint event] |
| writeback:writeback_bdi_register [Tracepoint event] |
| . |
| . |
| writeback:writeback_single_inode_requeue [Tracepoint event] |
| writeback:writeback_single_inode [Tracepoint event] |
| kmem:kmalloc [Tracepoint event] |
| kmem:kmem_cache_alloc [Tracepoint event] |
| kmem:mm_page_alloc [Tracepoint event] |
| kmem:mm_page_alloc_zone_locked [Tracepoint event] |
| kmem:mm_page_pcpu_drain [Tracepoint event] |
| kmem:mm_page_alloc_extfrag [Tracepoint event] |
| vmscan:mm_vmscan_kswapd_sleep [Tracepoint event] |
| vmscan:mm_vmscan_kswapd_wake [Tracepoint event] |
| vmscan:mm_vmscan_wakeup_kswapd [Tracepoint event] |
| vmscan:mm_vmscan_direct_reclaim_begin [Tracepoint event] |
| . |
| . |
| module:module_get [Tracepoint event] |
| module:module_put [Tracepoint event] |
| module:module_request [Tracepoint event] |
| sched:sched_kthread_stop [Tracepoint event] |
| sched:sched_wakeup [Tracepoint event] |
| sched:sched_wakeup_new [Tracepoint event] |
| sched:sched_process_fork [Tracepoint event] |
| sched:sched_process_exec [Tracepoint event] |
| sched:sched_stat_runtime [Tracepoint event] |
| rcu:rcu_utilization [Tracepoint event] |
| workqueue:workqueue_queue_work [Tracepoint event] |
| workqueue:workqueue_execute_end [Tracepoint event] |
| signal:signal_generate [Tracepoint event] |
| signal:signal_deliver [Tracepoint event] |
| timer:timer_init [Tracepoint event] |
| timer:timer_start [Tracepoint event] |
| timer:hrtimer_cancel [Tracepoint event] |
| timer:itimer_state [Tracepoint event] |
| timer:itimer_expire [Tracepoint event] |
| irq:irq_handler_entry [Tracepoint event] |
| irq:irq_handler_exit [Tracepoint event] |
| irq:softirq_entry [Tracepoint event] |
| irq:softirq_exit [Tracepoint event] |
| irq:softirq_raise [Tracepoint event] |
| printk:console [Tracepoint event] |
| task:task_newtask [Tracepoint event] |
| task:task_rename [Tracepoint event] |
| syscalls:sys_enter_socketcall [Tracepoint event] |
| syscalls:sys_exit_socketcall [Tracepoint event] |
| . |
| . |
| . |
| syscalls:sys_enter_unshare [Tracepoint event] |
| syscalls:sys_exit_unshare [Tracepoint event] |
| raw_syscalls:sys_enter [Tracepoint event] |
| raw_syscalls:sys_exit [Tracepoint event] |
| </literallayout> |
| </para> |
| |
| <informalexample> |
| <emphasis>Tying it Together:</emphasis> These are exactly the same set of events defined |
| by the trace event subsystem and exposed by |
| ftrace/tracecmd/kernelshark as files in |
| /sys/kernel/debug/tracing/events, by SystemTap as |
| kernel.trace("tracepoint_name") and (partially) accessed by LTTng. |
| </informalexample> |
| |
| <para> |
| Only a subset of these would be of interest to us when looking at |
| this workload, so let's choose the most likely subsystems |
| (identified by the string before the colon in the Tracepoint events) |
| and do a 'perf stat' run using only those wildcarded subsystems: |
| <literallayout class='monospaced'> |
| root@crownbay:~# perf stat -e skb:* -e net:* -e napi:* -e sched:* -e workqueue:* -e irq:* -e syscalls:* wget <ulink url='http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2'>http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2</ulink> |
| Performance counter stats for 'wget <ulink url='http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2'>http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2</ulink>': |
| |
| 23323 skb:kfree_skb |
| 0 skb:consume_skb |
| 49897 skb:skb_copy_datagram_iovec |
| 6217 net:net_dev_xmit |
| 6217 net:net_dev_queue |
| 7962 net:netif_receive_skb |
| 2 net:netif_rx |
| 8340 napi:napi_poll |
| 0 sched:sched_kthread_stop |
| 0 sched:sched_kthread_stop_ret |
| 3749 sched:sched_wakeup |
| 0 sched:sched_wakeup_new |
| 0 sched:sched_switch |
| 29 sched:sched_migrate_task |
| 0 sched:sched_process_free |
| 1 sched:sched_process_exit |
| 0 sched:sched_wait_task |
| 0 sched:sched_process_wait |
| 0 sched:sched_process_fork |
| 1 sched:sched_process_exec |
| 0 sched:sched_stat_wait |
| 2106519415641 sched:sched_stat_sleep |
| 0 sched:sched_stat_iowait |
| 147453613 sched:sched_stat_blocked |
| 12903026955 sched:sched_stat_runtime |
| 0 sched:sched_pi_setprio |
| 3574 workqueue:workqueue_queue_work |
| 3574 workqueue:workqueue_activate_work |
| 0 workqueue:workqueue_execute_start |
| 0 workqueue:workqueue_execute_end |
| 16631 irq:irq_handler_entry |
| 16631 irq:irq_handler_exit |
| 28521 irq:softirq_entry |
| 28521 irq:softirq_exit |
| 28728 irq:softirq_raise |
| 1 syscalls:sys_enter_sendmmsg |
| 1 syscalls:sys_exit_sendmmsg |
| 0 syscalls:sys_enter_recvmmsg |
| 0 syscalls:sys_exit_recvmmsg |
| 14 syscalls:sys_enter_socketcall |
| 14 syscalls:sys_exit_socketcall |
| . |
| . |
| . |
| 16965 syscalls:sys_enter_read |
| 16965 syscalls:sys_exit_read |
| 12854 syscalls:sys_enter_write |
| 12854 syscalls:sys_exit_write |
| . |
| . |
| . |
| |
| 58.029710972 seconds time elapsed |
| </literallayout> |
| Let's pick one of these tracepoints and tell perf to do a profile |
| using it as the sampling event: |
| <literallayout class='monospaced'> |
| root@crownbay:~# perf record -g -e sched:sched_wakeup wget <ulink url='http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2'>http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2</ulink> |
| </literallayout> |
| </para> |
| |
| <para> |
| <imagedata fileref="figures/sched-wakeup-profile.png" width="6in" depth="7in" align="center" scalefit="1" /> |
| </para> |
| |
| <para> |
| The screenshot above shows the results of running a profile using |
| sched:sched_switch tracepoint, which shows the relative costs of |
| various paths to sched_wakeup (note that sched_wakeup is the |
| name of the tracepoint - it's actually defined just inside |
| ttwu_do_wakeup(), which accounts for the function name actually |
| displayed in the profile: |
| <literallayout class='monospaced'> |
| /* |
| * Mark the task runnable and perform wakeup-preemption. |
| */ |
| static void |
| ttwu_do_wakeup(struct rq *rq, struct task_struct *p, int wake_flags) |
| { |
| trace_sched_wakeup(p, true); |
| . |
| . |
| . |
| } |
| </literallayout> |
| A couple of the more interesting callchains are expanded and |
| displayed above, basically some network receive paths that |
| presumably end up waking up wget (busybox) when network data is |
| ready. |
| </para> |
| |
| <para> |
| Note that because tracepoints are normally used for tracing, |
| the default sampling period for tracepoints is 1 i.e. for |
| tracepoints perf will sample on every event occurrence (this |
| can be changed using the -c option). This is in contrast to |
| hardware counters such as for example the default 'cycles' |
| hardware counter used for normal profiling, where sampling |
| periods are much higher (in the thousands) because profiling should |
| have as low an overhead as possible and sampling on every cycle |
| would be prohibitively expensive. |
| </para> |
| </section> |
| |
| <section id='using-perf-to-do-basic-tracing'> |
| <title>Using perf to do Basic Tracing</title> |
| |
| <para> |
| Profiling is a great tool for solving many problems or for |
| getting a high-level view of what's going on with a workload or |
| across the system. It is however by definition an approximation, |
| as suggested by the most prominent word associated with it, |
| 'sampling'. On the one hand, it allows a representative picture of |
| what's going on in the system to be cheaply taken, but on the other |
| hand, that cheapness limits its utility when that data suggests a |
| need to 'dive down' more deeply to discover what's really going |
| on. In such cases, the only way to see what's really going on is |
| to be able to look at (or summarize more intelligently) the |
| individual steps that go into the higher-level behavior exposed |
| by the coarse-grained profiling data. |
| </para> |
| |
| <para> |
| As a concrete example, we can trace all the events we think might |
| be applicable to our workload: |
| <literallayout class='monospaced'> |
| root@crownbay:~# perf record -g -e skb:* -e net:* -e napi:* -e sched:sched_switch -e sched:sched_wakeup -e irq:* |
| -e syscalls:sys_enter_read -e syscalls:sys_exit_read -e syscalls:sys_enter_write -e syscalls:sys_exit_write |
| wget <ulink url='http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2'>http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2</ulink> |
| </literallayout> |
| We can look at the raw trace output using 'perf script' with no |
| arguments: |
| <literallayout class='monospaced'> |
| root@crownbay:~# perf script |
| |
| perf 1262 [000] 11624.857082: sys_exit_read: 0x0 |
| perf 1262 [000] 11624.857193: sched_wakeup: comm=migration/0 pid=6 prio=0 success=1 target_cpu=000 |
| wget 1262 [001] 11624.858021: softirq_raise: vec=1 [action=TIMER] |
| wget 1262 [001] 11624.858074: softirq_entry: vec=1 [action=TIMER] |
| wget 1262 [001] 11624.858081: softirq_exit: vec=1 [action=TIMER] |
| wget 1262 [001] 11624.858166: sys_enter_read: fd: 0x0003, buf: 0xbf82c940, count: 0x0200 |
| wget 1262 [001] 11624.858177: sys_exit_read: 0x200 |
| wget 1262 [001] 11624.858878: kfree_skb: skbaddr=0xeb248d80 protocol=0 location=0xc15a5308 |
| wget 1262 [001] 11624.858945: kfree_skb: skbaddr=0xeb248000 protocol=0 location=0xc15a5308 |
| wget 1262 [001] 11624.859020: softirq_raise: vec=1 [action=TIMER] |
| wget 1262 [001] 11624.859076: softirq_entry: vec=1 [action=TIMER] |
| wget 1262 [001] 11624.859083: softirq_exit: vec=1 [action=TIMER] |
| wget 1262 [001] 11624.859167: sys_enter_read: fd: 0x0003, buf: 0xb7720000, count: 0x0400 |
| wget 1262 [001] 11624.859192: sys_exit_read: 0x1d7 |
| wget 1262 [001] 11624.859228: sys_enter_read: fd: 0x0003, buf: 0xb7720000, count: 0x0400 |
| wget 1262 [001] 11624.859233: sys_exit_read: 0x0 |
| wget 1262 [001] 11624.859573: sys_enter_read: fd: 0x0003, buf: 0xbf82c580, count: 0x0200 |
| wget 1262 [001] 11624.859584: sys_exit_read: 0x200 |
| wget 1262 [001] 11624.859864: sys_enter_read: fd: 0x0003, buf: 0xb7720000, count: 0x0400 |
| wget 1262 [001] 11624.859888: sys_exit_read: 0x400 |
| wget 1262 [001] 11624.859935: sys_enter_read: fd: 0x0003, buf: 0xb7720000, count: 0x0400 |
| wget 1262 [001] 11624.859944: sys_exit_read: 0x400 |
| </literallayout> |
| This gives us a detailed timestamped sequence of events that |
| occurred within the workload with respect to those events. |
| </para> |
| |
| <para> |
| In many ways, profiling can be viewed as a subset of tracing - |
| theoretically, if you have a set of trace events that's sufficient |
| to capture all the important aspects of a workload, you can derive |
| any of the results or views that a profiling run can. |
| </para> |
| |
| <para> |
| Another aspect of traditional profiling is that while powerful in |
| many ways, it's limited by the granularity of the underlying data. |
| Profiling tools offer various ways of sorting and presenting the |
| sample data, which make it much more useful and amenable to user |
| experimentation, but in the end it can't be used in an open-ended |
| way to extract data that just isn't present as a consequence of |
| the fact that conceptually, most of it has been thrown away. |
| </para> |
| |
| <para> |
| Full-blown detailed tracing data does however offer the opportunity |
| to manipulate and present the information collected during a |
| tracing run in an infinite variety of ways. |
| </para> |
| |
| <para> |
| Another way to look at it is that there are only so many ways that |
| the 'primitive' counters can be used on their own to generate |
| interesting output; to get anything more complicated than simple |
| counts requires some amount of additional logic, which is typically |
| very specific to the problem at hand. For example, if we wanted to |
| make use of a 'counter' that maps to the value of the time |
| difference between when a process was scheduled to run on a |
| processor and the time it actually ran, we wouldn't expect such |
| a counter to exist on its own, but we could derive one called say |
| 'wakeup_latency' and use it to extract a useful view of that metric |
| from trace data. Likewise, we really can't figure out from standard |
| profiling tools how much data every process on the system reads and |
| writes, along with how many of those reads and writes fail |
| completely. If we have sufficient trace data, however, we could |
| with the right tools easily extract and present that information, |
| but we'd need something other than pre-canned profiling tools to |
| do that. |
| </para> |
| |
| <para> |
| Luckily, there is a general-purpose way to handle such needs, |
| called 'programming languages'. Making programming languages |
| easily available to apply to such problems given the specific |
| format of data is called a 'programming language binding' for |
| that data and language. Perf supports two programming language |
| bindings, one for Python and one for Perl. |
| </para> |
| |
| <informalexample> |
| <emphasis>Tying it Together:</emphasis> Language bindings for manipulating and |
| aggregating trace data are of course not a new |
| idea. One of the first projects to do this was IBM's DProbes |
| dpcc compiler, an ANSI C compiler which targeted a low-level |
| assembly language running on an in-kernel interpreter on the |
| target system. This is exactly analogous to what Sun's DTrace |
| did, except that DTrace invented its own language for the purpose. |
| Systemtap, heavily inspired by DTrace, also created its own |
| one-off language, but rather than running the product on an |
| in-kernel interpreter, created an elaborate compiler-based |
| machinery to translate its language into kernel modules written |
| in C. |
| </informalexample> |
| |
| <para> |
| Now that we have the trace data in perf.data, we can use |
| 'perf script -g' to generate a skeleton script with handlers |
| for the read/write entry/exit events we recorded: |
| <literallayout class='monospaced'> |
| root@crownbay:~# perf script -g python |
| generated Python script: perf-script.py |
| </literallayout> |
| The skeleton script simply creates a python function for each |
| event type in the perf.data file. The body of each function simply |
| prints the event name along with its parameters. For example: |
| <literallayout class='monospaced'> |
| def net__netif_rx(event_name, context, common_cpu, |
| common_secs, common_nsecs, common_pid, common_comm, |
| skbaddr, len, name): |
| print_header(event_name, common_cpu, common_secs, common_nsecs, |
| common_pid, common_comm) |
| |
| print "skbaddr=%u, len=%u, name=%s\n" % (skbaddr, len, name), |
| </literallayout> |
| We can run that script directly to print all of the events |
| contained in the perf.data file: |
| <literallayout class='monospaced'> |
| root@crownbay:~# perf script -s perf-script.py |
| |
| in trace_begin |
| syscalls__sys_exit_read 0 11624.857082795 1262 perf nr=3, ret=0 |
| sched__sched_wakeup 0 11624.857193498 1262 perf comm=migration/0, pid=6, prio=0, success=1, target_cpu=0 |
| irq__softirq_raise 1 11624.858021635 1262 wget vec=TIMER |
| irq__softirq_entry 1 11624.858074075 1262 wget vec=TIMER |
| irq__softirq_exit 1 11624.858081389 1262 wget vec=TIMER |
| syscalls__sys_enter_read 1 11624.858166434 1262 wget nr=3, fd=3, buf=3213019456, count=512 |
| syscalls__sys_exit_read 1 11624.858177924 1262 wget nr=3, ret=512 |
| skb__kfree_skb 1 11624.858878188 1262 wget skbaddr=3945041280, location=3243922184, protocol=0 |
| skb__kfree_skb 1 11624.858945608 1262 wget skbaddr=3945037824, location=3243922184, protocol=0 |
| irq__softirq_raise 1 11624.859020942 1262 wget vec=TIMER |
| irq__softirq_entry 1 11624.859076935 1262 wget vec=TIMER |
| irq__softirq_exit 1 11624.859083469 1262 wget vec=TIMER |
| syscalls__sys_enter_read 1 11624.859167565 1262 wget nr=3, fd=3, buf=3077701632, count=1024 |
| syscalls__sys_exit_read 1 11624.859192533 1262 wget nr=3, ret=471 |
| syscalls__sys_enter_read 1 11624.859228072 1262 wget nr=3, fd=3, buf=3077701632, count=1024 |
| syscalls__sys_exit_read 1 11624.859233707 1262 wget nr=3, ret=0 |
| syscalls__sys_enter_read 1 11624.859573008 1262 wget nr=3, fd=3, buf=3213018496, count=512 |
| syscalls__sys_exit_read 1 11624.859584818 1262 wget nr=3, ret=512 |
| syscalls__sys_enter_read 1 11624.859864562 1262 wget nr=3, fd=3, buf=3077701632, count=1024 |
| syscalls__sys_exit_read 1 11624.859888770 1262 wget nr=3, ret=1024 |
| syscalls__sys_enter_read 1 11624.859935140 1262 wget nr=3, fd=3, buf=3077701632, count=1024 |
| syscalls__sys_exit_read 1 11624.859944032 1262 wget nr=3, ret=1024 |
| </literallayout> |
| That in itself isn't very useful; after all, we can accomplish |
| pretty much the same thing by simply running 'perf script' |
| without arguments in the same directory as the perf.data file. |
| </para> |
| |
| <para> |
| We can however replace the print statements in the generated |
| function bodies with whatever we want, and thereby make it |
| infinitely more useful. |
| </para> |
| |
| <para> |
| As a simple example, let's just replace the print statements in |
| the function bodies with a simple function that does nothing but |
| increment a per-event count. When the program is run against a |
| perf.data file, each time a particular event is encountered, |
| a tally is incremented for that event. For example: |
| <literallayout class='monospaced'> |
| def net__netif_rx(event_name, context, common_cpu, |
| common_secs, common_nsecs, common_pid, common_comm, |
| skbaddr, len, name): |
| inc_counts(event_name) |
| </literallayout> |
| Each event handler function in the generated code is modified |
| to do this. For convenience, we define a common function called |
| inc_counts() that each handler calls; inc_counts() simply tallies |
| a count for each event using the 'counts' hash, which is a |
| specialized hash function that does Perl-like autovivification, a |
| capability that's extremely useful for kinds of multi-level |
| aggregation commonly used in processing traces (see perf's |
| documentation on the Python language binding for details): |
| <literallayout class='monospaced'> |
| counts = autodict() |
| |
| def inc_counts(event_name): |
| try: |
| counts[event_name] += 1 |
| except TypeError: |
| counts[event_name] = 1 |
| </literallayout> |
| Finally, at the end of the trace processing run, we want to |
| print the result of all the per-event tallies. For that, we |
| use the special 'trace_end()' function: |
| <literallayout class='monospaced'> |
| def trace_end(): |
| for event_name, count in counts.iteritems(): |
| print "%-40s %10s\n" % (event_name, count) |
| </literallayout> |
| The end result is a summary of all the events recorded in the |
| trace: |
| <literallayout class='monospaced'> |
| skb__skb_copy_datagram_iovec 13148 |
| irq__softirq_entry 4796 |
| irq__irq_handler_exit 3805 |
| irq__softirq_exit 4795 |
| syscalls__sys_enter_write 8990 |
| net__net_dev_xmit 652 |
| skb__kfree_skb 4047 |
| sched__sched_wakeup 1155 |
| irq__irq_handler_entry 3804 |
| irq__softirq_raise 4799 |
| net__net_dev_queue 652 |
| syscalls__sys_enter_read 17599 |
| net__netif_receive_skb 1743 |
| syscalls__sys_exit_read 17598 |
| net__netif_rx 2 |
| napi__napi_poll 1877 |
| syscalls__sys_exit_write 8990 |
| </literallayout> |
| Note that this is pretty much exactly the same information we get |
| from 'perf stat', which goes a little way to support the idea |
| mentioned previously that given the right kind of trace data, |
| higher-level profiling-type summaries can be derived from it. |
| </para> |
| |
| <para> |
| Documentation on using the |
| <ulink url='http://linux.die.net/man/1/perf-script-python'>'perf script' python binding</ulink>. |
| </para> |
| </section> |
| |
| <section id='system-wide-tracing-and-profiling'> |
| <title>System-Wide Tracing and Profiling</title> |
| |
| <para> |
| The examples so far have focused on tracing a particular program or |
| workload - in other words, every profiling run has specified the |
| program to profile in the command-line e.g. 'perf record wget ...'. |
| </para> |
| |
| <para> |
| It's also possible, and more interesting in many cases, to run a |
| system-wide profile or trace while running the workload in a |
| separate shell. |
| </para> |
| |
| <para> |
| To do system-wide profiling or tracing, you typically use |
| the -a flag to 'perf record'. |
| </para> |
| |
| <para> |
| To demonstrate this, open up one window and start the profile |
| using the -a flag (press Ctrl-C to stop tracing): |
| <literallayout class='monospaced'> |
| root@crownbay:~# perf record -g -a |
| ^C[ perf record: Woken up 6 times to write data ] |
| [ perf record: Captured and wrote 1.400 MB perf.data (~61172 samples) ] |
| </literallayout> |
| In another window, run the wget test: |
| <literallayout class='monospaced'> |
| root@crownbay:~# wget <ulink url='http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2'>http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2</ulink> |
| Connecting to downloads.yoctoproject.org (140.211.169.59:80) |
| linux-2.6.19.2.tar.b 100% |*******************************| 41727k 0:00:00 ETA |
| </literallayout> |
| Here we see entries not only for our wget load, but for other |
| processes running on the system as well: |
| </para> |
| |
| <para> |
| <imagedata fileref="figures/perf-systemwide.png" width="6in" depth="7in" align="center" scalefit="1" /> |
| </para> |
| |
| <para> |
| In the snapshot above, we can see callchains that originate in |
| libc, and a callchain from Xorg that demonstrates that we're |
| using a proprietary X driver in userspace (notice the presence |
| of 'PVR' and some other unresolvable symbols in the expanded |
| Xorg callchain). |
| </para> |
| |
| <para> |
| Note also that we have both kernel and userspace entries in the |
| above snapshot. We can also tell perf to focus on userspace but |
| providing a modifier, in this case 'u', to the 'cycles' hardware |
| counter when we record a profile: |
| <literallayout class='monospaced'> |
| root@crownbay:~# perf record -g -a -e cycles:u |
| ^C[ perf record: Woken up 2 times to write data ] |
| [ perf record: Captured and wrote 0.376 MB perf.data (~16443 samples) ] |
| </literallayout> |
| </para> |
| |
| <para> |
| <imagedata fileref="figures/perf-report-cycles-u.png" width="6in" depth="7in" align="center" scalefit="1" /> |
| </para> |
| |
| <para> |
| Notice in the screenshot above, we see only userspace entries ([.]) |
| </para> |
| |
| <para> |
| Finally, we can press 'enter' on a leaf node and select the 'Zoom |
| into DSO' menu item to show only entries associated with a |
| specific DSO. In the screenshot below, we've zoomed into the |
| 'libc' DSO which shows all the entries associated with the |
| libc-xxx.so DSO. |
| </para> |
| |
| <para> |
| <imagedata fileref="figures/perf-systemwide-libc.png" width="6in" depth="7in" align="center" scalefit="1" /> |
| </para> |
| |
| <para> |
| We can also use the system-wide -a switch to do system-wide |
| tracing. Here we'll trace a couple of scheduler events: |
| <literallayout class='monospaced'> |
| root@crownbay:~# perf record -a -e sched:sched_switch -e sched:sched_wakeup |
| ^C[ perf record: Woken up 38 times to write data ] |
| [ perf record: Captured and wrote 9.780 MB perf.data (~427299 samples) ] |
| </literallayout> |
| We can look at the raw output using 'perf script' with no |
| arguments: |
| <literallayout class='monospaced'> |
| root@crownbay:~# perf script |
| |
| perf 1383 [001] 6171.460045: sched_wakeup: comm=kworker/1:1 pid=21 prio=120 success=1 target_cpu=001 |
| perf 1383 [001] 6171.460066: sched_switch: prev_comm=perf prev_pid=1383 prev_prio=120 prev_state=R+ ==> next_comm=kworker/1:1 next_pid=21 next_prio=120 |
| kworker/1:1 21 [001] 6171.460093: sched_switch: prev_comm=kworker/1:1 prev_pid=21 prev_prio=120 prev_state=S ==> next_comm=perf next_pid=1383 next_prio=120 |
| swapper 0 [000] 6171.468063: sched_wakeup: comm=kworker/0:3 pid=1209 prio=120 success=1 target_cpu=000 |
| swapper 0 [000] 6171.468107: sched_switch: prev_comm=swapper/0 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=kworker/0:3 next_pid=1209 next_prio=120 |
| kworker/0:3 1209 [000] 6171.468143: sched_switch: prev_comm=kworker/0:3 prev_pid=1209 prev_prio=120 prev_state=S ==> next_comm=swapper/0 next_pid=0 next_prio=120 |
| perf 1383 [001] 6171.470039: sched_wakeup: comm=kworker/1:1 pid=21 prio=120 success=1 target_cpu=001 |
| perf 1383 [001] 6171.470058: sched_switch: prev_comm=perf prev_pid=1383 prev_prio=120 prev_state=R+ ==> next_comm=kworker/1:1 next_pid=21 next_prio=120 |
| kworker/1:1 21 [001] 6171.470082: sched_switch: prev_comm=kworker/1:1 prev_pid=21 prev_prio=120 prev_state=S ==> next_comm=perf next_pid=1383 next_prio=120 |
| perf 1383 [001] 6171.480035: sched_wakeup: comm=kworker/1:1 pid=21 prio=120 success=1 target_cpu=001 |
| </literallayout> |
| </para> |
| |
| <section id='perf-filtering'> |
| <title>Filtering</title> |
| |
| <para> |
| Notice that there are a lot of events that don't really have |
| anything to do with what we're interested in, namely events |
| that schedule 'perf' itself in and out or that wake perf up. |
| We can get rid of those by using the '--filter' option - |
| for each event we specify using -e, we can add a --filter |
| after that to filter out trace events that contain fields |
| with specific values: |
| <literallayout class='monospaced'> |
| root@crownbay:~# perf record -a -e sched:sched_switch --filter 'next_comm != perf && prev_comm != perf' -e sched:sched_wakeup --filter 'comm != perf' |
| ^C[ perf record: Woken up 38 times to write data ] |
| [ perf record: Captured and wrote 9.688 MB perf.data (~423279 samples) ] |
| |
| |
| root@crownbay:~# perf script |
| |
| swapper 0 [000] 7932.162180: sched_switch: prev_comm=swapper/0 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=kworker/0:3 next_pid=1209 next_prio=120 |
| kworker/0:3 1209 [000] 7932.162236: sched_switch: prev_comm=kworker/0:3 prev_pid=1209 prev_prio=120 prev_state=S ==> next_comm=swapper/0 next_pid=0 next_prio=120 |
| perf 1407 [001] 7932.170048: sched_wakeup: comm=kworker/1:1 pid=21 prio=120 success=1 target_cpu=001 |
| perf 1407 [001] 7932.180044: sched_wakeup: comm=kworker/1:1 pid=21 prio=120 success=1 target_cpu=001 |
| perf 1407 [001] 7932.190038: sched_wakeup: comm=kworker/1:1 pid=21 prio=120 success=1 target_cpu=001 |
| perf 1407 [001] 7932.200044: sched_wakeup: comm=kworker/1:1 pid=21 prio=120 success=1 target_cpu=001 |
| perf 1407 [001] 7932.210044: sched_wakeup: comm=kworker/1:1 pid=21 prio=120 success=1 target_cpu=001 |
| perf 1407 [001] 7932.220044: sched_wakeup: comm=kworker/1:1 pid=21 prio=120 success=1 target_cpu=001 |
| swapper 0 [001] 7932.230111: sched_wakeup: comm=kworker/1:1 pid=21 prio=120 success=1 target_cpu=001 |
| swapper 0 [001] 7932.230146: sched_switch: prev_comm=swapper/1 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=kworker/1:1 next_pid=21 next_prio=120 |
| kworker/1:1 21 [001] 7932.230205: sched_switch: prev_comm=kworker/1:1 prev_pid=21 prev_prio=120 prev_state=S ==> next_comm=swapper/1 next_pid=0 next_prio=120 |
| swapper 0 [000] 7932.326109: sched_wakeup: comm=kworker/0:3 pid=1209 prio=120 success=1 target_cpu=000 |
| swapper 0 [000] 7932.326171: sched_switch: prev_comm=swapper/0 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=kworker/0:3 next_pid=1209 next_prio=120 |
| kworker/0:3 1209 [000] 7932.326214: sched_switch: prev_comm=kworker/0:3 prev_pid=1209 prev_prio=120 prev_state=S ==> next_comm=swapper/0 next_pid=0 next_prio=120 |
| </literallayout> |
| In this case, we've filtered out all events that have 'perf' |
| in their 'comm' or 'comm_prev' or 'comm_next' fields. Notice |
| that there are still events recorded for perf, but notice |
| that those events don't have values of 'perf' for the filtered |
| fields. To completely filter out anything from perf will |
| require a bit more work, but for the purpose of demonstrating |
| how to use filters, it's close enough. |
| </para> |
| |
| <informalexample> |
| <emphasis>Tying it Together:</emphasis> These are exactly the same set of event |
| filters defined by the trace event subsystem. See the |
| ftrace/tracecmd/kernelshark section for more discussion about |
| these event filters. |
| </informalexample> |
| |
| <informalexample> |
| <emphasis>Tying it Together:</emphasis> These event filters are implemented by a |
| special-purpose pseudo-interpreter in the kernel and are an |
| integral and indispensable part of the perf design as it |
| relates to tracing. kernel-based event filters provide a |
| mechanism to precisely throttle the event stream that appears |
| in user space, where it makes sense to provide bindings to real |
| programming languages for postprocessing the event stream. |
| This architecture allows for the intelligent and flexible |
| partitioning of processing between the kernel and user space. |
| Contrast this with other tools such as SystemTap, which does |
| all of its processing in the kernel and as such requires a |
| special project-defined language in order to accommodate that |
| design, or LTTng, where everything is sent to userspace and |
| as such requires a super-efficient kernel-to-userspace |
| transport mechanism in order to function properly. While |
| perf certainly can benefit from for instance advances in |
| the design of the transport, it doesn't fundamentally depend |
| on them. Basically, if you find that your perf tracing |
| application is causing buffer I/O overruns, it probably |
| means that you aren't taking enough advantage of the |
| kernel filtering engine. |
| </informalexample> |
| </section> |
| </section> |
| |
| <section id='using-dynamic-tracepoints'> |
| <title>Using Dynamic Tracepoints</title> |
| |
| <para> |
| perf isn't restricted to the fixed set of static tracepoints |
| listed by 'perf list'. Users can also add their own 'dynamic' |
| tracepoints anywhere in the kernel. For instance, suppose we |
| want to define our own tracepoint on do_fork(). We can do that |
| using the 'perf probe' perf subcommand: |
| <literallayout class='monospaced'> |
| root@crownbay:~# perf probe do_fork |
| Added new event: |
| probe:do_fork (on do_fork) |
| |
| You can now use it in all perf tools, such as: |
| |
| perf record -e probe:do_fork -aR sleep 1 |
| </literallayout> |
| Adding a new tracepoint via 'perf probe' results in an event |
| with all the expected files and format in |
| /sys/kernel/debug/tracing/events, just the same as for static |
| tracepoints (as discussed in more detail in the trace events |
| subsystem section: |
| <literallayout class='monospaced'> |
| root@crownbay:/sys/kernel/debug/tracing/events/probe/do_fork# ls -al |
| drwxr-xr-x 2 root root 0 Oct 28 11:42 . |
| drwxr-xr-x 3 root root 0 Oct 28 11:42 .. |
| -rw-r--r-- 1 root root 0 Oct 28 11:42 enable |
| -rw-r--r-- 1 root root 0 Oct 28 11:42 filter |
| -r--r--r-- 1 root root 0 Oct 28 11:42 format |
| -r--r--r-- 1 root root 0 Oct 28 11:42 id |
| |
| root@crownbay:/sys/kernel/debug/tracing/events/probe/do_fork# cat format |
| name: do_fork |
| ID: 944 |
| format: |
| field:unsigned short common_type; offset:0; size:2; signed:0; |
| field:unsigned char common_flags; offset:2; size:1; signed:0; |
| field:unsigned char common_preempt_count; offset:3; size:1; signed:0; |
| field:int common_pid; offset:4; size:4; signed:1; |
| field:int common_padding; offset:8; size:4; signed:1; |
| |
| field:unsigned long __probe_ip; offset:12; size:4; signed:0; |
| |
| print fmt: "(%lx)", REC->__probe_ip |
| </literallayout> |
| We can list all dynamic tracepoints currently in existence: |
| <literallayout class='monospaced'> |
| root@crownbay:~# perf probe -l |
| probe:do_fork (on do_fork) |
| probe:schedule (on schedule) |
| </literallayout> |
| Let's record system-wide ('sleep 30' is a trick for recording |
| system-wide but basically do nothing and then wake up after |
| 30 seconds): |
| <literallayout class='monospaced'> |
| root@crownbay:~# perf record -g -a -e probe:do_fork sleep 30 |
| [ perf record: Woken up 1 times to write data ] |
| [ perf record: Captured and wrote 0.087 MB perf.data (~3812 samples) ] |
| </literallayout> |
| Using 'perf script' we can see each do_fork event that fired: |
| <literallayout class='monospaced'> |
| root@crownbay:~# perf script |
| |
| # ======== |
| # captured on: Sun Oct 28 11:55:18 2012 |
| # hostname : crownbay |
| # os release : 3.4.11-yocto-standard |
| # perf version : 3.4.11 |
| # arch : i686 |
| # nrcpus online : 2 |
| # nrcpus avail : 2 |
| # cpudesc : Intel(R) Atom(TM) CPU E660 @ 1.30GHz |
| # cpuid : GenuineIntel,6,38,1 |
| # total memory : 1017184 kB |
| # cmdline : /usr/bin/perf record -g -a -e probe:do_fork sleep 30 |
| # event : name = probe:do_fork, type = 2, config = 0x3b0, config1 = 0x0, config2 = 0x0, excl_usr = 0, excl_kern |
| = 0, id = { 5, 6 } |
| # HEADER_CPU_TOPOLOGY info available, use -I to display |
| # ======== |
| # |
| matchbox-deskto 1197 [001] 34211.378318: do_fork: (c1028460) |
| matchbox-deskto 1295 [001] 34211.380388: do_fork: (c1028460) |
| pcmanfm 1296 [000] 34211.632350: do_fork: (c1028460) |
| pcmanfm 1296 [000] 34211.639917: do_fork: (c1028460) |
| matchbox-deskto 1197 [001] 34217.541603: do_fork: (c1028460) |
| matchbox-deskto 1299 [001] 34217.543584: do_fork: (c1028460) |
| gthumb 1300 [001] 34217.697451: do_fork: (c1028460) |
| gthumb 1300 [001] 34219.085734: do_fork: (c1028460) |
| gthumb 1300 [000] 34219.121351: do_fork: (c1028460) |
| gthumb 1300 [001] 34219.264551: do_fork: (c1028460) |
| pcmanfm 1296 [000] 34219.590380: do_fork: (c1028460) |
| matchbox-deskto 1197 [001] 34224.955965: do_fork: (c1028460) |
| matchbox-deskto 1306 [001] 34224.957972: do_fork: (c1028460) |
| matchbox-termin 1307 [000] 34225.038214: do_fork: (c1028460) |
| matchbox-termin 1307 [001] 34225.044218: do_fork: (c1028460) |
| matchbox-termin 1307 [000] 34225.046442: do_fork: (c1028460) |
| matchbox-deskto 1197 [001] 34237.112138: do_fork: (c1028460) |
| matchbox-deskto 1311 [001] 34237.114106: do_fork: (c1028460) |
| gaku 1312 [000] 34237.202388: do_fork: (c1028460) |
| </literallayout> |
| And using 'perf report' on the same file, we can see the |
| callgraphs from starting a few programs during those 30 seconds: |
| </para> |
| |
| <para> |
| <imagedata fileref="figures/perf-probe-do_fork-profile.png" width="6in" depth="7in" align="center" scalefit="1" /> |
| </para> |
| |
| <informalexample> |
| <emphasis>Tying it Together:</emphasis> The trace events subsystem accommodate static |
| and dynamic tracepoints in exactly the same way - there's no |
| difference as far as the infrastructure is concerned. See the |
| ftrace section for more details on the trace event subsystem. |
| </informalexample> |
| |
| <informalexample> |
| <emphasis>Tying it Together:</emphasis> Dynamic tracepoints are implemented under the |
| covers by kprobes and uprobes. kprobes and uprobes are also used |
| by and in fact are the main focus of SystemTap. |
| </informalexample> |
| </section> |
| </section> |
| |
| <section id='perf-documentation'> |
| <title>Documentation</title> |
| |
| <para> |
| Online versions of the man pages for the commands discussed in this |
| section can be found here: |
| <itemizedlist> |
| <listitem><para>The <ulink url='http://linux.die.net/man/1/perf-stat'>'perf stat' manpage</ulink>. |
| </para></listitem> |
| <listitem><para>The <ulink url='http://linux.die.net/man/1/perf-record'>'perf record' manpage</ulink>. |
| </para></listitem> |
| <listitem><para>The <ulink url='http://linux.die.net/man/1/perf-report'>'perf report' manpage</ulink>. |
| </para></listitem> |
| <listitem><para>The <ulink url='http://linux.die.net/man/1/perf-probe'>'perf probe' manpage</ulink>. |
| </para></listitem> |
| <listitem><para>The <ulink url='http://linux.die.net/man/1/perf-script'>'perf script' manpage</ulink>. |
| </para></listitem> |
| <listitem><para>Documentation on using the |
| <ulink url='http://linux.die.net/man/1/perf-script-python'>'perf script' python binding</ulink>. |
| </para></listitem> |
| <listitem><para>The top-level |
| <ulink url='http://linux.die.net/man/1/perf'>perf(1) manpage</ulink>. |
| </para></listitem> |
| </itemizedlist> |
| </para> |
| |
| <para> |
| Normally, you should be able to invoke the man pages via perf |
| itself e.g. 'perf help' or 'perf help record'. |
| </para> |
| |
| <para> |
| However, by default Yocto doesn't install man pages, but perf |
| invokes the man pages for most help functionality. This is a bug |
| and is being addressed by a Yocto bug: |
| <ulink url='https://bugzilla.yoctoproject.org/show_bug.cgi?id=3388'>Bug 3388 - perf: enable man pages for basic 'help' functionality</ulink>. |
| </para> |
| |
| <para> |
| The man pages in text form, along with some other files, such as |
| a set of examples, can be found in the 'perf' directory of the |
| kernel tree: |
| <literallayout class='monospaced'> |
| tools/perf/Documentation |
| </literallayout> |
| There's also a nice perf tutorial on the perf wiki that goes |
| into more detail than we do here in certain areas: |
| <ulink url='https://perf.wiki.kernel.org/index.php/Tutorial'>Perf Tutorial</ulink> |
| </para> |
| </section> |
| </section> |
| |
| <section id='profile-manual-ftrace'> |
| <title>ftrace</title> |
| |
| <para> |
| 'ftrace' literally refers to the 'ftrace function tracer' but in |
| reality this encompasses a number of related tracers along with |
| the infrastructure that they all make use of. |
| </para> |
| |
| <section id='ftrace-setup'> |
| <title>Setup</title> |
| |
| <para> |
| For this section, we'll assume you've already performed the basic |
| setup outlined in the General Setup section. |
| </para> |
| |
| <para> |
| ftrace, trace-cmd, and kernelshark run on the target system, |
| and are ready to go out-of-the-box - no additional setup is |
| necessary. For the rest of this section we assume you've ssh'ed |
| to the host and will be running ftrace on the target. kernelshark |
| is a GUI application and if you use the '-X' option to ssh you |
| can have the kernelshark GUI run on the target but display |
| remotely on the host if you want. |
| </para> |
| </section> |
| |
| <section id='basic-ftrace-usage'> |
| <title>Basic ftrace usage</title> |
| |
| <para> |
| 'ftrace' essentially refers to everything included in |
| the /tracing directory of the mounted debugfs filesystem |
| (Yocto follows the standard convention and mounts it |
| at /sys/kernel/debug). Here's a listing of all the files |
| found in /sys/kernel/debug/tracing on a Yocto system: |
| <literallayout class='monospaced'> |
| root@sugarbay:/sys/kernel/debug/tracing# ls |
| README kprobe_events trace |
| available_events kprobe_profile trace_clock |
| available_filter_functions options trace_marker |
| available_tracers per_cpu trace_options |
| buffer_size_kb printk_formats trace_pipe |
| buffer_total_size_kb saved_cmdlines tracing_cpumask |
| current_tracer set_event tracing_enabled |
| dyn_ftrace_total_info set_ftrace_filter tracing_on |
| enabled_functions set_ftrace_notrace tracing_thresh |
| events set_ftrace_pid |
| free_buffer set_graph_function |
| </literallayout> |
| The files listed above are used for various purposes - |
| some relate directly to the tracers themselves, others are |
| used to set tracing options, and yet others actually contain |
| the tracing output when a tracer is in effect. Some of the |
| functions can be guessed from their names, others need |
| explanation; in any case, we'll cover some of the files we |
| see here below but for an explanation of the others, please |
| see the ftrace documentation. |
| </para> |
| |
| <para> |
| We'll start by looking at some of the available built-in |
| tracers. |
| </para> |
| |
| <para> |
| cat'ing the 'available_tracers' file lists the set of |
| available tracers: |
| <literallayout class='monospaced'> |
| root@sugarbay:/sys/kernel/debug/tracing# cat available_tracers |
| blk function_graph function nop |
| </literallayout> |
| The 'current_tracer' file contains the tracer currently in |
| effect: |
| <literallayout class='monospaced'> |
| root@sugarbay:/sys/kernel/debug/tracing# cat current_tracer |
| nop |
| </literallayout> |
| The above listing of current_tracer shows that |
| the 'nop' tracer is in effect, which is just another |
| way of saying that there's actually no tracer |
| currently in effect. |
| </para> |
| |
| <para> |
| echo'ing one of the available_tracers into current_tracer |
| makes the specified tracer the current tracer: |
| <literallayout class='monospaced'> |
| root@sugarbay:/sys/kernel/debug/tracing# echo function > current_tracer |
| root@sugarbay:/sys/kernel/debug/tracing# cat current_tracer |
| function |
| </literallayout> |
| The above sets the current tracer to be the |
| 'function tracer'. This tracer traces every function |
| call in the kernel and makes it available as the |
| contents of the 'trace' file. Reading the 'trace' file |
| lists the currently buffered function calls that have been |
| traced by the function tracer: |
| <literallayout class='monospaced'> |
| root@sugarbay:/sys/kernel/debug/tracing# cat trace | less |
| |
| # tracer: function |
| # |
| # entries-in-buffer/entries-written: 310629/766471 #P:8 |
| # |
| # _-----=> irqs-off |
| # / _----=> need-resched |
| # | / _---=> hardirq/softirq |
| # || / _--=> preempt-depth |
| # ||| / delay |
| # TASK-PID CPU# |||| TIMESTAMP FUNCTION |
| # | | | |||| | | |
| <idle>-0 [004] d..1 470.867169: ktime_get_real <-intel_idle |
| <idle>-0 [004] d..1 470.867170: getnstimeofday <-ktime_get_real |
| <idle>-0 [004] d..1 470.867171: ns_to_timeval <-intel_idle |
| <idle>-0 [004] d..1 470.867171: ns_to_timespec <-ns_to_timeval |
| <idle>-0 [004] d..1 470.867172: smp_apic_timer_interrupt <-apic_timer_interrupt |
| <idle>-0 [004] d..1 470.867172: native_apic_mem_write <-smp_apic_timer_interrupt |
| <idle>-0 [004] d..1 470.867172: irq_enter <-smp_apic_timer_interrupt |
| <idle>-0 [004] d..1 470.867172: rcu_irq_enter <-irq_enter |
| <idle>-0 [004] d..1 470.867173: rcu_idle_exit_common.isra.33 <-rcu_irq_enter |
| <idle>-0 [004] d..1 470.867173: local_bh_disable <-irq_enter |
| <idle>-0 [004] d..1 470.867173: add_preempt_count <-local_bh_disable |
| <idle>-0 [004] d.s1 470.867174: tick_check_idle <-irq_enter |
| <idle>-0 [004] d.s1 470.867174: tick_check_oneshot_broadcast <-tick_check_idle |
| <idle>-0 [004] d.s1 470.867174: ktime_get <-tick_check_idle |
| <idle>-0 [004] d.s1 470.867174: tick_nohz_stop_idle <-tick_check_idle |
| <idle>-0 [004] d.s1 470.867175: update_ts_time_stats <-tick_nohz_stop_idle |
| <idle>-0 [004] d.s1 470.867175: nr_iowait_cpu <-update_ts_time_stats |
| <idle>-0 [004] d.s1 470.867175: tick_do_update_jiffies64 <-tick_check_idle |
| <idle>-0 [004] d.s1 470.867175: _raw_spin_lock <-tick_do_update_jiffies64 |
| <idle>-0 [004] d.s1 470.867176: add_preempt_count <-_raw_spin_lock |
| <idle>-0 [004] d.s2 470.867176: do_timer <-tick_do_update_jiffies64 |
| <idle>-0 [004] d.s2 470.867176: _raw_spin_lock <-do_timer |
| <idle>-0 [004] d.s2 470.867176: add_preempt_count <-_raw_spin_lock |
| <idle>-0 [004] d.s3 470.867177: ntp_tick_length <-do_timer |
| <idle>-0 [004] d.s3 470.867177: _raw_spin_lock_irqsave <-ntp_tick_length |
| . |
| . |
| . |
| </literallayout> |
| Each line in the trace above shows what was happening in |
| the kernel on a given cpu, to the level of detail of |
| function calls. Each entry shows the function called, |
| followed by its caller (after the arrow). |
| </para> |
| |
| <para> |
| The function tracer gives you an extremely detailed idea |
| of what the kernel was doing at the point in time the trace |
| was taken, and is a great way to learn about how the kernel |
| code works in a dynamic sense. |
| </para> |
| |
| <informalexample> |
| <emphasis>Tying it Together:</emphasis> The ftrace function tracer is also |
| available from within perf, as the ftrace:function tracepoint. |
| </informalexample> |
| |
| <para> |
| It is a little more difficult to follow the call chains than |
| it needs to be - luckily there's a variant of the function |
| tracer that displays the callchains explicitly, called the |
| 'function_graph' tracer: |
| <literallayout class='monospaced'> |
| root@sugarbay:/sys/kernel/debug/tracing# echo function_graph > current_tracer |
| root@sugarbay:/sys/kernel/debug/tracing# cat trace | less |
| |
| tracer: function_graph |
| |
| CPU DURATION FUNCTION CALLS |
| | | | | | | | |
| 7) 0.046 us | pick_next_task_fair(); |
| 7) 0.043 us | pick_next_task_stop(); |
| 7) 0.042 us | pick_next_task_rt(); |
| 7) 0.032 us | pick_next_task_fair(); |
| 7) 0.030 us | pick_next_task_idle(); |
| 7) | _raw_spin_unlock_irq() { |
| 7) 0.033 us | sub_preempt_count(); |
| 7) 0.258 us | } |
| 7) 0.032 us | sub_preempt_count(); |
| 7) + 13.341 us | } /* __schedule */ |
| 7) 0.095 us | } /* sub_preempt_count */ |
| 7) | schedule() { |
| 7) | __schedule() { |
| 7) 0.060 us | add_preempt_count(); |
| 7) 0.044 us | rcu_note_context_switch(); |
| 7) | _raw_spin_lock_irq() { |
| 7) 0.033 us | add_preempt_count(); |
| 7) 0.247 us | } |
| 7) | idle_balance() { |
| 7) | _raw_spin_unlock() { |
| 7) 0.031 us | sub_preempt_count(); |
| 7) 0.246 us | } |
| 7) | update_shares() { |
| 7) 0.030 us | __rcu_read_lock(); |
| 7) 0.029 us | __rcu_read_unlock(); |
| 7) 0.484 us | } |
| 7) 0.030 us | __rcu_read_lock(); |
| 7) | load_balance() { |
| 7) | find_busiest_group() { |
| 7) 0.031 us | idle_cpu(); |
| 7) 0.029 us | idle_cpu(); |
| 7) 0.035 us | idle_cpu(); |
| 7) 0.906 us | } |
| 7) 1.141 us | } |
| 7) 0.022 us | msecs_to_jiffies(); |
| 7) | load_balance() { |
| 7) | find_busiest_group() { |
| 7) 0.031 us | idle_cpu(); |
| . |
| . |
| . |
| 4) 0.062 us | msecs_to_jiffies(); |
| 4) 0.062 us | __rcu_read_unlock(); |
| 4) | _raw_spin_lock() { |
| 4) 0.073 us | add_preempt_count(); |
| 4) 0.562 us | } |
| 4) + 17.452 us | } |
| 4) 0.108 us | put_prev_task_fair(); |
| 4) 0.102 us | pick_next_task_fair(); |
| 4) 0.084 us | pick_next_task_stop(); |
| 4) 0.075 us | pick_next_task_rt(); |
| 4) 0.062 us | pick_next_task_fair(); |
| 4) 0.066 us | pick_next_task_idle(); |
| ------------------------------------------ |
| 4) kworker-74 => <idle>-0 |
| ------------------------------------------ |
| |
| 4) | finish_task_switch() { |
| 4) | _raw_spin_unlock_irq() { |
| 4) 0.100 us | sub_preempt_count(); |
| 4) 0.582 us | } |
| 4) 1.105 us | } |
| 4) 0.088 us | sub_preempt_count(); |
| 4) ! 100.066 us | } |
| . |
| . |
| . |
| 3) | sys_ioctl() { |
| 3) 0.083 us | fget_light(); |
| 3) | security_file_ioctl() { |
| 3) 0.066 us | cap_file_ioctl(); |
| 3) 0.562 us | } |
| 3) | do_vfs_ioctl() { |
| 3) | drm_ioctl() { |
| 3) 0.075 us | drm_ut_debug_printk(); |
| 3) | i915_gem_pwrite_ioctl() { |
| 3) | i915_mutex_lock_interruptible() { |
| 3) 0.070 us | mutex_lock_interruptible(); |
| 3) 0.570 us | } |
| 3) | drm_gem_object_lookup() { |
| 3) | _raw_spin_lock() { |
| 3) 0.080 us | add_preempt_count(); |
| 3) 0.620 us | } |
| 3) | _raw_spin_unlock() { |
| 3) 0.085 us | sub_preempt_count(); |
| 3) 0.562 us | } |
| 3) 2.149 us | } |
| 3) 0.133 us | i915_gem_object_pin(); |
| 3) | i915_gem_object_set_to_gtt_domain() { |
| 3) 0.065 us | i915_gem_object_flush_gpu_write_domain(); |
| 3) 0.065 us | i915_gem_object_wait_rendering(); |
| 3) 0.062 us | i915_gem_object_flush_cpu_write_domain(); |
| 3) 1.612 us | } |
| 3) | i915_gem_object_put_fence() { |
| 3) 0.097 us | i915_gem_object_flush_fence.constprop.36(); |
| 3) 0.645 us | } |
| 3) 0.070 us | add_preempt_count(); |
| 3) 0.070 us | sub_preempt_count(); |
| 3) 0.073 us | i915_gem_object_unpin(); |
| 3) 0.068 us | mutex_unlock(); |
| 3) 9.924 us | } |
| 3) + 11.236 us | } |
| 3) + 11.770 us | } |
| 3) + 13.784 us | } |
| 3) | sys_ioctl() { |
| </literallayout> |
| As you can see, the function_graph display is much easier to |
| follow. Also note that in addition to the function calls and |
| associated braces, other events such as scheduler events |
| are displayed in context. In fact, you can freely include |
| any tracepoint available in the trace events subsystem described |
| in the next section by simply enabling those events, and they'll |
| appear in context in the function graph display. Quite a |
| powerful tool for understanding kernel dynamics. |
| </para> |
| |
| <para> |
| Also notice that there are various annotations on the left |
| hand side of the display. For example if the total time it |
| took for a given function to execute is above a certain |
| threshold, an exclamation point or plus sign appears on the |
| left hand side. Please see the ftrace documentation for |
| details on all these fields. |
| </para> |
| </section> |
| |
| <section id='the-trace-events-subsystem'> |
| <title>The 'trace events' Subsystem</title> |
| |
| <para> |
| One especially important directory contained within |
| the /sys/kernel/debug/tracing directory is the 'events' |
| subdirectory, which contains representations of every |
| tracepoint in the system. Listing out the contents of |
| the 'events' subdirectory, we see mainly another set of |
| subdirectories: |
| <literallayout class='monospaced'> |
| root@sugarbay:/sys/kernel/debug/tracing# cd events |
| root@sugarbay:/sys/kernel/debug/tracing/events# ls -al |
| drwxr-xr-x 38 root root 0 Nov 14 23:19 . |
| drwxr-xr-x 5 root root 0 Nov 14 23:19 .. |
| drwxr-xr-x 19 root root 0 Nov 14 23:19 block |
| drwxr-xr-x 32 root root 0 Nov 14 23:19 btrfs |
| drwxr-xr-x 5 root root 0 Nov 14 23:19 drm |
| -rw-r--r-- 1 root root 0 Nov 14 23:19 enable |
| drwxr-xr-x 40 root root 0 Nov 14 23:19 ext3 |
| drwxr-xr-x 79 root root 0 Nov 14 23:19 ext4 |
| drwxr-xr-x 14 root root 0 Nov 14 23:19 ftrace |
| drwxr-xr-x 8 root root 0 Nov 14 23:19 hda |
| -r--r--r-- 1 root root 0 Nov 14 23:19 header_event |
| -r--r--r-- 1 root root 0 Nov 14 23:19 header_page |
| drwxr-xr-x 25 root root 0 Nov 14 23:19 i915 |
| drwxr-xr-x 7 root root 0 Nov 14 23:19 irq |
| drwxr-xr-x 12 root root 0 Nov 14 23:19 jbd |
| drwxr-xr-x 14 root root 0 Nov 14 23:19 jbd2 |
| drwxr-xr-x 14 root root 0 Nov 14 23:19 kmem |
| drwxr-xr-x 7 root root 0 Nov 14 23:19 module |
| drwxr-xr-x 3 root root 0 Nov 14 23:19 napi |
| drwxr-xr-x 6 root root 0 Nov 14 23:19 net |
| drwxr-xr-x 3 root root 0 Nov 14 23:19 oom |
| drwxr-xr-x 12 root root 0 Nov 14 23:19 power |
| drwxr-xr-x 3 root root 0 Nov 14 23:19 printk |
| drwxr-xr-x 8 root root 0 Nov 14 23:19 random |
| drwxr-xr-x 4 root root 0 Nov 14 23:19 raw_syscalls |
| drwxr-xr-x 3 root root 0 Nov 14 23:19 rcu |
| drwxr-xr-x 6 root root 0 Nov 14 23:19 rpm |
| drwxr-xr-x 20 root root 0 Nov 14 23:19 sched |
| drwxr-xr-x 7 root root 0 Nov 14 23:19 scsi |
| drwxr-xr-x 4 root root 0 Nov 14 23:19 signal |
| drwxr-xr-x 5 root root 0 Nov 14 23:19 skb |
| drwxr-xr-x 4 root root 0 Nov 14 23:19 sock |
| drwxr-xr-x 10 root root 0 Nov 14 23:19 sunrpc |
| drwxr-xr-x 538 root root 0 Nov 14 23:19 syscalls |
| drwxr-xr-x 4 root root 0 Nov 14 23:19 task |
| drwxr-xr-x 14 root root 0 Nov 14 23:19 timer |
| drwxr-xr-x 3 root root 0 Nov 14 23:19 udp |
| drwxr-xr-x 21 root root 0 Nov 14 23:19 vmscan |
| drwxr-xr-x 3 root root 0 Nov 14 23:19 vsyscall |
| drwxr-xr-x 6 root root 0 Nov 14 23:19 workqueue |
| drwxr-xr-x 26 root root 0 Nov 14 23:19 writeback |
| </literallayout> |
| Each one of these subdirectories corresponds to a |
| 'subsystem' and contains yet again more subdirectories, |
| each one of those finally corresponding to a tracepoint. |
| For example, here are the contents of the 'kmem' subsystem: |
| <literallayout class='monospaced'> |
| root@sugarbay:/sys/kernel/debug/tracing/events# cd kmem |
| root@sugarbay:/sys/kernel/debug/tracing/events/kmem# ls -al |
| drwxr-xr-x 14 root root 0 Nov 14 23:19 . |
| drwxr-xr-x 38 root root 0 Nov 14 23:19 .. |
| -rw-r--r-- 1 root root 0 Nov 14 23:19 enable |
| -rw-r--r-- 1 root root 0 Nov 14 23:19 filter |
| drwxr-xr-x 2 root root 0 Nov 14 23:19 kfree |
| drwxr-xr-x 2 root root 0 Nov 14 23:19 kmalloc |
| drwxr-xr-x 2 root root 0 Nov 14 23:19 kmalloc_node |
| drwxr-xr-x 2 root root 0 Nov 14 23:19 kmem_cache_alloc |
| drwxr-xr-x 2 root root 0 Nov 14 23:19 kmem_cache_alloc_node |
| drwxr-xr-x 2 root root 0 Nov 14 23:19 kmem_cache_free |
| drwxr-xr-x 2 root root 0 Nov 14 23:19 mm_page_alloc |
| drwxr-xr-x 2 root root 0 Nov 14 23:19 mm_page_alloc_extfrag |
| drwxr-xr-x 2 root root 0 Nov 14 23:19 mm_page_alloc_zone_locked |
| drwxr-xr-x 2 root root 0 Nov 14 23:19 mm_page_free |
| drwxr-xr-x 2 root root 0 Nov 14 23:19 mm_page_free_batched |
| drwxr-xr-x 2 root root 0 Nov 14 23:19 mm_page_pcpu_drain |
| </literallayout> |
| Let's see what's inside the subdirectory for a specific |
| tracepoint, in this case the one for kmalloc: |
| <literallayout class='monospaced'> |
| root@sugarbay:/sys/kernel/debug/tracing/events/kmem# cd kmalloc |
| root@sugarbay:/sys/kernel/debug/tracing/events/kmem/kmalloc# ls -al |
| drwxr-xr-x 2 root root 0 Nov 14 23:19 . |
| drwxr-xr-x 14 root root 0 Nov 14 23:19 .. |
| -rw-r--r-- 1 root root 0 Nov 14 23:19 enable |
| -rw-r--r-- 1 root root 0 Nov 14 23:19 filter |
| -r--r--r-- 1 root root 0 Nov 14 23:19 format |
| -r--r--r-- 1 root root 0 Nov 14 23:19 id |
| </literallayout> |
| The 'format' file for the tracepoint describes the event |
| in memory, which is used by the various tracing tools |
| that now make use of these tracepoint to parse the event |
| and make sense of it, along with a 'print fmt' field that |
| allows tools like ftrace to display the event as text. |
| Here's what the format of the kmalloc event looks like: |
| <literallayout class='monospaced'> |
| root@sugarbay:/sys/kernel/debug/tracing/events/kmem/kmalloc# cat format |
| name: kmalloc |
| ID: 313 |
| format: |
| field:unsigned short common_type; offset:0; size:2; signed:0; |
| field:unsigned char common_flags; offset:2; size:1; signed:0; |
| field:unsigned char common_preempt_count; offset:3; size:1; signed:0; |
| field:int common_pid; offset:4; size:4; signed:1; |
| field:int common_padding; offset:8; size:4; signed:1; |
| |
| field:unsigned long call_site; offset:16; size:8; signed:0; |
| field:const void * ptr; offset:24; size:8; signed:0; |
| field:size_t bytes_req; offset:32; size:8; signed:0; |
| field:size_t bytes_alloc; offset:40; size:8; signed:0; |
| field:gfp_t gfp_flags; offset:48; size:4; signed:0; |
| |
| print fmt: "call_site=%lx ptr=%p bytes_req=%zu bytes_alloc=%zu gfp_flags=%s", REC->call_site, REC->ptr, REC->bytes_req, REC->bytes_alloc, |
| (REC->gfp_flags) ? __print_flags(REC->gfp_flags, "|", {(unsigned long)(((( gfp_t)0x10u) | (( gfp_t)0x40u) | (( gfp_t)0x80u) | (( |
| gfp_t)0x20000u) | (( gfp_t)0x02u) | (( gfp_t)0x08u)) | (( gfp_t)0x4000u) | (( gfp_t)0x10000u) | (( gfp_t)0x1000u) | (( gfp_t)0x200u) | (( |
| gfp_t)0x400000u)), "GFP_TRANSHUGE"}, {(unsigned long)((( gfp_t)0x10u) | (( gfp_t)0x40u) | (( gfp_t)0x80u) | (( gfp_t)0x20000u) | (( |
| gfp_t)0x02u) | (( gfp_t)0x08u)), "GFP_HIGHUSER_MOVABLE"}, {(unsigned long)((( gfp_t)0x10u) | (( gfp_t)0x40u) | (( gfp_t)0x80u) | (( |
| gfp_t)0x20000u) | (( gfp_t)0x02u)), "GFP_HIGHUSER"}, {(unsigned long)((( gfp_t)0x10u) | (( gfp_t)0x40u) | (( gfp_t)0x80u) | (( |
| gfp_t)0x20000u)), "GFP_USER"}, {(unsigned long)((( gfp_t)0x10u) | (( gfp_t)0x40u) | (( gfp_t)0x80u) | (( gfp_t)0x80000u)), GFP_TEMPORARY"}, |
| {(unsigned long)((( gfp_t)0x10u) | (( gfp_t)0x40u) | (( gfp_t)0x80u)), "GFP_KERNEL"}, {(unsigned long)((( gfp_t)0x10u) | (( gfp_t)0x40u)), |
| "GFP_NOFS"}, {(unsigned long)((( gfp_t)0x20u)), "GFP_ATOMIC"}, {(unsigned long)((( gfp_t)0x10u)), "GFP_NOIO"}, {(unsigned long)(( |
| gfp_t)0x20u), "GFP_HIGH"}, {(unsigned long)(( gfp_t)0x10u), "GFP_WAIT"}, {(unsigned long)(( gfp_t)0x40u), "GFP_IO"}, {(unsigned long)(( |
| gfp_t)0x100u), "GFP_COLD"}, {(unsigned long)(( gfp_t)0x200u), "GFP_NOWARN"}, {(unsigned long)(( gfp_t)0x400u), "GFP_REPEAT"}, {(unsigned |
| long)(( gfp_t)0x800u), "GFP_NOFAIL"}, {(unsigned long)(( gfp_t)0x1000u), "GFP_NORETRY"}, {(unsigned long)(( gfp_t)0x4000u), "GFP_COMP"}, |
| {(unsigned long)(( gfp_t)0x8000u), "GFP_ZERO"}, {(unsigned long)(( gfp_t)0x10000u), "GFP_NOMEMALLOC"}, {(unsigned long)(( gfp_t)0x20000u), |
| "GFP_HARDWALL"}, {(unsigned long)(( gfp_t)0x40000u), "GFP_THISNODE"}, {(unsigned long)(( gfp_t)0x80000u), "GFP_RECLAIMABLE"}, {(unsigned |
| long)(( gfp_t)0x08u), "GFP_MOVABLE"}, {(unsigned long)(( gfp_t)0), "GFP_NOTRACK"}, {(unsigned long)(( gfp_t)0x400000u), "GFP_NO_KSWAPD"}, |
| {(unsigned long)(( gfp_t)0x800000u), "GFP_OTHER_NODE"} ) : "GFP_NOWAIT" |
| </literallayout> |
| The 'enable' file in the tracepoint directory is what allows |
| the user (or tools such as trace-cmd) to actually turn the |
| tracepoint on and off. When enabled, the corresponding |
| tracepoint will start appearing in the ftrace 'trace' |
| file described previously. For example, this turns on the |
| kmalloc tracepoint: |
| <literallayout class='monospaced'> |
| root@sugarbay:/sys/kernel/debug/tracing/events/kmem/kmalloc# echo 1 > enable |
| </literallayout> |
| At the moment, we're not interested in the function tracer or |
| some other tracer that might be in effect, so we first turn |
| it off, but if we do that, we still need to turn tracing on in |
| order to see the events in the output buffer: |
| <literallayout class='monospaced'> |
| root@sugarbay:/sys/kernel/debug/tracing# echo nop > current_tracer |
| root@sugarbay:/sys/kernel/debug/tracing# echo 1 > tracing_on |
| </literallayout> |
| Now, if we look at the the 'trace' file, we see nothing |
| but the kmalloc events we just turned on: |
| <literallayout class='monospaced'> |
| root@sugarbay:/sys/kernel/debug/tracing# cat trace | less |
| # tracer: nop |
| # |
| # entries-in-buffer/entries-written: 1897/1897 #P:8 |
| # |
| # _-----=> irqs-off |
| # / _----=> need-resched |
| # | / _---=> hardirq/softirq |
| # || / _--=> preempt-depth |
| # ||| / delay |
| # TASK-PID CPU# |||| TIMESTAMP FUNCTION |
| # | | | |||| | | |
| dropbear-1465 [000] ...1 18154.620753: kmalloc: call_site=ffffffff816650d4 ptr=ffff8800729c3000 bytes_req=2048 bytes_alloc=2048 gfp_flags=GFP_KERNEL |
| <idle>-0 [000] ..s3 18154.621640: kmalloc: call_site=ffffffff81619b36 ptr=ffff88006d555800 bytes_req=512 bytes_alloc=512 gfp_flags=GFP_ATOMIC |
| <idle>-0 [000] ..s3 18154.621656: kmalloc: call_site=ffffffff81619b36 ptr=ffff88006d555800 bytes_req=512 bytes_alloc=512 gfp_flags=GFP_ATOMIC |
| matchbox-termin-1361 [001] ...1 18154.755472: kmalloc: call_site=ffffffff81614050 ptr=ffff88006d5f0e00 bytes_req=512 bytes_alloc=512 gfp_flags=GFP_KERNEL|GFP_REPEAT |
| Xorg-1264 [002] ...1 18154.755581: kmalloc: call_site=ffffffff8141abe8 ptr=ffff8800734f4cc0 bytes_req=168 bytes_alloc=192 gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY |
| Xorg-1264 [002] ...1 18154.755583: kmalloc: call_site=ffffffff814192a3 ptr=ffff88001f822520 bytes_req=24 bytes_alloc=32 gfp_flags=GFP_KERNEL|GFP_ZERO |
| Xorg-1264 [002] ...1 18154.755589: kmalloc: call_site=ffffffff81419edb ptr=ffff8800721a2f00 bytes_req=64 bytes_alloc=64 gfp_flags=GFP_KERNEL|GFP_ZERO |
| matchbox-termin-1361 [001] ...1 18155.354594: kmalloc: call_site=ffffffff81614050 ptr=ffff88006db35400 bytes_req=576 bytes_alloc=1024 gfp_flags=GFP_KERNEL|GFP_REPEAT |
| Xorg-1264 [002] ...1 18155.354703: kmalloc: call_site=ffffffff8141abe8 ptr=ffff8800734f4cc0 bytes_req=168 bytes_alloc=192 gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY |
| Xorg-1264 [002] ...1 18155.354705: kmalloc: call_site=ffffffff814192a3 ptr=ffff88001f822520 bytes_req=24 bytes_alloc=32 gfp_flags=GFP_KERNEL|GFP_ZERO |
| Xorg-1264 [002] ...1 18155.354711: kmalloc: call_site=ffffffff81419edb ptr=ffff8800721a2f00 bytes_req=64 bytes_alloc=64 gfp_flags=GFP_KERNEL|GFP_ZERO |
| <idle>-0 [000] ..s3 18155.673319: kmalloc: call_site=ffffffff81619b36 ptr=ffff88006d555800 bytes_req=512 bytes_alloc=512 gfp_flags=GFP_ATOMIC |
| dropbear-1465 [000] ...1 18155.673525: kmalloc: call_site=ffffffff816650d4 ptr=ffff8800729c3000 bytes_req=2048 bytes_alloc=2048 gfp_flags=GFP_KERNEL |
| <idle>-0 [000] ..s3 18155.674821: kmalloc: call_site=ffffffff81619b36 ptr=ffff88006d554800 bytes_req=512 bytes_alloc=512 gfp_flags=GFP_ATOMIC |
| <idle>-0 [000] ..s3 18155.793014: kmalloc: call_site=ffffffff81619b36 ptr=ffff88006d554800 bytes_req=512 bytes_alloc=512 gfp_flags=GFP_ATOMIC |
| dropbear-1465 [000] ...1 18155.793219: kmalloc: call_site=ffffffff816650d4 ptr=ffff8800729c3000 bytes_req=2048 bytes_alloc=2048 gfp_flags=GFP_KERNEL |
| <idle>-0 [000] ..s3 18155.794147: kmalloc: call_site=ffffffff81619b36 ptr=ffff88006d555800 bytes_req=512 bytes_alloc=512 gfp_flags=GFP_ATOMIC |
| <idle>-0 [000] ..s3 18155.936705: kmalloc: call_site=ffffffff81619b36 ptr=ffff88006d555800 bytes_req=512 bytes_alloc=512 gfp_flags=GFP_ATOMIC |
| dropbear-1465 [000] ...1 18155.936910: kmalloc: call_site=ffffffff816650d4 ptr=ffff8800729c3000 bytes_req=2048 bytes_alloc=2048 gfp_flags=GFP_KERNEL |
| <idle>-0 [000] ..s3 18155.937869: kmalloc: call_site=ffffffff81619b36 ptr=ffff88006d554800 bytes_req=512 bytes_alloc=512 gfp_flags=GFP_ATOMIC |
| matchbox-termin-1361 [001] ...1 18155.953667: kmalloc: call_site=ffffffff81614050 ptr=ffff88006d5f2000 bytes_req=512 bytes_alloc=512 gfp_flags=GFP_KERNEL|GFP_REPEAT |
| Xorg-1264 [002] ...1 18155.953775: kmalloc: call_site=ffffffff8141abe8 ptr=ffff8800734f4cc0 bytes_req=168 bytes_alloc=192 gfp_flags=GFP_KERNEL|GFP_NOWARN|GFP_NORETRY |
| Xorg-1264 [002] ...1 18155.953777: kmalloc: call_site=ffffffff814192a3 ptr=ffff88001f822520 bytes_req=24 bytes_alloc=32 gfp_flags=GFP_KERNEL|GFP_ZERO |
| Xorg-1264 [002] ...1 18155.953783: kmalloc: call_site=ffffffff81419edb ptr=ffff8800721a2f00 bytes_req=64 bytes_alloc=64 gfp_flags=GFP_KERNEL|GFP_ZERO |
| <idle>-0 [000] ..s3 18156.176053: kmalloc: call_site=ffffffff81619b36 ptr=ffff88006d554800 bytes_req=512 bytes_alloc=512 gfp_flags=GFP_ATOMIC |
| dropbear-1465 [000] ...1 18156.176257: kmalloc: call_site=ffffffff816650d4 ptr=ffff8800729c3000 bytes_req=2048 bytes_alloc=2048 gfp_flags=GFP_KERNEL |
| <idle>-0 [000] ..s3 18156.177717: kmalloc: call_site=ffffffff81619b36 ptr=ffff88006d555800 bytes_req=512 bytes_alloc=512 gfp_flags=GFP_ATOMIC |
| <idle>-0 [000] ..s3 18156.399229: kmalloc: call_site=ffffffff81619b36 ptr=ffff88006d555800 bytes_req=512 bytes_alloc=512 gfp_flags=GFP_ATOMIC |
| dropbear-1465 [000] ...1 18156.399434: kmalloc: call_site=ffffffff816650d4 ptr=ffff8800729c3000 bytes_http://rostedt.homelinux.com/kernelshark/req=2048 bytes_alloc=2048 gfp_flags=GFP_KERNEL |
| <idle>-0 [000] ..s3 18156.400660: kmalloc: call_site=ffffffff81619b36 ptr=ffff88006d554800 bytes_req=512 bytes_alloc=512 gfp_flags=GFP_ATOMIC |
| matchbox-termin-1361 [001] ...1 18156.552800: kmalloc: call_site=ffffffff81614050 ptr=ffff88006db34800 bytes_req=576 bytes_alloc=1024 gfp_flags=GFP_KERNEL|GFP_REPEAT |
| </literallayout> |
| To again disable the kmalloc event, we need to send 0 to the |
| enable file: |
| <literallayout class='monospaced'> |
| root@sugarbay:/sys/kernel/debug/tracing/events/kmem/kmalloc# echo 0 > enable |
| </literallayout> |
| You can enable any number of events or complete subsystems |
| (by using the 'enable' file in the subsystem directory) and |
| get an arbitrarily fine-grained idea of what's going on in the |
| system by enabling as many of the appropriate tracepoints |
| as applicable. |
| </para> |
| |
| <para> |
| A number of the tools described in this HOWTO do just that, |
| including trace-cmd and kernelshark in the next section. |
| </para> |
| |
| <informalexample> |
| <emphasis>Tying it Together:</emphasis> These tracepoints and their representation |
| are used not only by ftrace, but by many of the other tools |
| covered in this document and they form a central point of |
| integration for the various tracers available in Linux. |
| They form a central part of the instrumentation for the |
| following tools: perf, lttng, ftrace, blktrace and SystemTap |
| </informalexample> |
| |
| <informalexample> |
| <emphasis>Tying it Together:</emphasis> Eventually all the special-purpose tracers |
| currently available in /sys/kernel/debug/tracing will be |
| removed and replaced with equivalent tracers based on the |
| 'trace events' subsystem. |
| </informalexample> |
| </section> |
| |
| <section id='trace-cmd-kernelshark'> |
| <title>trace-cmd/kernelshark</title> |
| |
| <para> |
| trace-cmd is essentially an extensive command-line 'wrapper' |
| interface that hides the details of all the individual files |
| in /sys/kernel/debug/tracing, allowing users to specify |
| specific particular events within the |
| /sys/kernel/debug/tracing/events/ subdirectory and to collect |
| traces and avoid having to deal with those details directly. |
| </para> |
| |
| <para> |
| As yet another layer on top of that, kernelshark provides a GUI |
| that allows users to start and stop traces and specify sets |
| of events using an intuitive interface, and view the |
| output as both trace events and as a per-CPU graphical |
| display. It directly uses 'trace-cmd' as the plumbing |
| that accomplishes all that underneath the covers (and |
| actually displays the trace-cmd command it uses, as we'll see). |
| </para> |
| |
| <para> |
| To start a trace using kernelshark, first start kernelshark: |
| <literallayout class='monospaced'> |
| root@sugarbay:~# kernelshark |
| </literallayout> |
| Then bring up the 'Capture' dialog by choosing from the |
| kernelshark menu: |
| <literallayout class='monospaced'> |
| Capture | Record |
| </literallayout> |
| That will display the following dialog, which allows you to |
| choose one or more events (or even one or more complete |
| subsystems) to trace: |
| </para> |
| |
| <para> |
| <imagedata fileref="figures/kernelshark-choose-events.png" width="6in" depth="6in" align="center" scalefit="1" /> |
| </para> |
| |
| <para> |
| Note that these are exactly the same sets of events described |
| in the previous trace events subsystem section, and in fact |
| is where trace-cmd gets them for kernelshark. |
| </para> |
| |
| <para> |
| In the above screenshot, we've decided to explore the |
| graphics subsystem a bit and so have chosen to trace all |
| the tracepoints contained within the 'i915' and 'drm' |
| subsystems. |
| </para> |
| |
| <para> |
| After doing that, we can start and stop the trace using |
| the 'Run' and 'Stop' button on the lower right corner of |
| the dialog (the same button will turn into the 'Stop' |
| button after the trace has started): |
| </para> |
| |
| <para> |
| <imagedata fileref="figures/kernelshark-output-display.png" width="6in" depth="6in" align="center" scalefit="1" /> |
| </para> |
| |
| <para> |
| Notice that the right-hand pane shows the exact trace-cmd |
| command-line that's used to run the trace, along with the |
| results of the trace-cmd run. |
| </para> |
| |
| <para> |
| Once the 'Stop' button is pressed, the graphical view magically |
| fills up with a colorful per-cpu display of the trace data, |
| along with the detailed event listing below that: |
| </para> |
| |
| <para> |
| <imagedata fileref="figures/kernelshark-i915-display.png" width="6in" depth="7in" align="center" scalefit="1" /> |
| </para> |
| |
| <para> |
| Here's another example, this time a display resulting |
| from tracing 'all events': |
| </para> |
| |
| <para> |
| <imagedata fileref="figures/kernelshark-all.png" width="6in" depth="7in" align="center" scalefit="1" /> |
| </para> |
| |
| <para> |
| The tool is pretty self-explanatory, but for more detailed |
| information on navigating through the data, see the |
| <ulink url='http://rostedt.homelinux.com/kernelshark/'>kernelshark website</ulink>. |
| </para> |
| </section> |
| |
| <section id='ftrace-documentation'> |
| <title>Documentation</title> |
| |
| <para> |
| The documentation for ftrace can be found in the kernel |
| Documentation directory: |
| <literallayout class='monospaced'> |
| Documentation/trace/ftrace.txt |
| </literallayout> |
| The documentation for the trace event subsystem can also |
| be found in the kernel Documentation directory: |
| <literallayout class='monospaced'> |
| Documentation/trace/events.txt |
| </literallayout> |
| There is a nice series of articles on using |
| ftrace and trace-cmd at LWN: |
| <itemizedlist> |
| <listitem><para><ulink url='http://lwn.net/Articles/365835/'>Debugging the kernel using Ftrace - part 1</ulink> |
| </para></listitem> |
| <listitem><para><ulink url='http://lwn.net/Articles/366796/'>Debugging the kernel using Ftrace - part 2</ulink> |
| </para></listitem> |
| <listitem><para><ulink url='http://lwn.net/Articles/370423/'>Secrets of the Ftrace function tracer</ulink> |
| </para></listitem> |
| <listitem><para><ulink url='https://lwn.net/Articles/410200/'>trace-cmd: A front-end for Ftrace</ulink> |
| </para></listitem> |
| </itemizedlist> |
| </para> |
| |
| <para> |
| There's more detailed documentation kernelshark usage here: |
| <ulink url='http://rostedt.homelinux.com/kernelshark/'>KernelShark</ulink> |
| </para> |
| |
| <para> |
| An amusing yet useful README (a tracing mini-HOWTO) can be |
| found in /sys/kernel/debug/tracing/README. |
| </para> |
| </section> |
| </section> |
| |
| <section id='profile-manual-systemtap'> |
| <title>systemtap</title> |
| |
| <para> |
| SystemTap is a system-wide script-based tracing and profiling tool. |
| </para> |
| |
| <para> |
| SystemTap scripts are C-like programs that are executed in the |
| kernel to gather/print/aggregate data extracted from the context |
| they end up being invoked under. |
| </para> |
| |
| <para> |
| For example, this probe from the |
| <ulink url='http://sourceware.org/systemtap/tutorial/'>SystemTap tutorial</ulink> |
| simply prints a line every time any process on the system open()s |
| a file. For each line, it prints the executable name of the |
| program that opened the file, along with its PID, and the name |
| of the file it opened (or tried to open), which it extracts |
| from the open syscall's argstr. |
| <literallayout class='monospaced'> |
| probe syscall.open |
| { |
| printf ("%s(%d) open (%s)\n", execname(), pid(), argstr) |
| } |
| |
| probe timer.ms(4000) # after 4 seconds |
| { |
| exit () |
| } |
| </literallayout> |
| Normally, to execute this probe, you'd simply install |
| systemtap on the system you want to probe, and directly run |
| the probe on that system e.g. assuming the name of the file |
| containing the above text is trace_open.stp: |
| <literallayout class='monospaced'> |
| # stap trace_open.stp |
| </literallayout> |
| What systemtap does under the covers to run this probe is 1) |
| parse and convert the probe to an equivalent 'C' form, 2) |
| compile the 'C' form into a kernel module, 3) insert the |
| module into the kernel, which arms it, and 4) collect the data |
| generated by the probe and display it to the user. |
| </para> |
| |
| <para> |
| In order to accomplish steps 1 and 2, the 'stap' program needs |
| access to the kernel build system that produced the kernel |
| that the probed system is running. In the case of a typical |
| embedded system (the 'target'), the kernel build system |
| unfortunately isn't typically part of the image running on |
| the target. It is normally available on the 'host' system |
| that produced the target image however; in such cases, |
| steps 1 and 2 are executed on the host system, and steps |
| 3 and 4 are executed on the target system, using only the |
| systemtap 'runtime'. |
| </para> |
| |
| <para> |
| The systemtap support in Yocto assumes that only steps |
| 3 and 4 are run on the target; it is possible to do |
| everything on the target, but this section assumes only |
| the typical embedded use-case. |
| </para> |
| |
| <para> |
| So basically what you need to do in order to run a systemtap |
| script on the target is to 1) on the host system, compile the |
| probe into a kernel module that makes sense to the target, 2) |
| copy the module onto the target system and 3) insert the |
| module into the target kernel, which arms it, and 4) collect |
| the data generated by the probe and display it to the user. |
| </para> |
| |
| <section id='systemtap-setup'> |
| <title>Setup</title> |
| |
| <para> |
| Those are a lot of steps and a lot of details, but |
| fortunately Yocto includes a script called 'crosstap' |
| that will take care of those details, allowing you to |
| simply execute a systemtap script on the remote target, |
| with arguments if necessary. |
| </para> |
| |
| <para> |
| In order to do this from a remote host, however, you |
| need to have access to the build for the image you |
| booted. The 'crosstap' script provides details on how |
| to do this if you run the script on the host without having |
| done a build: |
| <note> |
| SystemTap, which uses 'crosstap', assumes you can establish an |
| ssh connection to the remote target. |
| Please refer to the crosstap wiki page for details on verifying |
| ssh connections at |
| <ulink url='https://wiki.yoctoproject.org/wiki/Tracing_and_Profiling#systemtap'></ulink>. |
| Also, the ability to ssh into the target system is not enabled |
| by default in *-minimal images. |
| </note> |
| <literallayout class='monospaced'> |
| $ crosstap root@192.168.1.88 trace_open.stp |
| |
| Error: No target kernel build found. |
| Did you forget to create a local build of your image? |
| |
| 'crosstap' requires a local sdk build of the target system |
| (or a build that includes 'tools-profile') in order to build |
| kernel modules that can probe the target system. |
| |
| Practically speaking, that means you need to do the following: |
| - If you're running a pre-built image, download the release |
| and/or BSP tarballs used to build the image. |
| - If you're working from git sources, just clone the metadata |
| and BSP layers needed to build the image you'll be booting. |
| - Make sure you're properly set up to build a new image (see |
| the BSP README and/or the widely available basic documentation |
| that discusses how to build images). |
| - Build an -sdk version of the image e.g.: |
| $ bitbake core-image-sato-sdk |
| OR |
| - Build a non-sdk image but include the profiling tools: |
| [ edit local.conf and add 'tools-profile' to the end of |
| the EXTRA_IMAGE_FEATURES variable ] |
| $ bitbake core-image-sato |
| |
| Once you've build the image on the host system, you're ready to |
| boot it (or the equivalent pre-built image) and use 'crosstap' |
| to probe it (you need to source the environment as usual first): |
| |
| $ source oe-init-build-env |
| $ cd ~/my/systemtap/scripts |
| $ crosstap root@192.168.1.xxx myscript.stp |
| </literallayout> |
| So essentially what you need to do is build an SDK image or |
| image with 'tools-profile' as detailed in the |
| "<link linkend='profile-manual-general-setup'>General Setup</link>" |
| section of this manual, and boot the resulting target image. |
| </para> |
| |
| <note> |
| If you have a build directory containing multiple machines, |
| you need to have the MACHINE you're connecting to selected |
| in local.conf, and the kernel in that machine's build |
| directory must match the kernel on the booted system exactly, |
| or you'll get the above 'crosstap' message when you try to |
| invoke a script. |
| </note> |
| </section> |
| |
| <section id='running-a-script-on-a-target'> |
| <title>Running a Script on a Target</title> |
| |
| <para> |
| Once you've done that, you should be able to run a systemtap |
| script on the target: |
| <literallayout class='monospaced'> |
| $ cd /path/to/yocto |
| $ source oe-init-build-env |
| |
| ### Shell environment set up for builds. ### |
| |
| You can now run 'bitbake <replaceable>target</replaceable>' |
| |
| Common targets are: |
| core-image-minimal |
| core-image-sato |
| meta-toolchain |
| adt-installer |
| meta-ide-support |
| |
| You can also run generated qemu images with a command like 'runqemu qemux86' |
| </literallayout> |
| Once you've done that, you can cd to whatever directory |
| contains your scripts and use 'crosstap' to run the script: |
| <literallayout class='monospaced'> |
| $ cd /path/to/my/systemap/script |
| $ crosstap root@192.168.7.2 trace_open.stp |
| </literallayout> |
| If you get an error connecting to the target e.g.: |
| <literallayout class='monospaced'> |
| $ crosstap root@192.168.7.2 trace_open.stp |
| error establishing ssh connection on remote 'root@192.168.7.2' |
| </literallayout> |
| Try ssh'ing to the target and see what happens: |
| <literallayout class='monospaced'> |
| $ ssh root@192.168.7.2 |
| </literallayout> |
| A lot of the time, connection problems are due specifying a |
| wrong IP address or having a 'host key verification error'. |
| </para> |
| |
| <para> |
| If everything worked as planned, you should see something |
| like this (enter the password when prompted, or press enter |
| if it's set up to use no password): |
| <literallayout class='monospaced'> |
| $ crosstap root@192.168.7.2 trace_open.stp |
| root@192.168.7.2's password: |
| matchbox-termin(1036) open ("/tmp/vte3FS2LW", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0600) |
| matchbox-termin(1036) open ("/tmp/vteJMC7LW", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0600) |
| </literallayout> |
| </para> |
| </section> |
| |
| <section id='systemtap-documentation'> |
| <title>Documentation</title> |
| |
| <para> |
| The SystemTap language reference can be found here: |
| <ulink url='http://sourceware.org/systemtap/langref/'>SystemTap Language Reference</ulink> |
| </para> |
| |
| <para> |
| Links to other SystemTap documents, tutorials, and examples can be |
| found here: |
| <ulink url='http://sourceware.org/systemtap/documentation.html'>SystemTap documentation page</ulink> |
| </para> |
| </section> |
| </section> |
| |
| <section id='profile-manual-oprofile'> |
| <title>oprofile</title> |
| |
| <para> |
| oprofile itself is a command-line application that runs on the |
| target system. |
| </para> |
| |
| <section id='oprofile-setup'> |
| <title>Setup</title> |
| |
| <para> |
| For this section, we'll assume you've already performed the |
| basic setup outlined in the |
| "<link linkend='profile-manual-general-setup'>General Setup</link>" |
| section. |
| </para> |
| |
| <para> |
| For the section that deals with running oprofile from the command-line, |
| we assume you've ssh'ed to the host and will be running |
| oprofile on the target. |
| </para> |
| |
| <para> |
| oprofileui (oprofile-viewer) is a GUI-based program that runs |
| on the host and interacts remotely with the target. |
| See the oprofileui section for the exact steps needed to |
| install oprofileui on the host. |
| </para> |
| </section> |
| |
| <section id='oprofile-basic-usage'> |
| <title>Basic Usage</title> |
| |
| <para> |
| Oprofile as configured in Yocto is a system-wide profiler |
| (i.e. the version in Yocto doesn't yet make use of the |
| perf_events interface which would allow it to profile |
| specific processes and workloads). It relies on hardware |
| counter support in the hardware (but can fall back to a |
| timer-based mode), which means that it doesn't take |
| advantage of tracepoints or other event sources for example. |
| </para> |
| |
| <para> |
| It consists of a kernel module that collects samples and a |
| userspace daemon that writes the sample data to disk. |
| </para> |
| |
| <para> |
| The 'opcontrol' shell script is used for transparently |
| managing these components and starting and stopping |
| profiles, and the 'opreport' command is used to |
| display the results. |
| </para> |
| |
| <para> |
| The oprofile daemon should already be running, but before |
| you start profiling, you may need to change some settings |
| and some of these settings may require the daemon to not |
| be running. One of these settings is the path to the |
| vmlinux file, which you'll want to set using the --vmlinux |
| option if you want the kernel profiled: |
| <literallayout class='monospaced'> |
| root@crownbay:~# opcontrol --vmlinux=/boot/vmlinux-`uname -r` |
| The profiling daemon is currently active, so changes to the configuration |
| will be used the next time you restart oprofile after a --shutdown or --deinit. |
| </literallayout> |
| You can check if vmlinux file: is set using opcontrol --status: |
| <literallayout class='monospaced'> |
| root@crownbay:~# opcontrol --status |
| Daemon paused: pid 1334 |
| Separate options: library |
| vmlinux file: none |
| Image filter: none |
| Call-graph depth: 6 |
| </literallayout> |
| If it's not, you need to shutdown the daemon, add the setting |
| and restart the daemon: |
| <literallayout class='monospaced'> |
| root@crownbay:~# opcontrol --shutdown |
| Killing daemon. |
| |
| root@crownbay:~# opcontrol --vmlinux=/boot/vmlinux-`uname -r` |
| root@crownbay:~# opcontrol --start-daemon |
| Using default event: CPU_CLK_UNHALTED:100000:0:1:1 |
| Using 2.6+ OProfile kernel interface. |
| Reading module info. |
| Using log file /var/lib/oprofile/samples/oprofiled.log |
| Daemon started. |
| </literallayout> |
| If we check the status again we now see our updated settings: |
| <literallayout class='monospaced'> |
| root@crownbay:~# opcontrol --status |
| Daemon paused: pid 1649 |
| Separate options: library |
| vmlinux file: /boot/vmlinux-3.4.11-yocto-standard |
| Image filter: none |
| Call-graph depth: 6 |
| </literallayout> |
| We're now in a position to run a profile. For that we use |
| 'opcontrol --start': |
| <literallayout class='monospaced'> |
| root@crownbay:~# opcontrol --start |
| Profiler running. |
| </literallayout> |
| In another window, run our wget workload: |
| <literallayout class='monospaced'> |
| root@crownbay:~# rm linux-2.6.19.2.tar.bz2; wget <ulink url='http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2'>http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2</ulink>; sync |
| Connecting to downloads.yoctoproject.org (140.211.169.59:80) |
| linux-2.6.19.2.tar.b 100% |*******************************| 41727k 0:00:00 ETA |
| </literallayout> |
| To stop the profile we use 'opcontrol --shutdown', which not |
| only stops the profile but shuts down the daemon as well: |
| <literallayout class='monospaced'> |
| root@crownbay:~# opcontrol --shutdown |
| Stopping profiling. |
| Killing daemon. |
| </literallayout> |
| Oprofile writes sample data to /var/lib/oprofile/samples, |
| which you can look at if you're interested in seeing how the |
| samples are structured. This is also interesting because |
| it's related to how you dive down to get further details |
| about specific executables in OProfile. |
| </para> |
| |
| <para> |
| To see the default display output for a profile, simply type |
| 'opreport', which will show the results using the data in |
| /var/lib/oprofile/samples: |
| <literallayout class='monospaced'> |
| root@crownbay:~# opreport |
| |
| WARNING! The OProfile kernel driver reports sample buffer overflows. |
| Such overflows can result in incorrect sample attribution, invalid sample |
| files and other symptoms. See the oprofiled.log for details. |
| You should adjust your sampling frequency to eliminate (or at least minimize) |
| these overflows. |
| CPU: Intel Architectural Perfmon, speed 1.3e+06 MHz (estimated) |
| Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (No unit mask) count 100000 |
| CPU_CLK_UNHALT...| |
| samples| %| |
| ------------------ |
| 464365 79.8156 vmlinux-3.4.11-yocto-standard |
| 65108 11.1908 oprofiled |
| CPU_CLK_UNHALT...| |
| samples| %| |
| ------------------ |
| 64416 98.9372 oprofiled |
| 692 1.0628 libc-2.16.so |
| 36959 6.3526 no-vmlinux |
| 4378 0.7525 busybox |
| CPU_CLK_UNHALT...| |
| samples| %| |
| ------------------ |
| 2844 64.9612 libc-2.16.so |
| 1337 30.5391 busybox |
| 193 4.4084 ld-2.16.so |
| 2 0.0457 libnss_compat-2.16.so |
| 1 0.0228 libnsl-2.16.so |
| 1 0.0228 libnss_files-2.16.so |
| 4344 0.7467 bash |
| CPU_CLK_UNHALT...| |
| samples| %| |
| ------------------ |
| 2657 61.1648 bash |
| 1665 38.3287 libc-2.16.so |
| 18 0.4144 ld-2.16.so |
| 3 0.0691 libtinfo.so.5.9 |
| 1 0.0230 libdl-2.16.so |
| 3118 0.5359 nf_conntrack |
| 686 0.1179 matchbox-terminal |
| CPU_CLK_UNHALT...| |
| samples| %| |
| ------------------ |
| 214 31.1953 libglib-2.0.so.0.3200.4 |
| 114 16.6181 libc-2.16.so |
| 79 11.5160 libcairo.so.2.11200.2 |
| 78 11.3703 libgdk-x11-2.0.so.0.2400.8 |
| 51 7.4344 libpthread-2.16.so |
| 45 6.5598 libgobject-2.0.so.0.3200.4 |
| 29 4.2274 libvte.so.9.2800.2 |
| 25 3.6443 libX11.so.6.3.0 |
| 19 2.7697 libxcb.so.1.1.0 |
| 17 2.4781 libgtk-x11-2.0.so.0.2400.8 |
| 12 1.7493 librt-2.16.so |
| 3 0.4373 libXrender.so.1.3.0 |
| 671 0.1153 emgd |
| 411 0.0706 nf_conntrack_ipv4 |
| 391 0.0672 iptable_nat |
| 378 0.0650 nf_nat |
| 263 0.0452 Xorg |
| CPU_CLK_UNHALT...| |
| samples| %| |
| ------------------ |
| 106 40.3042 Xorg |
| 53 20.1521 libc-2.16.so |
| 31 11.7871 libpixman-1.so.0.27.2 |
| 26 9.8859 emgd_drv.so |
| 16 6.0837 libemgdsrv_um.so.1.5.15.3226 |
| 11 4.1825 libEMGD2d.so.1.5.15.3226 |
| 9 3.4221 libfb.so |
| 7 2.6616 libpthread-2.16.so |
| 1 0.3802 libudev.so.0.9.3 |
| 1 0.3802 libdrm.so.2.4.0 |
| 1 0.3802 libextmod.so |
| 1 0.3802 mouse_drv.so |
| . |
| . |
| . |
| 9 0.0015 connmand |
| CPU_CLK_UNHALT...| |
| samples| %| |
| ------------------ |
| 4 44.4444 libglib-2.0.so.0.3200.4 |
| 2 22.2222 libpthread-2.16.so |
| 1 11.1111 connmand |
| 1 11.1111 libc-2.16.so |
| 1 11.1111 librt-2.16.so |
| 6 0.0010 oprofile-server |
| CPU_CLK_UNHALT...| |
| samples| %| |
| ------------------ |
| 3 50.0000 libc-2.16.so |
| 1 16.6667 oprofile-server |
| 1 16.6667 libpthread-2.16.so |
| 1 16.6667 libglib-2.0.so.0.3200.4 |
| 5 8.6e-04 gconfd-2 |
| CPU_CLK_UNHALT...| |
| samples| %| |
| ------------------ |
| 2 40.0000 libdbus-1.so.3.7.2 |
| 2 40.0000 libglib-2.0.so.0.3200.4 |
| 1 20.0000 libc-2.16.so |
| </literallayout> |
| The output above shows the breakdown or samples by both |
| number of samples and percentage for each executable. |
| Within an executable, the sample counts are broken down |
| further into executable and shared libraries (DSOs) used |
| by the executable. |
| </para> |
| |
| <para> |
| To get even more detailed breakdowns by function, we need to |
| have the full paths to the DSOs, which we can get by |
| using -f with opreport: |
| <literallayout class='monospaced'> |
| root@crownbay:~# opreport -f |
| |
| CPU: Intel Architectural Perfmon, speed 1.3e+06 MHz (estimated) |
| Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (No unit mask) count 100000 |
| CPU_CLK_UNHALT...| |
| samples| %| |
| |
| 464365 79.8156 /boot/vmlinux-3.4.11-yocto-standard |
| 65108 11.1908 /usr/bin/oprofiled |
| CPU_CLK_UNHALT...| |
| samples| %| |
| ------------------ |
| 64416 98.9372 /usr/bin/oprofiled |
| 692 1.0628 /lib/libc-2.16.so |
| 36959 6.3526 /no-vmlinux |
| 4378 0.7525 /bin/busybox |
| CPU_CLK_UNHALT...| |
| samples| %| |
| ------------------ |
| 2844 64.9612 /lib/libc-2.16.so |
| 1337 30.5391 /bin/busybox |
| 193 4.4084 /lib/ld-2.16.so |
| 2 0.0457 /lib/libnss_compat-2.16.so |
| 1 0.0228 /lib/libnsl-2.16.so |
| 1 0.0228 /lib/libnss_files-2.16.so |
| 4344 0.7467 /bin/bash |
| CPU_CLK_UNHALT...| |
| samples| %| |
| ------------------ |
| 2657 61.1648 /bin/bash |
| 1665 38.3287 /lib/libc-2.16.so |
| 18 0.4144 /lib/ld-2.16.so |
| 3 0.0691 /lib/libtinfo.so.5.9 |
| 1 0.0230 /lib/libdl-2.16.so |
| . |
| . |
| . |
| </literallayout> |
| Using the paths shown in the above output and the -l option to |
| opreport, we can see all the functions that have hits in the |
| profile and their sample counts and percentages. Here's a |
| portion of what we get for the kernel: |
| <literallayout class='monospaced'> |
| root@crownbay:~# opreport -l /boot/vmlinux-3.4.11-yocto-standard |
| |
| CPU: Intel Architectural Perfmon, speed 1.3e+06 MHz (estimated) |
| Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (No unit mask) count 100000 |
| samples % symbol name |
| 233981 50.3873 intel_idle |
| 15437 3.3243 rb_get_reader_page |
| 14503 3.1232 ring_buffer_consume |
| 14092 3.0347 mutex_spin_on_owner |
| 13024 2.8047 read_hpet |
| 8039 1.7312 sub_preempt_count |
| 7096 1.5281 ioread32 |
| 6997 1.5068 add_preempt_count |
| 3985 0.8582 rb_advance_reader |
| 3488 0.7511 add_event_entry |
| 3303 0.7113 get_parent_ip |
| 3104 0.6684 rb_buffer_peek |
| 2960 0.6374 op_cpu_buffer_read_entry |
| 2614 0.5629 sync_buffer |
| 2545 0.5481 debug_smp_processor_id |
| 2456 0.5289 ohci_irq |
| 2397 0.5162 memset |
| 2349 0.5059 __copy_to_user_ll |
| 2185 0.4705 ring_buffer_event_length |
| 1918 0.4130 in_lock_functions |
| 1850 0.3984 __schedule |
| 1767 0.3805 __copy_from_user_ll_nozero |
| 1575 0.3392 rb_event_data_length |
| 1256 0.2705 memcpy |
| 1233 0.2655 system_call |
| 1213 0.2612 menu_select |
| </literallayout> |
| Notice that above we see an entry for the __copy_to_user_ll() |
| function that we've looked at with other profilers as well. |
| </para> |
| |
| <para> |
| Here's what we get when we do the same thing for the |
| busybox executable: |
| <literallayout class='monospaced'> |
| CPU: Intel Architectural Perfmon, speed 1.3e+06 MHz (estimated) |
| Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (No unit mask) count 100000 |
| samples % image name symbol name |
| 349 8.4198 busybox retrieve_file_data |
| 308 7.4306 libc-2.16.so _IO_file_xsgetn |
| 283 6.8275 libc-2.16.so __read_nocancel |
| 235 5.6695 libc-2.16.so syscall |
| 233 5.6212 libc-2.16.so clearerr |
| 215 5.1870 libc-2.16.so fread |
| 181 4.3667 libc-2.16.so __write_nocancel |
| 158 3.8118 libc-2.16.so __underflow |
| 151 3.6429 libc-2.16.so _dl_addr |
| 150 3.6188 busybox progress_meter |
| 150 3.6188 libc-2.16.so __poll_nocancel |
| 148 3.5706 libc-2.16.so _IO_file_underflow@@GLIBC_2.1 |
| 137 3.3052 busybox safe_poll |
| 125 3.0157 busybox bb_progress_update |
| 122 2.9433 libc-2.16.so __x86.get_pc_thunk.bx |
| 95 2.2919 busybox full_write |
| 81 1.9542 busybox safe_write |
| 77 1.8577 busybox xwrite |
| 72 1.7370 libc-2.16.so _IO_file_read |
| 71 1.7129 libc-2.16.so _IO_sgetn |
| 67 1.6164 libc-2.16.so poll |
| 52 1.2545 libc-2.16.so _IO_switch_to_get_mode |
| 45 1.0856 libc-2.16.so read |
| 34 0.8203 libc-2.16.so write |
| 32 0.7720 busybox monotonic_sec |
| 25 0.6031 libc-2.16.so vfprintf |
| 22 0.5308 busybox get_mono |
| 14 0.3378 ld-2.16.so strcmp |
| 14 0.3378 libc-2.16.so __x86.get_pc_thunk.cx |
| . |
| . |
| . |
| </literallayout> |
| Since we recorded the profile with a callchain depth of 6, we |
| should be able to see our __copy_to_user_ll() callchains in |
| the output, and indeed we can if we search around a bit in |
| the 'opreport --callgraph' output: |
| <literallayout class='monospaced'> |
| root@crownbay:~# opreport --callgraph /boot/vmlinux-3.4.11-yocto-standard |
| |
| 392 6.9639 vmlinux-3.4.11-yocto-standard sock_aio_read |
| 736 13.0751 vmlinux-3.4.11-yocto-standard __generic_file_aio_write |
| 3255 57.8255 vmlinux-3.4.11-yocto-standard inet_recvmsg |
| 785 0.1690 vmlinux-3.4.11-yocto-standard tcp_recvmsg |
| 1790 31.7940 vmlinux-3.4.11-yocto-standard local_bh_enable |
| 1238 21.9893 vmlinux-3.4.11-yocto-standard __kfree_skb |
| 992 17.6199 vmlinux-3.4.11-yocto-standard lock_sock_nested |
| 785 13.9432 vmlinux-3.4.11-yocto-standard tcp_recvmsg [self] |
| 525 9.3250 vmlinux-3.4.11-yocto-standard release_sock |
| 112 1.9893 vmlinux-3.4.11-yocto-standard tcp_cleanup_rbuf |
| 72 1.2789 vmlinux-3.4.11-yocto-standard skb_copy_datagram_iovec |
| |
| 170 0.0366 vmlinux-3.4.11-yocto-standard skb_copy_datagram_iovec |
| 1491 73.3038 vmlinux-3.4.11-yocto-standard memcpy_toiovec |
| 327 16.0767 vmlinux-3.4.11-yocto-standard skb_copy_datagram_iovec |
| 170 8.3579 vmlinux-3.4.11-yocto-standard skb_copy_datagram_iovec [self] |
| 20 0.9833 vmlinux-3.4.11-yocto-standard copy_to_user |
| |
| 2588 98.2909 vmlinux-3.4.11-yocto-standard copy_to_user |
| 2349 0.5059 vmlinux-3.4.11-yocto-standard __copy_to_user_ll |
| 2349 89.2138 vmlinux-3.4.11-yocto-standard __copy_to_user_ll [self] |
| 166 6.3046 vmlinux-3.4.11-yocto-standard do_page_fault |
| </literallayout> |
| Remember that by default OProfile sessions are cumulative |
| i.e. if you start and stop a profiling session, then start a |
| new one, the new one will not erase the previous run(s) but |
| will build on it. If you want to restart a profile from scratch, |
| you need to reset: |
| <literallayout class='monospaced'> |
| root@crownbay:~# opcontrol --reset |
| </literallayout> |
| </para> |
| </section> |
| |
| <section id='oprofileui-a-gui-for-oprofile'> |
| <title>OProfileUI - A GUI for OProfile</title> |
| |
| <para> |
| Yocto also supports a graphical UI for controlling and viewing |
| OProfile traces, called OProfileUI. To use it, you first need |
| to clone the oprofileui git repo, then configure, build, and |
| install it: |
| <literallayout class='monospaced'> |
| [trz@empanada tmp]$ git clone git://git.yoctoproject.org/oprofileui |
| [trz@empanada tmp]$ cd oprofileui |
| [trz@empanada oprofileui]$ ./autogen.sh |
| [trz@empanada oprofileui]$ sudo make install |
| </literallayout> |
| OprofileUI replaces the 'opreport' functionality with a GUI, |
| and normally doesn't require the user to use 'opcontrol' either. |
| If you want to profile the kernel, however, you need to either |
| use the UI to specify a vmlinux or use 'opcontrol' to specify |
| it on the target: |
| </para> |
| |
| <para> |
| First, on the target, check if vmlinux file: is set: |
| <literallayout class='monospaced'> |
| root@crownbay:~# opcontrol --status |
| </literallayout> |
| If not: |
| <literallayout class='monospaced'> |
| root@crownbay:~# opcontrol --shutdown |
| root@crownbay:~# opcontrol --vmlinux=/boot/vmlinux-`uname -r` |
| root@crownbay:~# opcontrol --start-daemon |
| </literallayout> |
| Now, start the oprofile UI on the host system: |
| <literallayout class='monospaced'> |
| [trz@empanada oprofileui]$ oprofile-viewer |
| </literallayout> |
| To run a profile on the remote system, first connect to the |
| remote system by pressing the 'Connect' button and supplying |
| the IP address and port of the remote system (the default |
| port is 4224). |
| </para> |
| |
| <para> |
| The oprofile server should automatically be started already. |
| If not, the connection will fail and you either typed in the |
| wrong IP address and port (see below), or you need to start |
| the server yourself: |
| <literallayout class='monospaced'> |
| root@crownbay:~# oprofile-server |
| </literallayout> |
| Or, to specify a specific port: |
| <literallayout class='monospaced'> |
| root@crownbay:~# oprofile-server --port 8888 |
| </literallayout> |
| Once connected, press the 'Start' button and then run the |
| wget workload on the remote system: |
| <literallayout class='monospaced'> |
| root@crownbay:~# rm linux-2.6.19.2.tar.bz2; wget <ulink url='http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2'>http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2</ulink>; sync |
| Connecting to downloads.yoctoproject.org (140.211.169.59:80) |
| linux-2.6.19.2.tar.b 100% |*******************************| 41727k 0:00:00 ETA |
| </literallayout> |
| Once the workload completes, press the 'Stop' button. At that |
| point the OProfile viewer will download the profile files it's |
| collected (this may take some time, especially if the kernel |
| was profiled). While it downloads the files, you should see |
| something like the following: |
| </para> |
| |
| <para> |
| <imagedata fileref="figures/oprofileui-downloading.png" width="6in" depth="7in" align="center" scalefit="1" /> |
| </para> |
| |
| <para> |
| Once the profile files have been retrieved, you should see a |
| list of the processes that were profiled: |
| </para> |
| |
| <para> |
| <imagedata fileref="figures/oprofileui-processes.png" width="6in" depth="7in" align="center" scalefit="1" /> |
| </para> |
| |
| <para> |
| If you select one of them, you should see all the symbols that |
| were hit during the profile. Selecting one of them will show a |
| list of callers and callees of the chosen function in two |
| panes below the top pane. For example, here's what we see |
| when we select __copy_to_user_ll(): |
| </para> |
| |
| <para> |
| <imagedata fileref="figures/oprofileui-copy-to-user.png" width="6in" depth="7in" align="center" scalefit="1" /> |
| </para> |
| |
| <para> |
| As another example, we can look at the busybox process and see |
| that the progress meter made a system call: |
| </para> |
| |
| <para> |
| <imagedata fileref="figures/oprofileui-busybox.png" width="6in" depth="7in" align="center" scalefit="1" /> |
| </para> |
| </section> |
| |
| <section id='oprofile-documentation'> |
| <title>Documentation</title> |
| |
| <para> |
| Yocto already has some information on setting up and using |
| OProfile and oprofileui. As this document doesn't cover |
| everything in detail, it may be worth taking a look at the |
| "<ulink url='&YOCTO_DOCS_DEV_URL;#platdev-oprofile'>Profiling with OProfile</ulink>" |
| section in the Yocto Project Development Manual |
| </para> |
| |
| <para> |
| The OProfile manual can be found here: |
| <ulink url='http://oprofile.sourceforge.net/doc/index.html'>OProfile manual</ulink> |
| </para> |
| |
| <para> |
| The OProfile website contains links to the above manual and |
| bunch of other items including an extensive set of examples: |
| <ulink url='http://oprofile.sourceforge.net/about/'>About OProfile</ulink> |
| </para> |
| </section> |
| </section> |
| |
| <section id='profile-manual-sysprof'> |
| <title>Sysprof</title> |
| |
| <para> |
| Sysprof is a very easy to use system-wide profiler that consists |
| of a single window with three panes and a few buttons which allow |
| you to start, stop, and view the profile from one place. |
| </para> |
| |
| <section id='sysprof-setup'> |
| <title>Setup</title> |
| |
| <para> |
| For this section, we'll assume you've already performed the |
| basic setup outlined in the General Setup section. |
| </para> |
| |
| <para> |
| Sysprof is a GUI-based application that runs on the target |
| system. For the rest of this document we assume you've |
| ssh'ed to the host and will be running Sysprof on the |
| target (you can use the '-X' option to ssh and have the |
| Sysprof GUI run on the target but display remotely on the |
| host if you want). |
| </para> |
| </section> |
| |
| <section id='sysprof-basic-usage'> |
| <title>Basic Usage</title> |
| |
| <para> |
| To start profiling the system, you simply press the 'Start' |
| button. To stop profiling and to start viewing the profile data |
| in one easy step, press the 'Profile' button. |
| </para> |
| |
| <para> |
| Once you've pressed the profile button, the three panes will |
| fill up with profiling data: |
| </para> |
| |
| <para> |
| <imagedata fileref="figures/sysprof-copy-to-user.png" width="6in" depth="4in" align="center" scalefit="1" /> |
| </para> |
| |
| <para> |
| The left pane shows a list of functions and processes. |
| Selecting one of those expands that function in the right |
| pane, showing all its callees. Note that this caller-oriented |
| display is essentially the inverse of perf's default |
| callee-oriented callchain display. |
| </para> |
| |
| <para> |
| In the screenshot above, we're focusing on __copy_to_user_ll() |
| and looking up the callchain we can see that one of the callers |
| of __copy_to_user_ll is sys_read() and the complete callpath |
| between them. Notice that this is essentially a portion of the |
| same information we saw in the perf display shown in the perf |
| section of this page. |
| </para> |
| |
| <para> |
| <imagedata fileref="figures/sysprof-copy-from-user.png" width="6in" depth="4in" align="center" scalefit="1" /> |
| </para> |
| |
| <para> |
| Similarly, the above is a snapshot of the Sysprof display of a |
| copy-from-user callchain. |
| </para> |
| |
| <para> |
| Finally, looking at the third Sysprof pane in the lower left, |
| we can see a list of all the callers of a particular function |
| selected in the top left pane. In this case, the lower pane is |
| showing all the callers of __mark_inode_dirty: |
| </para> |
| |
| <para> |
| <imagedata fileref="figures/sysprof-callers.png" width="6in" depth="4in" align="center" scalefit="1" /> |
| </para> |
| |
| <para> |
| Double-clicking on one of those functions will in turn change the |
| focus to the selected function, and so on. |
| </para> |
| |
| <informalexample> |
| <emphasis>Tying it Together:</emphasis> If you like sysprof's 'caller-oriented' |
| display, you may be able to approximate it in other tools as |
| well. For example, 'perf report' has the -g (--call-graph) |
| option that you can experiment with; one of the options is |
| 'caller' for an inverted caller-based callgraph display. |
| </informalexample> |
| </section> |
| |
| <section id='sysprof-documentation'> |
| <title>Documentation</title> |
| |
| <para> |
| There doesn't seem to be any documentation for Sysprof, but |
| maybe that's because it's pretty self-explanatory. |
| The Sysprof website, however, is here: |
| <ulink url='http://sysprof.com/'>Sysprof, System-wide Performance Profiler for Linux</ulink> |
| </para> |
| </section> |
| </section> |
| |
| <section id='lttng-linux-trace-toolkit-next-generation'> |
| <title>LTTng (Linux Trace Toolkit, next generation)</title> |
| |
| <section id='lttng-setup'> |
| <title>Setup</title> |
| |
| <para> |
| For this section, we'll assume you've already performed the |
| basic setup outlined in the General Setup section. |
| </para> |
| |
| <para> |
| LTTng is run on the target system by ssh'ing to it. |
| However, if you want to see the traces graphically, |
| install Eclipse as described in section |
| "<link linkend='manually-copying-a-trace-to-the-host-and-viewing-it-in-eclipse'>Manually copying a trace to the host and viewing it in Eclipse (i.e. using Eclipse without network support)</link>" |
| and follow the directions to manually copy traces to the host and |
| view them in Eclipse (i.e. using Eclipse without network support). |
| </para> |
| |
| <note> |
| Be sure to download and install/run the 'SR1' or later Juno release |
| of eclipse e.g.: |
| <ulink url='http://www.eclipse.org/downloads/download.php?file=/technology/epp/downloads/release/juno/SR1/eclipse-cpp-juno-SR1-linux-gtk-x86_64.tar.gz'>http://www.eclipse.org/downloads/download.php?file=/technology/epp/downloads/release/juno/SR1/eclipse-cpp-juno-SR1-linux-gtk-x86_64.tar.gz</ulink> |
| </note> |
| </section> |
| |
| <section id='collecting-and-viewing-traces'> |
| <title>Collecting and Viewing Traces</title> |
| |
| <para> |
| Once you've applied the above commits and built and booted your |
| image (you need to build the core-image-sato-sdk image or use one of the |
| other methods described in the General Setup section), you're |
| ready to start tracing. |
| </para> |
| |
| <section id='collecting-and-viewing-a-trace-on-the-target-inside-a-shell'> |
| <title>Collecting and viewing a trace on the target (inside a shell)</title> |
| |
| <para> |
| First, from the host, ssh to the target: |
| <literallayout class='monospaced'> |
| $ ssh -l root 192.168.1.47 |
| The authenticity of host '192.168.1.47 (192.168.1.47)' can't be established. |
| RSA key fingerprint is 23:bd:c8:b1:a8:71:52:00:ee:00:4f:64:9e:10:b9:7e. |
| Are you sure you want to continue connecting (yes/no)? yes |
| Warning: Permanently added '192.168.1.47' (RSA) to the list of known hosts. |
| root@192.168.1.47's password: |
| </literallayout> |
| Once on the target, use these steps to create a trace: |
| <literallayout class='monospaced'> |
| root@crownbay:~# lttng create |
| Spawning a session daemon |
| Session auto-20121015-232120 created. |
| Traces will be written in /home/root/lttng-traces/auto-20121015-232120 |
| </literallayout> |
| Enable the events you want to trace (in this case all |
| kernel events): |
| <literallayout class='monospaced'> |
| root@crownbay:~# lttng enable-event --kernel --all |
| All kernel events are enabled in channel channel0 |
| </literallayout> |
| Start the trace: |
| <literallayout class='monospaced'> |
| root@crownbay:~# lttng start |
| Tracing started for session auto-20121015-232120 |
| </literallayout> |
| And then stop the trace after awhile or after running |
| a particular workload that you want to trace: |
| <literallayout class='monospaced'> |
| root@crownbay:~# lttng stop |
| Tracing stopped for session auto-20121015-232120 |
| </literallayout> |
| You can now view the trace in text form on the target: |
| <literallayout class='monospaced'> |
| root@crownbay:~# lttng view |
| [23:21:56.989270399] (+?.?????????) sys_geteuid: { 1 }, { } |
| [23:21:56.989278081] (+0.000007682) exit_syscall: { 1 }, { ret = 0 } |
| [23:21:56.989286043] (+0.000007962) sys_pipe: { 1 }, { fildes = 0xB77B9E8C } |
| [23:21:56.989321802] (+0.000035759) exit_syscall: { 1 }, { ret = 0 } |
| [23:21:56.989329345] (+0.000007543) sys_mmap_pgoff: { 1 }, { addr = 0x0, len = 10485760, prot = 3, flags = 131362, fd = 4294967295, pgoff = 0 } |
| [23:21:56.989351694] (+0.000022349) exit_syscall: { 1 }, { ret = -1247805440 } |
| [23:21:56.989432989] (+0.000081295) sys_clone: { 1 }, { clone_flags = 0x411, newsp = 0xB5EFFFE4, parent_tid = 0xFFFFFFFF, child_tid = 0x0 } |
| [23:21:56.989477129] (+0.000044140) sched_stat_runtime: { 1 }, { comm = "lttng-consumerd", tid = 1193, runtime = 681660, vruntime = 43367983388 } |
| [23:21:56.989486697] (+0.000009568) sched_migrate_task: { 1 }, { comm = "lttng-consumerd", tid = 1193, prio = 20, orig_cpu = 1, dest_cpu = 1 } |
| [23:21:56.989508418] (+0.000021721) hrtimer_init: { 1 }, { hrtimer = 3970832076, clockid = 1, mode = 1 } |
| [23:21:56.989770462] (+0.000262044) hrtimer_cancel: { 1 }, { hrtimer = 3993865440 } |
| [23:21:56.989771580] (+0.000001118) hrtimer_cancel: { 0 }, { hrtimer = 3993812192 } |
| [23:21:56.989776957] (+0.000005377) hrtimer_expire_entry: { 1 }, { hrtimer = 3993865440, now = 79815980007057, function = 3238465232 } |
| [23:21:56.989778145] (+0.000001188) hrtimer_expire_entry: { 0 }, { hrtimer = 3993812192, now = 79815980008174, function = 3238465232 } |
| [23:21:56.989791695] (+0.000013550) softirq_raise: { 1 }, { vec = 1 } |
| [23:21:56.989795396] (+0.000003701) softirq_raise: { 0 }, { vec = 1 } |
| [23:21:56.989800635] (+0.000005239) softirq_raise: { 0 }, { vec = 9 } |
| [23:21:56.989807130] (+0.000006495) sched_stat_runtime: { 1 }, { comm = "lttng-consumerd", tid = 1193, runtime = 330710, vruntime = 43368314098 } |
| [23:21:56.989809993] (+0.000002863) sched_stat_runtime: { 0 }, { comm = "lttng-sessiond", tid = 1181, runtime = 1015313, vruntime = 36976733240 } |
| [23:21:56.989818514] (+0.000008521) hrtimer_expire_exit: { 0 }, { hrtimer = 3993812192 } |
| [23:21:56.989819631] (+0.000001117) hrtimer_expire_exit: { 1 }, { hrtimer = 3993865440 } |
| [23:21:56.989821866] (+0.000002235) hrtimer_start: { 0 }, { hrtimer = 3993812192, function = 3238465232, expires = 79815981000000, softexpires = 79815981000000 } |
| [23:21:56.989822984] (+0.000001118) hrtimer_start: { 1 }, { hrtimer = 3993865440, function = 3238465232, expires = 79815981000000, softexpires = 79815981000000 } |
| [23:21:56.989832762] (+0.000009778) softirq_entry: { 1 }, { vec = 1 } |
| [23:21:56.989833879] (+0.000001117) softirq_entry: { 0 }, { vec = 1 } |
| [23:21:56.989838069] (+0.000004190) timer_cancel: { 1 }, { timer = 3993871956 } |
| [23:21:56.989839187] (+0.000001118) timer_cancel: { 0 }, { timer = 3993818708 } |
| [23:21:56.989841492] (+0.000002305) timer_expire_entry: { 1 }, { timer = 3993871956, now = 79515980, function = 3238277552 } |
| [23:21:56.989842819] (+0.000001327) timer_expire_entry: { 0 }, { timer = 3993818708, now = 79515980, function = 3238277552 } |
| [23:21:56.989854831] (+0.000012012) sched_stat_runtime: { 1 }, { comm = "lttng-consumerd", tid = 1193, runtime = 49237, vruntime = 43368363335 } |
| [23:21:56.989855949] (+0.000001118) sched_stat_runtime: { 0 }, { comm = "lttng-sessiond", tid = 1181, runtime = 45121, vruntime = 36976778361 } |
| [23:21:56.989861257] (+0.000005308) sched_stat_sleep: { 1 }, { comm = "kworker/1:1", tid = 21, delay = 9451318 } |
| [23:21:56.989862374] (+0.000001117) sched_stat_sleep: { 0 }, { comm = "kworker/0:0", tid = 4, delay = 9958820 } |
| [23:21:56.989868241] (+0.000005867) sched_wakeup: { 0 }, { comm = "kworker/0:0", tid = 4, prio = 120, success = 1, target_cpu = 0 } |
| [23:21:56.989869358] (+0.000001117) sched_wakeup: { 1 }, { comm = "kworker/1:1", tid = 21, prio = 120, success = 1, target_cpu = 1 } |
| [23:21:56.989877460] (+0.000008102) timer_expire_exit: { 1 }, { timer = 3993871956 } |
| [23:21:56.989878577] (+0.000001117) timer_expire_exit: { 0 }, { timer = 3993818708 } |
| . |
| . |
| . |
| </literallayout> |
| You can now safely destroy the trace session (note that |
| this doesn't delete the trace - it's still there |
| in ~/lttng-traces): |
| <literallayout class='monospaced'> |
| root@crownbay:~# lttng destroy |
| Session auto-20121015-232120 destroyed at /home/root |
| </literallayout> |
| Note that the trace is saved in a directory of the same |
| name as returned by 'lttng create', under the ~/lttng-traces |
| directory (note that you can change this by supplying your |
| own name to 'lttng create'): |
| <literallayout class='monospaced'> |
| root@crownbay:~# ls -al ~/lttng-traces |
| drwxrwx--- 3 root root 1024 Oct 15 23:21 . |
| drwxr-xr-x 5 root root 1024 Oct 15 23:57 .. |
| drwxrwx--- 3 root root 1024 Oct 15 23:21 auto-20121015-232120 |
| </literallayout> |
| </para> |
| </section> |
| |
| <section id='collecting-and-viewing-a-userspace-trace-on-the-target-inside-a-shell'> |
| <title>Collecting and viewing a userspace trace on the target (inside a shell)</title> |
| |
| <para> |
| For LTTng userspace tracing, you need to have a properly |
| instrumented userspace program. For this example, we'll use |
| the 'hello' test program generated by the lttng-ust build. |
| </para> |
| |
| <para> |
| The 'hello' test program isn't installed on the rootfs by |
| the lttng-ust build, so we need to copy it over manually. |
| First cd into the build directory that contains the hello |
| executable: |
| <literallayout class='monospaced'> |
| $ cd build/tmp/work/core2_32-poky-linux/lttng-ust/2.0.5-r0/git/tests/hello/.libs |
| </literallayout> |
| Copy that over to the target machine: |
| <literallayout class='monospaced'> |
| $ scp hello root@192.168.1.20: |
| </literallayout> |
| You now have the instrumented lttng 'hello world' test |
| program on the target, ready to test. |
| </para> |
| |
| <para> |
| First, from the host, ssh to the target: |
| <literallayout class='monospaced'> |
| $ ssh -l root 192.168.1.47 |
| The authenticity of host '192.168.1.47 (192.168.1.47)' can't be established. |
| RSA key fingerprint is 23:bd:c8:b1:a8:71:52:00:ee:00:4f:64:9e:10:b9:7e. |
| Are you sure you want to continue connecting (yes/no)? yes |
| Warning: Permanently added '192.168.1.47' (RSA) to the list of known hosts. |
| root@192.168.1.47's password: |
| </literallayout> |
| Once on the target, use these steps to create a trace: |
| <literallayout class='monospaced'> |
| root@crownbay:~# lttng create |
| Session auto-20190303-021943 created. |
| Traces will be written in /home/root/lttng-traces/auto-20190303-021943 |
| </literallayout> |
| Enable the events you want to trace (in this case all |
| userspace events): |
| <literallayout class='monospaced'> |
| root@crownbay:~# lttng enable-event --userspace --all |
| All UST events are enabled in channel channel0 |
| </literallayout> |
| Start the trace: |
| <literallayout class='monospaced'> |
| root@crownbay:~# lttng start |
| Tracing started for session auto-20190303-021943 |
| </literallayout> |
| Run the instrumented hello world program: |
| <literallayout class='monospaced'> |
| root@crownbay:~# ./hello |
| Hello, World! |
| Tracing... done. |
| </literallayout> |
| And then stop the trace after awhile or after running a |
| particular workload that you want to trace: |
| <literallayout class='monospaced'> |
| root@crownbay:~# lttng stop |
| Tracing stopped for session auto-20190303-021943 |
| </literallayout> |
| You can now view the trace in text form on the target: |
| <literallayout class='monospaced'> |
| root@crownbay:~# lttng view |
| [02:31:14.906146544] (+?.?????????) hello:1424 ust_tests_hello:tptest: { cpu_id = 1 }, { intfield = 0, intfield2 = 0x0, longfield = 0, netintfield = 0, netintfieldhex = 0x0, arrfield1 = [ [0] = 1, [1] = 2, [2] = 3 ], arrfield2 = "test", _seqfield1_length = 4, seqfield1 = [ [0] = 116, [1] = 101, [2] = 115, [3] = 116 ], _seqfield2_length = 4, seqfield2 = "test", stringfield = "test", floatfield = 2222, doublefield = 2, boolfield = 1 } |
| [02:31:14.906170360] (+0.000023816) hello:1424 ust_tests_hello:tptest: { cpu_id = 1 }, { intfield = 1, intfield2 = 0x1, longfield = 1, netintfield = 1, netintfieldhex = 0x1, arrfield1 = [ [0] = 1, [1] = 2, [2] = 3 ], arrfield2 = "test", _seqfield1_length = 4, seqfield1 = [ [0] = 116, [1] = 101, [2] = 115, [3] = 116 ], _seqfield2_length = 4, seqfield2 = "test", stringfield = "test", floatfield = 2222, doublefield = 2, boolfield = 1 } |
| [02:31:14.906183140] (+0.000012780) hello:1424 ust_tests_hello:tptest: { cpu_id = 1 }, { intfield = 2, intfield2 = 0x2, longfield = 2, netintfield = 2, netintfieldhex = 0x2, arrfield1 = [ [0] = 1, [1] = 2, [2] = 3 ], arrfield2 = "test", _seqfield1_length = 4, seqfield1 = [ [0] = 116, [1] = 101, [2] = 115, [3] = 116 ], _seqfield2_length = 4, seqfield2 = "test", stringfield = "test", floatfield = 2222, doublefield = 2, boolfield = 1 } |
| [02:31:14.906194385] (+0.000011245) hello:1424 ust_tests_hello:tptest: { cpu_id = 1 }, { intfield = 3, intfield2 = 0x3, longfield = 3, netintfield = 3, netintfieldhex = 0x3, arrfield1 = [ [0] = 1, [1] = 2, [2] = 3 ], arrfield2 = "test", _seqfield1_length = 4, seqfield1 = [ [0] = 116, [1] = 101, [2] = 115, [3] = 116 ], _seqfield2_length = 4, seqfield2 = "test", stringfield = "test", floatfield = 2222, doublefield = 2, boolfield = 1 } |
| . |
| . |
| . |
| </literallayout> |
| You can now safely destroy the trace session (note that |
| this doesn't delete the trace - it's still |
| there in ~/lttng-traces): |
| <literallayout class='monospaced'> |
| root@crownbay:~# lttng destroy |
| Session auto-20190303-021943 destroyed at /home/root |
| </literallayout> |
| </para> |
| </section> |
| |
| <section id='manually-copying-a-trace-to-the-host-and-viewing-it-in-eclipse'> |
| <title>Manually copying a trace to the host and viewing it in Eclipse (i.e. using Eclipse without network support)</title> |
| |
| <para> |
| If you already have an LTTng trace on a remote target and |
| would like to view it in Eclipse on the host, you can easily |
| copy it from the target to the host and import it into |
| Eclipse to view it using the LTTng Eclipse plug-in already |
| bundled in the Eclipse (Juno SR1 or greater). |
| </para> |
| |
| <para> |
| Using the trace we created in the previous section, archive |
| it and copy it to your host system: |
| <literallayout class='monospaced'> |
| root@crownbay:~/lttng-traces# tar zcvf auto-20121015-232120.tar.gz auto-20121015-232120 |
| auto-20121015-232120/ |
| auto-20121015-232120/kernel/ |
| auto-20121015-232120/kernel/metadata |
| auto-20121015-232120/kernel/channel0_1 |
| auto-20121015-232120/kernel/channel0_0 |
| |
| $ scp root@192.168.1.47:lttng-traces/auto-20121015-232120.tar.gz . |
| root@192.168.1.47's password: |
| auto-20121015-232120.tar.gz 100% 1566KB 1.5MB/s 00:01 |
| </literallayout> |
| Unarchive it on the host: |
| <literallayout class='monospaced'> |
| $ gunzip -c auto-20121015-232120.tar.gz | tar xvf - |
| auto-20121015-232120/ |
| auto-20121015-232120/kernel/ |
| auto-20121015-232120/kernel/metadata |
| auto-20121015-232120/kernel/channel0_1 |
| auto-20121015-232120/kernel/channel0_0 |
| </literallayout> |
| We can now import the trace into Eclipse and view it: |
| <orderedlist> |
| <listitem><para>First, start eclipse and open the |
| 'LTTng Kernel' perspective by selecting the following |
| menu item: |
| <literallayout class='monospaced'> |
| Window | Open Perspective | Other... |
| </literallayout></para></listitem> |
| <listitem><para>In the dialog box that opens, select |
| 'LTTng Kernel' from the list.</para></listitem> |
| <listitem><para>Back at the main menu, select the |
| following menu item: |
| <literallayout class='monospaced'> |
| File | New | Project... |
| </literallayout></para></listitem> |
| <listitem><para>In the dialog box that opens, select |
| the 'Tracing | Tracing Project' wizard and press |
| 'Next>'.</para></listitem> |
| <listitem><para>Give the project a name and press |
| 'Finish'.</para></listitem> |
| <listitem><para>In the 'Project Explorer' pane under |
| the project you created, right click on the |
| 'Traces' item.</para></listitem> |
| <listitem><para>Select 'Import..." and in the dialog |
| that's displayed:</para></listitem> |
| <listitem><para>Browse the filesystem and find the |
| select the 'kernel' directory containing the trace |
| you copied from the target |
| e.g. auto-20121015-232120/kernel</para></listitem> |
| <listitem><para>'Checkmark' the directory in the tree |
| that's displayed for the trace</para></listitem> |
| <listitem><para>Below that, select 'Common Trace Format: |
| Kernel Trace' for the 'Trace Type'</para></listitem> |
| <listitem><para>Press 'Finish' to close the dialog |
| </para></listitem> |
| <listitem><para>Back in the 'Project Explorer' pane, |
| double-click on the 'kernel' item for the |
| trace you just imported under 'Traces' |
| </para></listitem> |
| </orderedlist> |
| You should now see your trace data displayed graphically |
| in several different views in Eclipse: |
| </para> |
| |
| <para> |
| <imagedata fileref="figures/lttngmain0.png" width="6in" depth="6in" align="center" scalefit="1" /> |
| </para> |
| |
| <para> |
| You can access extensive help information on how to use |
| the LTTng plug-in to search and analyze captured traces via |
| the Eclipse help system: |
| <literallayout class='monospaced'> |
| Help | Help Contents | LTTng Plug-in User Guide |
| </literallayout> |
| </para> |
| </section> |
| |
| <section id='collecting-and-viewing-a-trace-in-eclipse'> |
| <title>Collecting and viewing a trace in Eclipse</title> |
| |
| <note> |
| This section on collecting traces remotely doesn't currently |
| work because of Eclipse 'RSE' connectivity problems. Manually |
| tracing on the target, copying the trace files to the host, |
| and viewing the trace in Eclipse on the host as outlined in |
| previous steps does work however - please use the manual |
| steps outlined above to view traces in Eclipse. |
| </note> |
| |
| <para> |
| In order to trace a remote target, you also need to add |
| a 'tracing' group on the target and connect as a user |
| who's part of that group e.g: |
| <literallayout class='monospaced'> |
| # adduser tomz |
| # groupadd -r tracing |
| # usermod -a -G tracing tomz |
| </literallayout> |
| <orderedlist> |
| <listitem><para>First, start eclipse and open the |
| 'LTTng Kernel' perspective by selecting the following |
| menu item: |
| <literallayout class='monospaced'> |
| Window | Open Perspective | Other... |
| </literallayout></para></listitem> |
| <listitem><para>In the dialog box that opens, select |
| 'LTTng Kernel' from the list.</para></listitem> |
| <listitem><para>Back at the main menu, select the |
| following menu item: |
| <literallayout class='monospaced'> |
| File | New | Project... |
| </literallayout></para></listitem> |
| <listitem><para>In the dialog box that opens, select |
| the 'Tracing | Tracing Project' wizard and |
| press 'Next>'.</para></listitem> |
| <listitem><para>Give the project a name and press |
| 'Finish'. That should result in an entry in the |
| 'Project' subwindow.</para></listitem> |
| <listitem><para>In the 'Control' subwindow just below |
| it, press 'New Connection'.</para></listitem> |
| <listitem><para>Add a new connection, giving it the |
| hostname or IP address of the target system. |
| </para></listitem> |
| <listitem><para>Provide the username and password |
| of a qualified user (a member of the 'tracing' group) |
| or root account on the target system. |
| </para></listitem> |
| <listitem><para>Provide appropriate answers to whatever |
| else is asked for e.g. 'secure storage password' |
| can be anything you want. |
| If you get an 'RSE Error' it may be due to proxies. |
| It may be possible to get around the problem by |
| changing the following setting: |
| <literallayout class='monospaced'> |
| Window | Preferences | Network Connections |
| </literallayout> |
| Switch 'Active Provider' to 'Direct' |
| </para></listitem> |
| </orderedlist> |
| </para> |
| </section> |
| </section> |
| |
| <section id='lltng-documentation'> |
| <title>Documentation</title> |
| |
| <para> |
| You can find the primary LTTng Documentation on the |
| <ulink url='https://lttng.org/docs/'>LTTng Documentation</ulink> |
| site. |
| The documentation on this site is appropriate for intermediate to |
| advanced software developers who are working in a Linux environment |
| and are interested in efficient software tracing. |
| </para> |
| |
| <para> |
| For information on LTTng in general, visit the |
| <ulink url='http://lttng.org/lttng2.0'>LTTng Project</ulink> |
| site. |
| You can find a "Getting Started" link on this site that takes |
| you to an LTTng Quick Start. |
| </para> |
| |
| <para> |
| Finally, you can access extensive help information on how to use |
| the LTTng plug-in to search and analyze captured traces via the |
| Eclipse help system: |
| <literallayout class='monospaced'> |
| Help | Help Contents | LTTng Plug-in User Guide |
| </literallayout> |
| </para> |
| </section> |
| </section> |
| |
| <section id='profile-manual-blktrace'> |
| <title>blktrace</title> |
| |
| <para> |
| blktrace is a tool for tracing and reporting low-level disk I/O. |
| blktrace provides the tracing half of the equation; its output can |
| be piped into the blkparse program, which renders the data in a |
| human-readable form and does some basic analysis: |
| </para> |
| |
| <section id='blktrace-setup'> |
| <title>Setup</title> |
| |
| <para> |
| For this section, we'll assume you've already performed the |
| basic setup outlined in the |
| "<link linkend='profile-manual-general-setup'>General Setup</link>" |
| section. |
| </para> |
| |
| <para> |
| blktrace is an application that runs on the target system. |
| You can run the entire blktrace and blkparse pipeline on the |
| target, or you can run blktrace in 'listen' mode on the target |
| and have blktrace and blkparse collect and analyze the data on |
| the host (see the |
| "<link linkend='using-blktrace-remotely'>Using blktrace Remotely</link>" |
| section below). |
| For the rest of this section we assume you've ssh'ed to the |
| host and will be running blkrace on the target. |
| </para> |
| </section> |
| |
| <section id='blktrace-basic-usage'> |
| <title>Basic Usage</title> |
| |
| <para> |
| To record a trace, simply run the 'blktrace' command, giving it |
| the name of the block device you want to trace activity on: |
| <literallayout class='monospaced'> |
| root@crownbay:~# blktrace /dev/sdc |
| </literallayout> |
| In another shell, execute a workload you want to trace. |
| <literallayout class='monospaced'> |
| root@crownbay:/media/sdc# rm linux-2.6.19.2.tar.bz2; wget <ulink url='http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2'>http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2</ulink>; sync |
| Connecting to downloads.yoctoproject.org (140.211.169.59:80) |
| linux-2.6.19.2.tar.b 100% |*******************************| 41727k 0:00:00 ETA |
| </literallayout> |
| Press Ctrl-C in the blktrace shell to stop the trace. It will |
| display how many events were logged, along with the per-cpu file |
| sizes (blktrace records traces in per-cpu kernel buffers and |
| simply dumps them to userspace for blkparse to merge and sort |
| later). |
| <literallayout class='monospaced'> |
| ^C=== sdc === |
| CPU 0: 7082 events, 332 KiB data |
| CPU 1: 1578 events, 74 KiB data |
| Total: 8660 events (dropped 0), 406 KiB data |
| </literallayout> |
| If you examine the files saved to disk, you see multiple files, |
| one per CPU and with the device name as the first part of the |
| filename: |
| <literallayout class='monospaced'> |
| root@crownbay:~# ls -al |
| drwxr-xr-x 6 root root 1024 Oct 27 22:39 . |
| drwxr-sr-x 4 root root 1024 Oct 26 18:24 .. |
| -rw-r--r-- 1 root root 339938 Oct 27 22:40 sdc.blktrace.0 |
| -rw-r--r-- 1 root root 75753 Oct 27 22:40 sdc.blktrace.1 |
| </literallayout> |
| To view the trace events, simply invoke 'blkparse' in the |
| directory containing the trace files, giving it the device name |
| that forms the first part of the filenames: |
| <literallayout class='monospaced'> |
| root@crownbay:~# blkparse sdc |
| |
| 8,32 1 1 0.000000000 1225 Q WS 3417048 + 8 [jbd2/sdc-8] |
| 8,32 1 2 0.000025213 1225 G WS 3417048 + 8 [jbd2/sdc-8] |
| 8,32 1 3 0.000033384 1225 P N [jbd2/sdc-8] |
| 8,32 1 4 0.000043301 1225 I WS 3417048 + 8 [jbd2/sdc-8] |
| 8,32 1 0 0.000057270 0 m N cfq1225 insert_request |
| 8,32 1 0 0.000064813 0 m N cfq1225 add_to_rr |
| 8,32 1 5 0.000076336 1225 U N [jbd2/sdc-8] 1 |
| 8,32 1 0 0.000088559 0 m N cfq workload slice:150 |
| 8,32 1 0 0.000097359 0 m N cfq1225 set_active wl_prio:0 wl_type:1 |
| 8,32 1 0 0.000104063 0 m N cfq1225 Not idling. st->count:1 |
| 8,32 1 0 0.000112584 0 m N cfq1225 fifo= (null) |
| 8,32 1 0 0.000118730 0 m N cfq1225 dispatch_insert |
| 8,32 1 0 0.000127390 0 m N cfq1225 dispatched a request |
| 8,32 1 0 0.000133536 0 m N cfq1225 activate rq, drv=1 |
| 8,32 1 6 0.000136889 1225 D WS 3417048 + 8 [jbd2/sdc-8] |
| 8,32 1 7 0.000360381 1225 Q WS 3417056 + 8 [jbd2/sdc-8] |
| 8,32 1 8 0.000377422 1225 G WS 3417056 + 8 [jbd2/sdc-8] |
| 8,32 1 9 0.000388876 1225 P N [jbd2/sdc-8] |
| 8,32 1 10 0.000397886 1225 Q WS 3417064 + 8 [jbd2/sdc-8] |
| 8,32 1 11 0.000404800 1225 M WS 3417064 + 8 [jbd2/sdc-8] |
| 8,32 1 12 0.000412343 1225 Q WS 3417072 + 8 [jbd2/sdc-8] |
| 8,32 1 13 0.000416533 1225 M WS 3417072 + 8 [jbd2/sdc-8] |
| 8,32 1 14 0.000422121 1225 Q WS 3417080 + 8 [jbd2/sdc-8] |
| 8,32 1 15 0.000425194 1225 M WS 3417080 + 8 [jbd2/sdc-8] |
| 8,32 1 16 0.000431968 1225 Q WS 3417088 + 8 [jbd2/sdc-8] |
| 8,32 1 17 0.000435251 1225 M WS 3417088 + 8 [jbd2/sdc-8] |
| 8,32 1 18 0.000440279 1225 Q WS 3417096 + 8 [jbd2/sdc-8] |
| 8,32 1 19 0.000443911 1225 M WS 3417096 + 8 [jbd2/sdc-8] |
| 8,32 1 20 0.000450336 1225 Q WS 3417104 + 8 [jbd2/sdc-8] |
| 8,32 1 21 0.000454038 1225 M WS 3417104 + 8 [jbd2/sdc-8] |
| 8,32 1 22 0.000462070 1225 Q WS 3417112 + 8 [jbd2/sdc-8] |
| 8,32 1 23 0.000465422 1225 M WS 3417112 + 8 [jbd2/sdc-8] |
| 8,32 1 24 0.000474222 1225 I WS 3417056 + 64 [jbd2/sdc-8] |
| 8,32 1 0 0.000483022 0 m N cfq1225 insert_request |
| 8,32 1 25 0.000489727 1225 U N [jbd2/sdc-8] 1 |
| 8,32 1 0 0.000498457 0 m N cfq1225 Not idling. st->count:1 |
| 8,32 1 0 0.000503765 0 m N cfq1225 dispatch_insert |
| 8,32 1 0 0.000512914 0 m N cfq1225 dispatched a request |
| 8,32 1 0 0.000518851 0 m N cfq1225 activate rq, drv=2 |
| . |
| . |
| . |
| 8,32 0 0 58.515006138 0 m N cfq3551 complete rqnoidle 1 |
| 8,32 0 2024 58.516603269 3 C WS 3156992 + 16 [0] |
| 8,32 0 0 58.516626736 0 m N cfq3551 complete rqnoidle 1 |
| 8,32 0 0 58.516634558 0 m N cfq3551 arm_idle: 8 group_idle: 0 |
| 8,32 0 0 58.516636933 0 m N cfq schedule dispatch |
| 8,32 1 0 58.516971613 0 m N cfq3551 slice expired t=0 |
| 8,32 1 0 58.516982089 0 m N cfq3551 sl_used=13 disp=6 charge=13 iops=0 sect=80 |
| 8,32 1 0 58.516985511 0 m N cfq3551 del_from_rr |
| 8,32 1 0 58.516990819 0 m N cfq3551 put_queue |
| |
| CPU0 (sdc): |
| Reads Queued: 0, 0KiB Writes Queued: 331, 26,284KiB |
| Read Dispatches: 0, 0KiB Write Dispatches: 485, 40,484KiB |
| Reads Requeued: 0 Writes Requeued: 0 |
| Reads Completed: 0, 0KiB Writes Completed: 511, 41,000KiB |
| Read Merges: 0, 0KiB Write Merges: 13, 160KiB |
| Read depth: 0 Write depth: 2 |
| IO unplugs: 23 Timer unplugs: 0 |
| CPU1 (sdc): |
| Reads Queued: 0, 0KiB Writes Queued: 249, 15,800KiB |
| Read Dispatches: 0, 0KiB Write Dispatches: 42, 1,600KiB |
| Reads Requeued: 0 Writes Requeued: 0 |
| Reads Completed: 0, 0KiB Writes Completed: 16, 1,084KiB |
| Read Merges: 0, 0KiB Write Merges: 40, 276KiB |
| Read depth: 0 Write depth: 2 |
| IO unplugs: 30 Timer unplugs: 1 |
| |
| Total (sdc): |
| Reads Queued: 0, 0KiB Writes Queued: 580, 42,084KiB |
| Read Dispatches: 0, 0KiB Write Dispatches: 527, 42,084KiB |
| Reads Requeued: 0 Writes Requeued: 0 |
| Reads Completed: 0, 0KiB Writes Completed: 527, 42,084KiB |
| Read Merges: 0, 0KiB Write Merges: 53, 436KiB |
| IO unplugs: 53 Timer unplugs: 1 |
| |
| Throughput (R/W): 0KiB/s / 719KiB/s |
| Events (sdc): 6,592 entries |
| Skips: 0 forward (0 - 0.0%) |
| Input file sdc.blktrace.0 added |
| Input file sdc.blktrace.1 added |
| </literallayout> |
| The report shows each event that was found in the blktrace data, |
| along with a summary of the overall block I/O traffic during |
| the run. You can look at the |
| <ulink url='http://linux.die.net/man/1/blkparse'>blkparse</ulink> |
| manpage to learn the |
| meaning of each field displayed in the trace listing. |
| </para> |
| |
| <section id='blktrace-live-mode'> |
| <title>Live Mode</title> |
| |
| <para> |
| blktrace and blkparse are designed from the ground up to |
| be able to operate together in a 'pipe mode' where the |
| stdout of blktrace can be fed directly into the stdin of |
| blkparse: |
| <literallayout class='monospaced'> |
| root@crownbay:~# blktrace /dev/sdc -o - | blkparse -i - |
| </literallayout> |
| This enables long-lived tracing sessions to run without |
| writing anything to disk, and allows the user to look for |
| certain conditions in the trace data in 'real-time' by |
| viewing the trace output as it scrolls by on the screen or |
| by passing it along to yet another program in the pipeline |
| such as grep which can be used to identify and capture |
| conditions of interest. |
| </para> |
| |
| <para> |
| There's actually another blktrace command that implements |
| the above pipeline as a single command, so the user doesn't |
| have to bother typing in the above command sequence: |
| <literallayout class='monospaced'> |
| root@crownbay:~# btrace /dev/sdc |
| </literallayout> |
| </para> |
| </section> |
| |
| <section id='using-blktrace-remotely'> |
| <title>Using blktrace Remotely</title> |
| |
| <para> |
| Because blktrace traces block I/O and at the same time |
| normally writes its trace data to a block device, and |
| in general because it's not really a great idea to make |
| the device being traced the same as the device the tracer |
| writes to, blktrace provides a way to trace without |
| perturbing the traced device at all by providing native |
| support for sending all trace data over the network. |
| </para> |
| |
| <para> |
| To have blktrace operate in this mode, start blktrace on |
| the target system being traced with the -l option, along with |
| the device to trace: |
| <literallayout class='monospaced'> |
| root@crownbay:~# blktrace -l /dev/sdc |
| server: waiting for connections... |
| </literallayout> |
| On the host system, use the -h option to connect to the |
| target system, also passing it the device to trace: |
| <literallayout class='monospaced'> |
| $ blktrace -d /dev/sdc -h 192.168.1.43 |
| blktrace: connecting to 192.168.1.43 |
| blktrace: connected! |
| </literallayout> |
| On the target system, you should see this: |
| <literallayout class='monospaced'> |
| server: connection from 192.168.1.43 |
| </literallayout> |
| In another shell, execute a workload you want to trace. |
| <literallayout class='monospaced'> |
| root@crownbay:/media/sdc# rm linux-2.6.19.2.tar.bz2; wget <ulink url='http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2'>http://downloads.yoctoproject.org/mirror/sources/linux-2.6.19.2.tar.bz2</ulink>; sync |
| Connecting to downloads.yoctoproject.org (140.211.169.59:80) |
| linux-2.6.19.2.tar.b 100% |*******************************| 41727k 0:00:00 ETA |
| </literallayout> |
| When it's done, do a Ctrl-C on the host system to |
| stop the trace: |
| <literallayout class='monospaced'> |
| ^C=== sdc === |
| CPU 0: 7691 events, 361 KiB data |
| CPU 1: 4109 events, 193 KiB data |
| Total: 11800 events (dropped 0), 554 KiB data |
| </literallayout> |
| On the target system, you should also see a trace |
| summary for the trace just ended: |
| <literallayout class='monospaced'> |
| server: end of run for 192.168.1.43:sdc |
| === sdc === |
| CPU 0: 7691 events, 361 KiB data |
| CPU 1: 4109 events, 193 KiB data |
| Total: 11800 events (dropped 0), 554 KiB data |
| </literallayout> |
| The blktrace instance on the host will save the target |
| output inside a hostname-timestamp directory: |
| <literallayout class='monospaced'> |
| $ ls -al |
| drwxr-xr-x 10 root root 1024 Oct 28 02:40 . |
| drwxr-sr-x 4 root root 1024 Oct 26 18:24 .. |
| drwxr-xr-x 2 root root 1024 Oct 28 02:40 192.168.1.43-2012-10-28-02:40:56 |
| </literallayout> |
| cd into that directory to see the output files: |
| <literallayout class='monospaced'> |
| $ ls -l |
| -rw-r--r-- 1 root root 369193 Oct 28 02:44 sdc.blktrace.0 |
| -rw-r--r-- 1 root root 197278 Oct 28 02:44 sdc.blktrace.1 |
| </literallayout> |
| And run blkparse on the host system using the device name: |
| <literallayout class='monospaced'> |
| $ blkparse sdc |
| |
| 8,32 1 1 0.000000000 1263 Q RM 6016 + 8 [ls] |
| 8,32 1 0 0.000036038 0 m N cfq1263 alloced |
| 8,32 1 2 0.000039390 1263 G RM 6016 + 8 [ls] |
| 8,32 1 3 0.000049168 1263 I RM 6016 + 8 [ls] |
| 8,32 1 0 0.000056152 0 m N cfq1263 insert_request |
| 8,32 1 0 0.000061600 0 m N cfq1263 add_to_rr |
| 8,32 1 0 0.000075498 0 m N cfq workload slice:300 |
| . |
| . |
| . |
| 8,32 0 0 177.266385696 0 m N cfq1267 arm_idle: 8 group_idle: 0 |
| 8,32 0 0 177.266388140 0 m N cfq schedule dispatch |
| 8,32 1 0 177.266679239 0 m N cfq1267 slice expired t=0 |
| 8,32 1 0 177.266689297 0 m N cfq1267 sl_used=9 disp=6 charge=9 iops=0 sect=56 |
| 8,32 1 0 177.266692649 0 m N cfq1267 del_from_rr |
| 8,32 1 0 177.266696560 0 m N cfq1267 put_queue |
| |
| CPU0 (sdc): |
| Reads Queued: 0, 0KiB Writes Queued: 270, 21,708KiB |
| Read Dispatches: 59, 2,628KiB Write Dispatches: 495, 39,964KiB |
| Reads Requeued: 0 Writes Requeued: 0 |
| Reads Completed: 90, 2,752KiB Writes Completed: 543, 41,596KiB |
| Read Merges: 0, 0KiB Write Merges: 9, 344KiB |
| Read depth: 2 Write depth: 2 |
| IO unplugs: 20 Timer unplugs: 1 |
| CPU1 (sdc): |
| Reads Queued: 688, 2,752KiB Writes Queued: 381, 20,652KiB |
| Read Dispatches: 31, 124KiB Write Dispatches: 59, 2,396KiB |
| Reads Requeued: 0 Writes Requeued: 0 |
| Reads Completed: 0, 0KiB Writes Completed: 11, 764KiB |
| Read Merges: 598, 2,392KiB Write Merges: 88, 448KiB |
| Read depth: 2 Write depth: 2 |
| IO unplugs: 52 Timer unplugs: 0 |
| |
| Total (sdc): |
| Reads Queued: 688, 2,752KiB Writes Queued: 651, 42,360KiB |
| Read Dispatches: 90, 2,752KiB Write Dispatches: 554, 42,360KiB |
| Reads Requeued: 0 Writes Requeued: 0 |
| Reads Completed: 90, 2,752KiB Writes Completed: 554, 42,360KiB |
| Read Merges: 598, 2,392KiB Write Merges: 97, 792KiB |
| IO unplugs: 72 Timer unplugs: 1 |
| |
| Throughput (R/W): 15KiB/s / 238KiB/s |
| Events (sdc): 9,301 entries |
| Skips: 0 forward (0 - 0.0%) |
| </literallayout> |
| You should see the trace events and summary just as |
| you would have if you'd run the same command on the target. |
| </para> |
| </section> |
| |
| <section id='tracing-block-io-via-ftrace'> |
| <title>Tracing Block I/O via 'ftrace'</title> |
| |
| <para> |
| It's also possible to trace block I/O using only |
| <link linkend='the-trace-events-subsystem'>trace events subsystem</link>, |
| which can be useful for casual tracing |
| if you don't want to bother dealing with the userspace tools. |
| </para> |
| |
| <para> |
| To enable tracing for a given device, use |
| /sys/block/xxx/trace/enable, where xxx is the device name. |
| This for example enables tracing for /dev/sdc: |
| <literallayout class='monospaced'> |
| root@crownbay:/sys/kernel/debug/tracing# echo 1 > /sys/block/sdc/trace/enable |
| </literallayout> |
| Once you've selected the device(s) you want to trace, |
| selecting the 'blk' tracer will turn the blk tracer on: |
| <literallayout class='monospaced'> |
| root@crownbay:/sys/kernel/debug/tracing# cat available_tracers |
| blk function_graph function nop |
| |
| root@crownbay:/sys/kernel/debug/tracing# echo blk > current_tracer |
| </literallayout> |
| Execute the workload you're interested in: |
| <literallayout class='monospaced'> |
| root@crownbay:/sys/kernel/debug/tracing# cat /media/sdc/testfile.txt |
| </literallayout> |
| And look at the output (note here that we're using |
| 'trace_pipe' instead of trace to capture this trace - |
| this allows us to wait around on the pipe for data to |
| appear): |
| <literallayout class='monospaced'> |
| root@crownbay:/sys/kernel/debug/tracing# cat trace_pipe |
| cat-3587 [001] d..1 3023.276361: 8,32 Q R 1699848 + 8 [cat] |
| cat-3587 [001] d..1 3023.276410: 8,32 m N cfq3587 alloced |
| cat-3587 [001] d..1 3023.276415: 8,32 G R 1699848 + 8 [cat] |
| cat-3587 [001] d..1 3023.276424: 8,32 P N [cat] |
| cat-3587 [001] d..2 3023.276432: 8,32 I R 1699848 + 8 [cat] |
| cat-3587 [001] d..1 3023.276439: 8,32 m N cfq3587 insert_request |
| cat-3587 [001] d..1 3023.276445: 8,32 m N cfq3587 add_to_rr |
| cat-3587 [001] d..2 3023.276454: 8,32 U N [cat] 1 |
| cat-3587 [001] d..1 3023.276464: 8,32 m N cfq workload slice:150 |
| cat-3587 [001] d..1 3023.276471: 8,32 m N cfq3587 set_active wl_prio:0 wl_type:2 |
| cat-3587 [001] d..1 3023.276478: 8,32 m N cfq3587 fifo= (null) |
| cat-3587 [001] d..1 3023.276483: 8,32 m N cfq3587 dispatch_insert |
| cat-3587 [001] d..1 3023.276490: 8,32 m N cfq3587 dispatched a request |
| cat-3587 [001] d..1 3023.276497: 8,32 m N cfq3587 activate rq, drv=1 |
| cat-3587 [001] d..2 3023.276500: 8,32 D R 1699848 + 8 [cat] |
| </literallayout> |
| And this turns off tracing for the specified device: |
| <literallayout class='monospaced'> |
| root@crownbay:/sys/kernel/debug/tracing# echo 0 > /sys/block/sdc/trace/enable |
| </literallayout> |
| </para> |
| </section> |
| </section> |
| |
| <section id='blktrace-documentation'> |
| <title>Documentation</title> |
| |
| <para> |
| Online versions of the man pages for the commands discussed |
| in this section can be found here: |
| <itemizedlist> |
| <listitem><para><ulink url='http://linux.die.net/man/8/blktrace'>http://linux.die.net/man/8/blktrace</ulink> |
| </para></listitem> |
| <listitem><para><ulink url='http://linux.die.net/man/1/blkparse'>http://linux.die.net/man/1/blkparse</ulink> |
| </para></listitem> |
| <listitem><para><ulink url='http://linux.die.net/man/8/btrace'>http://linux.die.net/man/8/btrace</ulink> |
| </para></listitem> |
| </itemizedlist> |
| </para> |
| |
| <para> |
| The above manpages, along with manpages for the other |
| blktrace utilities (btt, blkiomon, etc) can be found in the |
| /doc directory of the blktrace tools git repo: |
| <literallayout class='monospaced'> |
| $ git clone git://git.kernel.dk/blktrace.git |
| </literallayout> |
| </para> |
| </section> |
| </section> |
| </chapter> |
| <!-- |
| vim: expandtab tw=80 ts=4 |
| --> |