what's this

Hi This is me

JULIA EVANS

E S

in son
and in this zine I want to tell you about

i
3
These are 5 ways I've changed how I think
about debugging

T
Remember the bug is happening
q for a logical reason
It's never magic Really Evenwhen it makes no sense

F I'm confident I can fix it

before Ein iF ot now

gie waIIIeI
DE Talk to my coworkers
 know my debugging toolkit
before now
I want to know

Ei
t
Em
to find out

HE most importanthy I learned to like it I

95507
IgIIII
7
a
before now
EE
facial expression
determination

whatyouilllearriumt

I can't teach you to

in 20 pagesdebugging
though I'll try anyway9 I care show you some of
0
my dmmmmfThMebuggigookit though

These are the tools I reach for when I have

a question about a program I want to know
the answer to By the end of this zine I hope
to have given you a few new tools to use B
 section 1 I10 and
e system calls
In this zine there are 3 sections
of tools that I love

For each tool I'll tell you why it's useful

and give an example Each one is either

lowwmgoros.im
Some of the most basic questions you
might have when you log into a
misbehaving machine are

is this machine writing to or

reading
from disk The network
are the
programs reading files
Which files

So we're starting with finding out

whichresources arebeingused and
whato urprogram saredoing
Let's go
 IFTdstat I 3
i

I loved stat because it's

super simple Every second it
prints out how many bytes were
written to the network1disk that second

Once I had an intermittently slow database

server I opened up d stat and stared at the

output while monitoring database speed

$ dstat

send recv

E
Eidsxax
I
fire'ingttihnisiserinoodima

Legighwfetete 39874337 73
i 0
wit
s
i o back to normal
i
could 300 MB coming in over the
network mean a 300 MB database query

I YESIE
This was an AWESOME CLUE that
helped me isolate the problem query
 I stracewmjr.inO
strace is
my favorite program It prints
every system call your program used It's a cool
way to get an overall picture of what your
program is doing and I using it to answer
questions like whichfiles are being opened

I
I$ strace python my_program.py 1

reaadopen("/home/bork/.config_file) = 3
file
file read(3, "the contents of the file") descriptor
Is hundreds of lines
ie connect(5, "172.217.0.163")
Ef sendto(5, "hi!!!") send hi
g to 172.2170.163

strace can make your programs

run 50 x slower Don't run it on
9
WARNING
your production database

I can'tdojustice to strace here but I have a

whole other zine about it at
wizard zines com
 open Snoop
eBPF EIh3
kindof
When you run

$ opensnoop -p PID

it will print out inrealtimemh every file

being opened by a program You might think

andyouwould be right But

qyjoffroa.fi
$ strace -e open -p PID strace can make your
program run 5Ox slower
OpenSnoopwon't slow you down

I how to get it how it works

Requires Ubuntu 16.041 opensnoop is a script
or a 4.141 kernel version that uses a new kernel
i i l i

Installation instructions atfeature called I

PE
github.com iovisor bcceBPF is fast
There'salso an opensnoop
on OSX BSD
there are lots of eBpe is powered
thhfg
yhdattf E
5e.pro
 ii i

section 2
networking
I've devoted a lot of space to
in this zine
networking tools and I want to explain why

A lot of the programs I work with

communicate over HTTP
request response

if
my
EE's fprogramI
Eire if
Every programming language uses the
same network protocols So the network
is a nice language independent place
to answer questions like

was the request wrong or was

it the response
ismy service even running
my program is slow
Whose fault is that

Let's go
 WEE
net cat
HTTP requests are and you
just text
can use netcat to make
any HTTP request
you want by hand
Here's how to make a request for
example.com Try it
printf "GET / HTTP/1.1\r\nHost:
example.com\r\n\r\n"
| nc example.com 80

You should get a response back with

a bunch of HTML 1
You can also use net cat to send huge
files over a local network quickly

ftp.t
i
target Ichi
$ hostname -I d
ii
t.ee tonthese
cat bigfile |
I
l
I 192.168.2.132 t nc 192.168.2.132 9931 I
I
$ nc -l 9931 > bigfile i
this listens on the port this sends the data

you can also use ifconfig to get your IP address

 At netstataa owni.io

Every network request gets sent to a

port like 80 on a computer To

receive a request a
program aka server
needs to be listening on the port Finding
out which programs are listening on which

ports is really easy It's just

tuna please I
also known as
$ sudo netstat -tunapl

Here's what you'll see

proto local address PID / program name
tcp 0.0.0.0:8080 2993 / python
I port

So I nets tat because

it tells me which processes
are
using which ports

On Osx use II II III instead

WEE ng rep greeting
ngrep is
my favorite
starter network
spy tool Try it right now Run
sudo ngrep -d any metafilter

Then go to http metafitter.com

in your browser You should see
matching network packets in rig rep's
output1 We are SPIES 8

Once at work I made a change

to a client so that it sent
some id
with all its requests I wanted to make
sure it was working so I ran

sudo ngrep some_id l

I
s

I found out everything was

okay
hot tcp dump
tcpdump the most difficult
is
see
networking tool we'll discuss wifardzainzesncomhg
here and it took me a all about
while to it I use it to tapdump
save network traffic to analyze later
so

T
sudo tcpdump port 8997 port 8997 is

-w service.pcap actually a tiny

i Et
apcap file packetcapture is the
EEIIEI.in
language These
setting's ia
Some situations where I'll use tcpdTmp
I'm sending a request to a machine and
I want to know where it's even
getting there
tcpdump port 80
prints every packet on port 80
I have some slow network connections
and I want to know whether to blame
the client or server I also need wires hark
B I just want to print out packets to
see them Hopdump A
 wire shark os.in
Wireshark is an amazingly GUI tool
for network analysis Here's an exercise to
learn it Run this
$ sudo tcpdump port 80 -w http.pcap

while that's running open metafitter com in

your browser Then press Ctrl C to stoptcpdump
Now we have a pcap file to analyze
$ wireshark http.pcap

Explore the Wire shark interface

Questions you can try to answer

What HTTP headers did your browser

send to metafitter com

hint search frame contains "GET"

How longdid the longest request take
hint click statistics Conversations

Howmany packets were sent to replace with

metafitter com's servers metafittercom's IP
6
hint search ip.dst == 54.186.13.33
 section 3 CPU percent

Your programs spend a lot of time

onthe CPU Billions of cycles
What are they DOING

This section is about using perth

to answer that question perf is a
Linux only tool that is extremely
useful and not as well known
as it should be

Some things I didn't have space for in this

section but wanted to mention anyway

A valgrind
A the Java ecosystem's fantastic tools

jstack VisualVM Your kit which your

language is probably jealous of
ftrace for linux kernel tracing
LTTng ditto
e BPF
 perf
perf is not simple or
elegant It's a weird
multi tool that does a few different very
useful things First it's a sampling
profiler
Try running
$ sudo perf record python
pressCtrl1C after a few seconds Sp at

You can look at the results with

$ sudo perf report
Mine says it spent 5 of its time in the
Py Dict Get Item function Cool 1 We learned
a
tiny thing about the CPython interpreter

howsyouCfunctions1workseverywheres7Jifyouuseperftoprofile.pe
rf
be installed on
can
a Python program it'll prettymuch any Linux
show you the C functions machine The exact
symbols from the CPython features it has will
interpreter not the depend on your
Python functions Kernel version
 perf is for everyone
One day I had a server that was using
100 of its CPU Within about 60 seconds
I knew it was doing regular expression
matching in Ruby How perf top is like
top but for functions instead of programmes
$ sudo perf top
process PID % function
ruby 1957 77 match_at
i F
perftop doesn'talwayshelp Ruby's internal regexp
But it's easyto try andsometimes matchingfunction
I learn somethingto
murmuring

especially for Java and node devs

Remember when I said perf only knows
C functions It's not quite true node js
and the JVM Java Scala Clojure have both
taughtperf about their functions

TE
I Inode HJavaIE
Use the Look up perf mapagent
perf basic prof On GitHub and follow
command line option the directions
 funny
flamegraphse
A A A
Flamegraphs are an awesome way to
visualize CPU performance Generate them
with Brendan Gregg's flamegraph.pl tool
I github.com/brendangregg/flamegraph

Here's what they look like

n
r a
rain
m
eat
m m
A
biteoo.lt
8opanda2igator8Oomaio
They're constructed from collections usually
thousands of stack traces sampled from
a program The one above means 80 of
main
the stack traces started with alligator
main

f
and 10 with panda
eat J
You construct them from perf recordings
can

see Brendan Gregg's flame graph GitHub for now

but lots of other unrelated tools can produce
I them
 0
spy on your CPU
Your CPU has a small
cache on it the L1 cache ts
t
it o

If you're trying to do an operation

in microseconds CPU cache use matters

KyFkhmoywfroogIrakmnoYsifsing
thosecaches
perfstattf.q
tip
pass e
eE.is to
D
F
Eiositehl $ perf stat ls
This runs ls and prints a report at the end
Your CPU can track all kinds of counters
howitI
works aboutwhatit's doing perf stat asks
it to countthings like L1 cache misses
report the results
Hardware is cod I've never used perf stat
f in earnest but I think it's awesome that you
can get so much info from your CPUgo
 pheivwmsomhe9 fi gwnefe.arned
Thanks for reading
Thanks to my partner Kamal for
reviewing and to the amazing
Monica D inculescu C not Waldorf
for the cover art

To learn more see

my blog jvns.ca
wizard zines
my other
A zines com
doe brendangregg.com

But really you just need to experiment

Try these tools everywhere See where they
help you track down bugs and where they don't

E I
js
qjmottheatmdeidnmtu.NO
that problem oh well9

It takes practice but I find these tools

both fun and useful for my job1 I hope
you will tooo