Debug performance issues with strace

Have you ever been asked why is this program running so slowly?  Or why does it take 15 seconds to startup?

Strace might be the tool for you.  It’s a system call tracer which let’s you see what this program is requesting from the system. For example, if you run ls to a directory some key thins will be:
getdents(3, /* 47 entries */, 32768)    = 1928  (Get entries from a directory)
write(1, ..   (write out to stdout the output of ls)

You can add timing information to get more of an idea of what’s happening, I usually add:
-tt  (print absolute timestamps in usecs, very useful for comparing to nmap output for instance)
-T  (print time in each syscall)
-ff (follow forks, if it spawns other processes this will you separate them out to different files)

So run it all `strace -tt -T -ff ls’ and now it looks like:
14:09:10.747357 getdents(3, /* 48 entries */, 32768) = 1952 <0.000056>
14:09:10.747724 write(1, .....) = 57 <0.000031

Absolute time  syscall 

When using this on anything non-trivial, I suggest exporting it to a file (-o file).  You can also try analyzing it live as the long delays should cause strace output to stop.

I’ve got the strace output, it’s huge!  Now what?

Let’s start at the first one, which will be the lowest number, output_name.10001 for example. Check the time stamp at the top and the bottom, does it cover the time frame where you are investigating?  If so, dig in and find where the jump in time was.

In this example it was:
09:13:32.329980 write(1, "[2]... ", 7)  = 7 <0.000033>
09:13:32.330041 wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 4}], 0, NULL) = 14963 <4512.975521>
10:28:45.305644 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=14963, si_status=4, si_utime=0, si_stime=0} ---
10:28:45.305907 write(1, "[1]... ", 7)  = 7 <0.000036>

Or in other words, it’s really in that child process 14963 so let’s open up output_name.14963
Scroll throw to the first jump in time and we see:
09:13:20.418886 connect(5, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("")}, 16) = -1 ETIMEDOUT (Connection timed out) <127.317319>
... about 30 lines...
09:15:28.738327 connect(5, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("")}, 16) = -1 ETIMEDOUT (Connection timed out) <127.253925>
... many more lines here ...
10:26:37.977564 connect(5, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("")}, 16) = -1 ETIMEDOUT (Connection timed out) <127.326639>

So it’s timing out trying to connect to an IP address and then trying again over 10 times.  Using strace will help you the most when the wait is caused by something on the system level (disk, networking, etc).

Do tech support? Come check out sosreport

Sosreport is used by both Red Hat and Canonical support as the primary tool used to collect information from customers’ machines when they are having an issue.   It can both collect files and run commands.  It’s open source and you can use it too.

I’m doing a Show & Tell session on it in this coming UOS: http://summit.ubuntu.com/uos-1505/meeting/22457/sosreport-for-troubleshooting.  

I’ve got about 10-15 minutes of stuff to show.  Questions are welcome.

Getting cheaper all the time…

One of my goals of the $500 open-to-the-core* project was making Ubuntu laptops cheaper.   Most have either been really high end or bottom of the barrel.

I’m pleased to share two new laptops at new price points!

The new System76 Lemur $599

That’s $200 less then their previous lowest priced system.

 

The new Dell XPS 13  $949

I believe that’s about $300 less than the base model from last year.

Packaging Notes

I’ve done easy fixes (debdiffs) in Ubuntu and find I need to look up exactly how I want to do a debdiff every time.   Last time I had to look at 5 different docs to get all the commands I needed.   The bug I based this on was a debian only change (Init script), I plan to update it next time I have an actual source change.

  1. Start a new VM/ Cloud instance
  2. sudo apt-get install packaging-dev
  3.  apt-get source <package_name>  ;  apt-get build-dep <package_name>
  4. cd into-directory-created
  5. Make the change (if it’s only a debian/ change)
  6. dch -i   (document it)
  7. debuild -S -us -uc  (build it)
  8. debdiff rrdtool_1.4.7-1.dsc rrdtool_1.4.7-1ubuntu1.dsc > rrdtool_1.4.7-1ubuntu1.debdiff   (make the debdiff – note to me, change the name later)
  9. cd into-directory; DEB_BUILD_OPTIONS=’nostrip noopt debug’ fakeroot debian/rules binary  (build it)
  10.  Test it

Docs used:

  1. http://packaging.ubuntu.com/html/traditional-packaging.html
  2. http://packaging.ubuntu.com/html/fixing-a-bug-example.html
  3. http://cheesehead-techblog.blogspot.com/2008/10/creating-patch-to-fix-ubuntu-bug.html
  4. https://wiki.debian.org/IntroDebianPackaging
  5. https://wiki.debian.org/BuildingTutorial

Would you crowdfund a $500 Ubuntu “open to the core” laptop?

UPDATE 2 (11/28) – We’re 77% of the way to 1000.  I guesstimate we would have raised at least $300,000 if this we’re a live campaign.

UPDATE – I’ve removed the silly US restriction.  I know there are more options in Europe, China, India, etc, but why shouldn’t you get access to the “open to the core” laptop!
This would definitely come with at least 3 USB ports (and at least one USB 3.0 port).

Since Jolla had success with crowdfunding a tablet, it’s a good time to see if we can get some mid-range Ubuntu laptops for sale to consumers in as many places as possible.  I’d like to get some ideas about whether there is enough demand for a very open $500 Ubuntu laptop.

Would you crowdfund this? (Core Goals)

  • 15″ 1080p Matte Screen
  • 720p Webcam with microphone
  • Spill-resistant and nice to type on keyboard
  • Intel i3+ or AMD A6+
  • Built-in Intel or AMD graphics with no proprietary firmware
  • 4 GB Ram
  • 128 GB SSD (this would be the one component that might have to be proprietary as I’m not aware of another option)
  • Ethernet 10/100/1000
  • Wireless up to N
  • HDMI
  • SD card reader
  • CoreBoot (No proprietary BIOS)
  • Ubuntu 14.04 preloaded of course
  • Agreement with manufacturer to continue selling this laptop (or similar one) with Ubuntu preloaded to consumers for at least 3 years.

Stretch Goals? Or should they be core goals?

Will only be added if they don’t push the cost up significantly (or if everyone really wants them) and can be done with 100% open source software/firmware.

  • Touchscreen
  • Convertible to Tablet
  • GPS
  • FM Tuner (and built-in antenna)
  • Digital TV Tuner (and built-in antenna)
  • Ruggedized
  • Direct sunlight readable screen
  • “Frontlight” tech.  (think Amazon PaperWhite)
  • Bluetooth
  • Backlit keyboard
  • USB Power Adapter

Take my quick survey if you want to see this happen.  If at least 1000 people say “Yes,” I’ll approach manufacturers.   The first version might just end up being a Chromebook modified with better specs, but I think that would be fine.

Link to survey – http://goo.gl/forms/bwmBf92O1d