Wed, 29 Feb 2012

Graphite - Scalable Realtime Graphing

Recently at work we've been trying to get into the measuring and monitoring business. In looking around the measuring side I came across graphite, which describes itself as "scalable realtime graphing".

I won't go into many details, you can read them on the graphite site. Basically, it's a fairly simple and robust-looking system for gathering tuples. What kind of tuples, you say?

some.label value timestamp

Which doesn't sound like much, until you think in terms of:

system.edu_columbia_cul_verdi.cpu.load_average.1min 0.79 1330568950

Which might represent the 1 minute load average on a particular system at a point in time. Graphite itself doesn't have any tools that measure anything — it's just a data collection and visualization setup — but there are plenty of scripts and systems that allow you to collect all kinds of common data. After a couple hours getting graphite itself up and running, and some slight modification to an existing script, I was feeding various system metrics from a few machines into graphite. A bit of noodling with the frontend later, I had some nifty looking graphs.

The ease of this all, and, I suspect, why there's a good number of scripts and setups that feed data into graphite, stems from how simple it is to get data into graphite. Just open up a tcp socket and start sending tuples, separated by linebreaks. If you want to be a bit efficient, throw a Python pickle of an array of tuples at another socket. If you want to be easy, throw a UDP packet at a third socket. That's it. There's a handful of libraries that applications can use to make this even more simple — something which excites me in the "If it moves, graph it; if it doesn't move, graph it in case it starts" philosophy.

Even better, graphite doesn't care about the label at all, which means as soon as you decide to measure something you just need to pick out a label, what the label means, and start slinging it at graphite. Graphite does use rule based on regular expressions to determine how granular and how long to store data, e.g. we might configure graphite to store systems.* labels at, say, 1 minute increments for a day, 5 minute increments for a month, and 15 minute increments for 3 years, and website.* metrics with a different policy. A default catches everything for which there is no rule, so if you want to measure something, just start saying it and fix the details later.

The other half that makes graphite such a nifty tool is the ease at which you can combine different metrics: with the web front end you just drag graphs on top of each other to start building up correlations. "Does the load on the system correlate to something an application is doing?"

Every so often I come across a tool that fits a particular niche very well, graphite is one of them. I'm actually excited about moving from playing with this to deploying it everywhere at work, which I think is about 30% of its utlity right there.

Future plans include tying graphite into a monitoring tool (probably xymon — having used Xymon's predecessor Hobbit in the past, I know it's pretty easy to cause events in it. Since graphite is already getting a lot of data already, you can use that data to trigger Xymon, which can do it's normal notification. I'm thinking of a ZeroMQ publisher that just publishes all of the graphite updates as they come in, and various subscribers can read the data and send Xymon events based on them.

Posted at: 22:12 | category: /computers/sysadmin | Link

Tue, 21 Feb 2012

Banana Nutella Snack

Slice a banana in half lengthwise. Spread Nutella on both halves (it won't stick well since the banana isn't exactly the worlds easiest thing to spread on, but stirring the Nutella well to loosen it up helps). Spring some ground flaxseed on both halves, put bag together, eat. Have a napkin handy, it's messy.

Posted at: 21:16 | category: /food/2012 | Link

Radish Parsley Salad

Mentioned by a friend, adapted from a recipe I found online.

Cut off the ends of the radishes, cut in half, then cut into 1/8 inch slices, making little semi-circles of radish. Pull the leaves of the celery (at least getting most of the big stems out). Prep the celery, slice very thin. Throw this all in a bowl.

Juice the lemon, and add the juice and the olive oil to the bowl. Salt and pepper to taste, toss well. Refrigerate for several hours.

When you first make this the parsley will be rather strong, but with time it mellows to perfectly go with the radishes. Note that if you have any lemon juice in the bottom of the bowl the red from the radish peels will start to bleed. Thought for next time: zest the lemon and add that to the mix.

Posted at: 20:27 | category: /food/2012 | Link

Cucumber Onion Salad

Craving something like this that my family used to make a lot in the summer when I was growing up, but not wanting to put a lot of mayo on it, I found this very simple recipe online and adapted it slightly.

Wash the cucumbers, then slice very thin. For bonus visual appeal, before slicing run a citrus zester down the cucumber. Slice the onion very thin. Put both of these in a bowl, throw on some dill and rice vinegar. Cover and refrigerate for several hours.

You want the sweetest onion you can get, unless you like strong onion breath, although I will note that it seems that the longer this sits, the more mild the onion becomes.

Posted at: 20:22 | category: /food/2012 | Link