Wed, 31 Oct 2012

Resurget Aquis

Those of you on The Quote File have already seen this, for everyone else, here's something you can read in full.

Before I go there, however, a bit of an update on how things have been around New York City, during and after Hurricane Sandy. You may have been following my brief notes via Twitter or Facebook, but those are really more suited towards short things; and while I'm certain to have a much fuller treatment of this past week in the next issue of Late Night Thinking, I thought I'd give you all an outline of how things are right now.

I live in Inwood, which is the far northern tip of the island of Manhattan. It gets relatively hilly in this part of the city, although I live just a few blocks from Dyckman Avenue, which is a kind of crimp in the end of the island and a bit of a low spot. "Zone C" for flood evacuations ends just across the street from me; for reference, the only areas the city ordered evacuations of were "Zone A". "Zone C" is essentially "hellfire will be raining down before you evacuate", and I suppose I could stand at the front door of my apartment and laugh at the people across the street as they washed out into the Hudson. Eyeballing it, though (and I did, trust me), my side of the street didn't seem that much higher than the other side of the street, so I waylaid any plans to laugh and point.

I was suffering through a stupid bit of what I like to call head cancer over the weekend as the storm approached, so my preperations were minimal. I normally keep a stock of staple food in the pantry anyways, so other than buying a couple things I was low on and making sure I had batteries with good charge and enough cough syrup to keep me doped up for a week, about the only thing I really did was make sure all the windows were shut, stack some towels next to them in case of leaks, and filling my two giant soup cauldrens with water for drinking and a large plastic tote bucket with water for cleaning in case the city supply crapped out.

And then I waited. And slept a lot. And watched bad disaster movies, which seemed a fitting theme for the day — "The Day After Tomorrow", "The Core" and "Star Trek IV". I maintained power and internet throughout the entire storm, although there was a half-hour where the UPS on my computer was a bit twitchy. It was persistently windy, although not more than I'd expect in a large midwest storm; it was also persistently rainy to the same degree. The only real excitement was seeing the glow from something exploding in New Jersey — I live next to a park on a high ridge, after that is the Hudson River and after that is New Jersey, so I'm guessing some electrical transformer over in Fort Lee decided to give up the ghost.

Work was canceled on Monday and Tuesday. Both days I wandered around taking some photos. Everything really interesting was roped off or hand a cop standing near it to keep people away, and I wasn't *that* much of an idiot to try to get around it. In my neighborhood there were a bunch of tree branches and limbs down, with a few large trees down up north, especially around Inwood Hills Park. Something near the boathouse up there got smashed, at least from the bits of lumber strewn about the marsh next to it. Some awnings on buildings and over businesses got twisted a bit but the biggest sign in my neighborhood was the sense that Fall happened all in one day — there was a carpet of tree leaves covering the sidewalks and streets.

Most of the damage in the city is "downtown" — the southern tip of Manhattan, and in the low-lying areas of Queens, although all five boroughs got hit in one way or another. You've all seen the pictures and videos of it, and I spent most of yesterday and a good deal of time at work today following things online. Winds were heavier in the flatter areas and near New York Harbor, and the surge of water that proceeded the storm was the highest surge in recorded history. Some neighborhoods are wiped out. Many people are without power, and will be so until the weekend or into next week.

Both the MTA (the local public transit authority) and Con Edison (the local energy concern) preemptively shut down things before the storm hit to minimize damage and to make it easier to restore service after the storm had passed. I'm an infrastructure nerd, so I followed both of these things closely. ConEd shut of the electric grid to parts of south Manhattan, and the MTA moved all of the subway cars and buses to yards and depots that aren't affected (or aren't *as* affected) by flooding.

As far as I've heard, things are about what you'd expect for a giant city that was in the sights of a giant storm to be. There were deaths, some accidental, some sadly preventable. A bit of looting, but I haven't heard of anything excessive, and as far as I know we're nowhere near "Escape from New York" territory here. The most desperate things are, again, power and mass transit. I can't speak much about power — I have it, work has it, and both never really lost it. The areas without power are going to be hellish for the next few days or a week until it comes back.

I can speak about mass transit. We were back at work today — Columbia being mostly northern Manhattan was spared the brunt of the storm, at least physically, although of course many of it's employees are spread throughout all five boroughs and beyond. Some bus service was restored as early as last night, with more today. It was packed, it took a long time, but we got there in the end. The buses, however, don't carry near as much traffic as the subway system does, and they have to deal with all the other traffic, which, as you all know even if you've never been here, is horrible in the best of times.

Sadly, however, what they subways have to share space with now is sea water. A lot of sea water, which is corrosive and full of gunk. Seven of the MTA tunnels under the East River were partially or totally filled with water. Three have already been pumped out. Some subway traffic crosses the East River via bridge, but a lot goes through tubes underground. And, even after the tunnels have been pumped out, you still have to deal with the stations and tunnels of southern Manhattan being flooded, and, well, southern and midtown Manhattan is basically where nearly all of the subway lines go. Even after they are pumped out, they have to be inspected — are the tunnels themselves physically secure? Is the signalling equipment undamaged by salt water? Are the tracks and switches free of debris? The South Ferry Station, which is nearly as far south as you can go on the island before walking into the harbor, is full of water from the tracks to the mezzanine, at least twenty feet of gunk. There are plenty of pictures online of tracks flooded, tracks washed out, tracks with *boats* on them, tracks with *boats on them with part of the dock with a deck chair **still attached** on them*. It's a mess.

That said, in what I consider a minor miracle, some limited subway service will be available tomorrow — I was hoping for a bit by this weekend, and I think it's amazing that I'll be taking the train to work tomorrow. It will be spotty, it will be packed, it will be slow, and it will basically pretend anything south of Times Square doesn't exist, but it will be there.

So, how am I? Pretty good, considering everything. Sadly, my favorite thing to do in the city, wander around, will be curtailed for a while, but I'm high and dry and have only the cough of a pack-a-day-65-year-old to show for my experience. And I can't really blame the cough on Sandy. The city will rebuild and go on, it's what we do. Things are going to be painful for a while, more for some than others, but in a year we'll be selling you t-shirts in Times Square about it.

I'll leave on this — what can you do to help, should you feel the desire to do so. First, we don't need your shovels, or your tarps, or your cans of soup. We need your money. The single most effective way to help is to donate to a large, established, trusted relief organization — they can take thousands of cans of soup worth of money and get a semi full of ten-thousand cans of soup here better than a thousand of you can. I'm a fan of the Red Cross, but any of the following would be useful:

The second most useful thing you can do is donate blood, if you are able to do so.

Thank you all to everyone who's asked about me, either via e-mail or Facebook or Twitter or IM, I appreciate it. Come visit me sometime, I've got a spare couch and I'm a good tour guide. Just bring a stout pair of walking shoes, and, well, give us a couple months.

Posted at: 22:15 | category: /nyc | Link

Thu, 25 Oct 2012

A ZeroMQ Protocol for Remctl

Here are some initial notes on writing a ZeroMQ backend for remctl. This is more of a quick brain-dump of ideas. The idea behind this is that often I use remctl as an authenticated communication layer, but it's native method of "fork a program, return the output" is painful for things that have expensive setup times (establishing a database connection, etc.), so I often end up writing a shim that talks down a socket to a persistent daemon. Having done too much local socket programming, I'd like to use ZeroMQ to make life much easier.

  1. Modify remctld to support talking to a ZeroMQ req/rep socket using the protocol below.
  2. Add an option to the remctld configuration, "type=", with values "exec" (the default) and "zeromq". If the type is "exec", the "executable" is just that. If it is "zeromq", the "executable" is treated as a ZeroMQ req/rep socket talking the protocol below.

The protocol:

  1. Send a multipart ZeroMQ request with the following components:
    1. The string "ARGUMENT", a string representation of the number of arguments to follow in the next part, and then each argument in subsequent parts. As is the convention in remctl, the subcommand is the first argument.
    2. The string "ENVIRONMENT", with a string representation of the number of environment variables to follow in the next part, then, for each environment variable, the name and value in separate parts.
  2. Return the following components, again in a ZeroMQ multipart message:
    1. For each chunk of output, return three parts:
      1. The string "OUTPUT"
      2. A string reprentation of the output stream number. Note: while the remctl protocol supports many output streams, the stock remctl client only supports STDOUT (1) and STDERR (2)).
      3. The output
    2. Finally, return either "RETURNCODE" and in the following part a string representation of the return code, or "ERRORCODE" and in the following part a string representation of a remctl MESSAGE_ERROR error code.

Example exchange: note that the parts of the ZeroMQ message are separated by |.

ARGUMENT|3|foo|bar|baz|ENVIRONMENT|4|REMOTE_USER|kula@TPROA.NET|REMOTE_ADDR|204.11.35.93|REMOTE_HOST|keymaster.tproa.net|REMCTL_COMMAND|test

OUTPUT|1|Here is some sample output\n|OUTPUT|2|Error output\n|OUTPUT|2|More err\n|OUTPUT|1|Another line\n|RETURNCODE|23

This is equivalent to a user kula@TPROA.NET on host keymaster.tproa.net with address 204.11.35.93 running remctl [remserver] test foo bar baz and getting back "Here is some sample output\n" on STDOUT, "Error output\n" and "More err\n" on STDERR, the line "Another line\n" on STDOUT and exiting with a return code of 23.

Random notes:

  1. Because of ZeroMQ's all-or-nothing method of operation, there won't be any stream of output back to the client. It will get all bundled up and sent back in one chunk via remctld.
  2. We count arguments and environment variables because we know them, but we don't count output to simplify handlers — they can just keep gloming on output until they are done.
  3. I represent all the integers above as strings so you don't have to worry about which byte-ordering integers are in, or how many bits to expect.
  4. The "ERRORCODE" option is primarily there so you can return ERROR_ACCESS from your backend, based on the user/host/addr making the connection. Or ERROR_UNKNOWN_COMMAND, I guess.
  5. To make the message smaller, I'm willing to accept the following string substitutions:
    1. ARGUMENT -> A
    2. ENVIRONMENT -> E
    3. OUTPUT -> O
    4. RETURNCODE -> R
    5. ERRORCODE -> E

Posted at: 15:21 | category: /computers/remctl | Link

Getting a list of git tags to a particular ref

For some reason, I'm always doing something related to backups, and frequently I need a way of testing backup software to see how it operates, if it's doing things properly and working the way I expect, etc.

One way I came up to test is to use a large git repository as a testbed — it's got easily reproducable history and you can check the correctness of any restore --- does a particular restore match what git thinks a particular ref should be? To do this, it's nice to find a linear path between two points in a repository's history. For that, I did this:

git rev-list --simplify-by-decoration --pretty=oneline --reverse HEAD

If you do that, you get a list, in chronological order, of every tag between the start of a repository and your current HEAD. If you want every commit, remove --simplify-by-decoration. The output is "[sha1] [title]"; you can use the [sha1] to check out that tag in the repository.

This works well for me with the OpenAFS git repository; I'm familiar with it, and it's got a rich set of tags. But any repo should do, as long as it's got enough history for you to find it useful.

Posted at: 14:25 | category: /computers/git | Link