Daily Abstraction

Since I'm going to be moving to New York, we're selling our Brookline condo. If you know anyone who might be interested please spread the word -- especially if you know someone who works near the Longwood Medical Area. The first open house will be this Sunday, Feb 25.

Light-filled Brookline 2+ bedroom condo with updated bath and study. Working fireplace, hardwood floors, high ceilings, and lovely detail throughout. Abundant closets plus 100 square ft basement storage space. Shared roof deck and laundry. Located on a one-way road across from Emerald Necklace park and river. Five minute walk to many parks, Longwood medical area, Coolidge Corner, Brookline Village, and green D line T. Transferable rental parking ($100/mo) paid through 2007.

I'm going to be moving to New York City!

I'll be joining Goldman Sachs, working as a developer on a core part of their technology infrastructure. It's a big change for me -- the biggest company I've ever worked for so far only had about 200 people, and GS is approximately two orders of magnitude larger than that. I'll be working in a skyscraper right in downtown Manhattan, just a couple blocks from Wall Street and that bronze bull that you've probably seen pictures of before.

I've been asked to be discreet about the details of the job, so all I'll say is that it involves language design and optimization, highly scalable systems, and financial derivatives. Here's a paper that'll give you a flavor for what I'm talking about: Composing Financial Contracts, by Simon Peyton Jones. Unfortunately I won't actually be writing haskell code, but at least I'll be working with a group of people who deeply appreciate functional programming.

I won't actually be moving for a few months yet (date still TBD). In the meantime I'm still working at StreamBase, implementing cool new features for our upcoming release.

The In Our Time podcast has been pleasing me lately. I started paying attention when I came across a link to their episode on The Poincaré Conjecture. If you like history and culture, the rest of the episodes are also pretty interesting.

I went to a show by Firedrill! tonight at the Broad Institute. Firedrill! is a Boston-based a capella group who sings pieces by people like Peter Gabriel, Aretha Franklin, Van Morrison, and Seal. They use their voice to simulate drums, bass, guitar, sound effects, etc. They're really impressive, and very entertaining. They have a couple mp3's on their website, plus some more on their myspace page. My favorites were "Can't Stop" by Martin Sexton, and "If I Ain't Got You" by Alicia Keys.

If you haven't heard of StumbleUpon, you have to check it out. It has introduced me to dozens of cool sites that I never would have found otherwise. It makes the web seem much more vibrant, organic, and diverse than if you just rely on the few sites that you've become habituated to visiting.

I got admitted to the master's program at Brown! I'm starting in January (spring term). I got the acceptance letter in late April, but I didn't write about it here because I hadn't mentioned it yet to the people at work. But last week the VP Eng let it slip during the weekly development meeting, so now I can share the news with the world at large.

My wife has a friend who lives in London. We stayed at her house when we visited there two years ago. I found out yesterday that she was actually riding on one of the trains that got bombed. She's fine, and apparently in relatively good humor about the whole thing.

I'm finding it hard to absorb this information. It somehow seems related to when the tidal wave destroyed the hotel we had been staying at in Thailand only three weeks after we left.

There was an earthquake in Indonesia this morning. It caused tidal waves in India, Thailand, Sri Lanka, etc.

Here's a quote from a tourist that was staying in the exact same hotel where we stayed for three days, less than one month ago:

Swedish tourist Kjell Sköld, of Gothenburg, has described how the rising waters of the wave trapped him, his wife, Bibi, and their young children in their bungalow at Andaman Bay Resort, at Bang Tao.

“The water went out then came back in very, very quickly, taking everything with it,” said Mr Sköld. “When the water came into the bungalow, we put everything on the beds … all the windows were closed, so the water kept pushing everything up towards the roof. It pushed us up to the roof, then the roof came off and we floated away.”

After being washed down to land, Mr and Mrs Sköld managed to get their seven-year-old daughter, Stephanie, to safety in a high cement building, but then realized that their 10-year-old son, Sebastian, was missing.

He was found a few minutes later, about 200 meters away, sitting in a tree the flood had swept him to. “I can’t put into words what it feels like to be missing your son,” said Mr Sköld.

On November 1st, my wife successfully defended her PhD. To celebrate, we took a 2.5-week vacation in Thailand. Last night we uploaded the pictures from the trip here.

Last night, as I lay in my bed around midnight, I heard distant thumping noises coming from the direction of downtown Boston. It sounded like fireworks. Considering that last night was the first night of the Democratic National Convention, I figured it was either fireworks or a terrorist attack.

When at one point the thumping noises started coming frequently enough to merge into one long, continuous buzzing noise, I decided to get up and check the boston.com website, just to make sure that they weren't announcing some kind of disaster. The website was silent -- no news is good news. For good measure, I also checked the DNC website's events calender, but they didn't say there were any fireworks planned.

My point is this: if the politicians are going to keep scaring people over the risk of terrorist attack, and if they're going to specifically warn about attacks during the political conventions, and they're going to shut down major roads in order to avoid these risks, then they should have the decency and consideration not to set off fireworks in the middle of the night without mentioning it ahead of time.

Last Friday, May 28th, at 9:00 am, my wife and I got married. It was exciting and stressful and wonderful and very significant, and I am very happy. Afterwards we went to Montreal and Quebec City for our honeymoon. The weather there was almost perfect.

And that's all I really want to write here. All the rest of it was just for the two of us.

Last night, Cambridge city hall opened at midnight to hand out marriage applications to same-sex couples. My soon-to-be-wife and I decided to go, not because we're in a particular hurry, but because we wanted to be a part of the celebration.

I went to see "The Prince and Me" over the weekend. Out of about 300 attendees, there were maybe four guys. I enjoyed the movie quite a bit, but probably only because my expectations were just about zero.

Oh, and I also saw a preview for Sky Captain and the World of Tomorrow. I definitely want to see that. It looks like some kind of cross between King Kong, Brazil, and The Spirits Within.

My partner and I bought a new car this weekend. My old car, a 1993 Audi, has been slowly falling apart over the last year or two, and the time has come to move on.

We spent a couple days agonizing over whether to get a Toyota Camry or a Honda Accord. We compared safety ratings, resale value, aesthetics, responsiveness, and probably half a dozen other criteria. Eventually the decision came down to the fact that the Accord is much more attractive inside and it comes with more features, but the Camry has a much smoother ride and is a little cheaper. In the end, the smoother ride was the deciding factor. The Honda people say that the Accord has a "sporty" suspension, but I think "bumpy" is a more appropriate word. It was literally the bumpiest car we drove; even exceeding the RAV-4.

So here it is: our new car. Tricked out with a 200-watt stereo and side curtain airbags.

I'm starting a new job next week.

[That sentence should be read to the sound of trumpets in the background. It's a big change in my life, and I want to make sure you read it accompanied by a sufficiently impressive soundtrack.]

My new job will be at a small startup founded about eight months ago. I will be their 11th employee. They're still in stealth mode, so I can't tell you much about it yet -- even the name of the company is still secret at this point. However I can tell you that the company's main product will have a similar flavor to Endeca's: high throughput, large data volume, multithreaded, distributed, server-side C++.

My new job will allow me to take on more responsibility than I have at Endeca, and that's my primary reason for moving. I will start as the tech lead for a small group, and if things go well, I will probably have the opportunity to grow into more of a management role in a year or two.

I will be leading the QA group, so my initial tasks will be to design test harnesses, the test system, set up the nightly tests, get code coverage metrics, do performance measurements, etc. I've been thinking about how to ensure code quality for a while now (Debugging with Code Coverage, Sacrificing Quality, QuickCheck), so I'm looking forward to being able to focus on this aspect of the engineering process. I've got a few ideas that I want to try out.

Some people think of QA as just "pushing buttons" to try to make a program crash, but that's not what I'll be doing -- if that's all they needed, the company could have hired someone for a lot less than what they're paying me. Instead, my focus will be on automated testing (unit tests, regression tests, etc), and I will also be continually looking for ways to help ensure that the code is written correctly the first time (clear requirements, design reviews, designing for testability, encouraging test-driven development, etc).

I'll also, of course, offer any advice I might have on building scalable systems, since that's what I've been doing for the last four years.

One of the things I'm really looking forward to is working with the VP of Engineering at the new company. She has offered to help mentor me with the management aspects of my new job. I'm not a complete beginner when it comes to management, but I'm still on the steep part of the curve, so a mentor will be very helpful.

Another thing I'm looking forward to is having a broad strategic view of where the company is going, and what challenges it's facing. Because the company is still so small, I'll be able to know what everybody is working on. Longer term, I expect that that kind of broad understanding will come in handy in the rest of my career.

Until today I had been ignoring the various social networking sites like Orkut, because I just didn't see the point. But today I received my first invitation to join Orkut, from Dan Sugalski.

Dan is close enough to the borderline between friend and stranger that accepting the invitation suddenly seemed worth while. If the invitation had come from someone I knew better, like someone at work, I would have ignored it. If it had come from someone I knew less, like someone I'd never met in person, I would have ignored it. I'm not sure quite why I feel that way, and in fact I was taken by surprise to realize that there were any social connections that I would want to make public via a social networking site. I wonder whether it's a common reaction.

Once I had joined, I quickly became sucked in. I received an Orkut message from a co-worker and then... well... it's so easy to add links to people, that I ended up adding half a dozen co-workers within just a few minutes. Then I noticed that one of them belonged to the computer science "community", and I liked the little graphic, so of course I had to join too. And at that point the snowball was too big to stop.

So, for anybody who cares, here's my Orkut profile.

It's really really cold in Boston right now. A co-worker says a friend of his had a washing machine freeze solid while doing a load of clothes. Today my hair froze solid in the 15 seconds it took to get from the door to my car. It felt like I was wearing a plastic wig.

My first (and so far only) law of human behavior:

When talking to someone about a disagreement they had with somone else, you will take the side of the person you're talking to.

Keep this in mind if you ever ask someone for advice about an inter-personal conflict.

Here's a picture of what I've been doing this weekend:

This is a picture of one grid of a microarray. Each microarray has 4x12 grids, and each grid has 16x17 spots, for a total of 13,056 spots per microarray. My task is to look at the circle around each and every spot, and make sure it's the right size, that it's centered, and that it doesn't have any defects. It currently takes me about six hours per microarray, but eventually I hope to get that down to 2-3 hours. The goal is to finish nine microarrays (117,504 spots) by the end of the week.

Why am I doing this? Because my partner is in her last year of a PhD in molecular biology, and she has a lot of work to do before she's done. So I'm helping. There's a lot of stuff I can't help with, but circling spots is something that even a trained pidgeon could probably manage.

Because I'm a programmer, and because this activity really only occupies the visual part of your brain, leaving the rest of your brain idle, I've been spending my time thinking about how I could do this task more efficiently. I've considered trying to train a neural net. I've also considered reorganizing the user interface so that you can deal with "spots that look similar" en masse, instead of having to go through them in the same order as the physical layout of the microarray.

Also, since most of the grids in the microarray look similar (e.g. there's always a row of bright orange spots on the lower left), it might be useful to group all the spots that are in the same position, so that you can work with them all at the same time.

Unfortunately, as slow as pixing is, programming is even slower. So I'll probably never get enough incentive to try to implement these ideas.

It's been five months and 100 posts since the last time I asked for reader feedback, and in that time "About Kim" has gone from having 15 readers to over 200, so I figure it's time to ask again: Why do you read this blog? What would you like to see more of? What's getting old?

Also, feel free to share anything that you think would be interesting to the other readers: links to similar blogs, articles, projects, etc.

I'm 27 years old today. I like the number 27 because it's 3³. When I turned 26 last year I freaked out about how close I was to turning 30, because I had lost track of my age and thought I was actually 27. But now that I've had another year to adjust to the thought of being 27, I've made my peace with the idea of turning 30 eventually.

I had lunch with John Sequeira today. He's currently working on a .NET book for O'Reilly.

He's a mainly database guy, and he was curious about whether faceted navigation really adds anything over and above what OLAP and RDBMS's provide, so we discussed Diamond Wiki and why faceted navigation doesn't fit the relational model. His impression was that the product I work on is really a "hybrid" technology -- a combination of text search and star schema structure -- and that we therefore have the potential to create convenient solutions to problems that can be difficult to solve if you're only only using standard database technology. I'm glad he thinks that, since I'm not very familiar with database stuff and I wouldn't have been able to defend the technology very well if he had decided that it was just a home-grown implementation of what the OLAP folks have been doing for twenty years.

I'm interested in learning more about OLAP now, because it seems similar to some of the multi-dimentional summarization and analytics that we do. Unfortunately I don't think I can say much more about that without getting in trouble for talking about the company's products.

My website has just become the #1 result on Google when searching for my last name. I wonder whether kimbly.com will ever become the #1 result for "Kimberley" (it's currently #24). I strongly doubt it'll ever become the #1 result for "Kim", since I've got some stiff competition on that one (Lil' Kim being but one).

Via je', I discovered that John Sequeira has a blog. This serendipity really messes with my head, because I've never met je', while I have met John -- I even did some subcontracting for him once.

For the last three or four years, I've frequently had trouble concentrating. Sometimes I can spend an entire day unable to really concentrate on anything, whether it's writing code, talking with someone else, or even just reading a book. I have the vague memory that my mind used to be much sharper, and that I've somehow dumbed down noticeably since then, although it's been so long that I'm beginning to doubt the veracity of those memories.

More or less concurrent with this onset of stupidity, I started having very strong reactions to food: irritability, dizziness, and headaches. I explained this to my doctor, and she said I probably have hypoglycemia. So I assumed that the times when I had trouble concentrating were due to having low blood sugar, and I tried to compensate by eating food that I knew I would digest slowly, such as proteins, fats, and complex carbohydrates. This certainly helped me avoid the dizziness and headaches. However the confusion still didn't really go away, so I decided it was probably due to something else -- like my sleep schedule, perhaps, or maybe even just getting older, or being bored.

About a week ago a friend of mine got me a blood sugar meter, and I discovered to my surprise that when I have trouble concentrating, I actually have high blood sugar. So, recently I've tried keeping my blood sugar deliberately low by eating very small meals rather infrequently.

I've discovered that when my blood sugar is below normal, I can concentrate easily, and my fingers just fly through the code. In fact, I concentrate so much that I have regained that annoying habit of requiring a couple minutes in order to shift my attention from one thing to another. You know what I'm talking about -- you see it whenever you interrupt someone at work and they stare at you for a while before suddenly snapping out of it and asking you to repeat what you just said because they weren't actually listening.

As soon as I eat something, it only takes about ten minutes before my concentration is lost. I can literally feel it going. Actually, it feels like my concentration is still around, but it wants to concentrate on my forehead instead of whatever it is I'm trying to work on. Yes, my forehead, as silly as that sounds.

The drawback to keeping my blood sugar low is that it makes me dizzy, irritable, and gives me headaches. That's why I never really noticed before now how obvious it is that my problem is having too much food, not too little -- I had spent a considerable amount of effort making sure I never had low blood sugar. The only time I don't mind the symptoms is when I'm sitting in a chair, staring at a computer monitor for hours on end, without moving my head. If I move my head, I notice how dizzy I am.

I'll continue looking for a better balance of attention span and comfort, but for now my strategy is to eat very little during the work day, and then eat whatever I want once I get home. I'm also trying to eat breakfast as soon as I wake up, so that the worst of the confusion happens while I'm still at home, yet I have enough food to tide me over for a few hours once I get to work.

Higher ups at Endeca have gotten annoyed at me for writing about anything related to what I do at work. I was asked to remove all mention of Endeca from the Inverted Index Shootout pages, and to remove the Why Faceted Navigation Is Hard page entirely, even though faceted navigation is described in several places.

I have been careful not to write about anything that isn't already public knowledge, but apparently it is the fact that I digested this information into an easy-to-understand form that is causing people concern. My boss would have me believe that it is a universal in "corporate culture" that secrecy is to be preferred over openness, and that vague marketing hype is to be preferred over honest communication. However, I am not yet so cynical.

I am obviously very upset about this. I understand their concerns, but I would have hoped that someone who had technical expertise would have been able to state with authority that what I wrote is nothing to be worried about. Indeed, I have been told that the people in engineering all agree that this is a "tempest in a teapot".

Nevertheless, hotter heads apparently prevail.

Therefore, this is the last page I will write that will mention the name Endeca. If they only want publicity that came directly from the marketing department, then so be it.

I did a Google search for "Kim Burchett", and I was shocked to discover that not only are there several other people with my name, but Google ranks someone else as the top hit. Therefore, this post is partly an attempt to guarantee myself top billing, partly self-absorbed navel gazing (or voyeurism?), and partly a bid to ensure that people don't confuse me with these other people. Follow along if you like. I expect I'll freak out at least one of my same-named brethren by virtue of the fact that I'm a complete stranger who's talking about them on the internet.

My partner and I are buying a house. We went to the open house on sunday, made an offer on tuesday morning, and it was accepted five hours later. I think we're both still reeling a bit from how quickly we got ourselves into massive debt.

It's a small but beautiful 2nd floor condo on a quiet one-way road, right across the street from a park that has a little river through it.

Therefore, don't expect many posts for a few days. I'm busy dealing with crunch time at work (we're about two weeks before a release and the bugs are thick -- we're working all weekend), plus getting everything lined up for the house (inspection, attorney, mortgage, and tons of supporting documentation).

I recently passed the 100-post threshold, so now it's feedback time, y'all. Why do you read my blog? What's good, what's bad, what would be better? I know from the referrer logs that I have about 15 regular readers (depending on your definition of "regular"), but I want to connect humans to IP addresses. So now's the time to introduce yourself, and provide that meta-level commentary that you've been storing up. Talk amongst yourselves :)

Today's lesson: if you never make an effort to do anything, you'll get bored out of your mind. A life of leisure is neither fun nor rewarding.

If you're bored and unmotivated at work, and you've been spending your time surfing the web looking desperately for something more interesting, you may be stuck in a vicious circle. If you're bored with work and try to avoid doing it, you will become even more bored. That sense of boredom will permeate the work you're not doing, making you feel like the work itself is boring, which makes you try even harder to avoid doing it. (There's a chicken and egg problem here -- does the boredom come first, or the avoidance?)

If you find yourself needing, desperately, to work on something more interesting, then you may be stuck in this trap. If you begin to feel like your skills are atrophying due to lack of challenge, then you may be stuck in this trap. If you resent every little difficulty you encounter while working, because it means you actually have to pay attention to what you're doing, then you are probably stuck in this trap.

The only way to get out of this is to realize two things: 1) that moods are largely self-reinforcing, and 2) that if you don't actually put effort into doing something -- anything -- then you will be bored. Follow the chain of cause and effect backwards, and realize that it's your own damn fault you're going crazy.

Paying attention to something boring is like doing exercise -- at first, you hate it and you'd really rather go back to sleep. But eventually the endorphin high will hit you, and maybe it'll get you high enough to get out of your funk. After all, there's a reason you decided to become a programmer in the first place, right? You like programming. The endorphins are there waiting for you -- all you have to do is put in enough energy to shake them loose.

They feel so good...

Pictures from my vacation. London, Barcelona, France (Villefranche, Canne), Italy (Florence, Rome, Naples), Greece (Mykonos, Santorini, Athens), Malta, and Vienna.

I'll be on vacation in Europe starting today, until July 6th (three weeks). I may post a couple things here while I'm gone, but probably not much. Purple lines are airplanes, red lines are a cruise ship.

I think the subject of my paper will be "Automatic Selection of List Representation". There are so many different ways to represent a list, and each has its own tradeoffs in terms of memory usage and speed. For example, you have:

• singly-linked lists (the favorite of nearly every functional language)
• arrays (the favorite of nearly every imperative language)
• doubly-linked lists
• singly-linked lists with a tail pointer (if appending is important)
• sorted lists
• vectors (ala std::vector in C++)
• hashtables (if order is unimportant)
• binary trees (if lookup and insertion are more important than memory usage)

I will probably work on this in the context of haskell, since 1) lists are ubiquitous in haskell, and 2) haskell serves as a pretty good specification language, due to the lack of side effects.

The first thing to do, though, is to make sure that this hasn't been done already. Besides, if it has been done, I'd love to read about it. So I'm off to look for prior art.

I just spoke with the director of graduate study at Boston University, about whether I had any chance of attending graduate school without an undergraduate degree. Answer: not unless I'm clearly exceptional, for example if I had published one or more research papers. UMass Boston's answer was even worse; they wouldn't make any exceptions at all.

I take two things from this.

First, a master's degree is not so much about learning, as it is about putting in a certain number of hours. Otherwise, one might expect that I could simply take a test, pass an interview, or show some other proof of my ability to understand graduate-level classes.

Second, since one of my motivations for attending graduate school is to make a difference in the world by contributing to the state of the art, I may as well just start doing it. By contributing to the state of the art, I stand a chance of being admitted to graduate school. Of course, that begs the question of whether I would need to go to graduate school at that point, but I'll ignore that for now.

Either I am the kind of person who is able to publish a paper on my own, or I am not. If I am, then why not just do it? If I am not, then it's better to find out before going through grad school only to be disappointed.

Therefore, as soon as I finish the language shootout (which I still want to finish; I'm looking forward to erlang, which is up next), I will work on publishing a paper. I know it won't be easy; I've seen what it takes to publish something. But at least I know the area I'm going to work on: automatic implementation of data structures.

Now I know why I keep hearing airplanes flying overhead.

(Thanks to mapquest).