Chasing the Declarative Wild Goose

Recently at work, I've been part of a committee that is trying to figure out a strategic plan for implementing various features. The features are all big and hefty, with several possible implementation paths, and our goal is to try to avoid the kind of schizophrenic codebase that can result from solving every problem on a case by case basis, with no thought given to the possible simplifications and generalizations. Because this committee is thinking about features that may guide the company for the next year or two, I have to remain vague about what exactly the features are, but I wanted to try to share a lesson that I've been learning. Hopefully the vagueness won't make the lesson incomprehensible.

After considering the broad range of features that we wanted to tackle, we realized that about half of them could be solved all at once, as mere instances of a broader problem. Solving the broader problem, however, appeared to be somewhat of a research project. It wasn't clear whether we would be able to come up with a good solution, or whether it would turn into a tarpit.

So I started diving into more detail about how exactly the broader solution would work. After two days and eleven pages, I realized that it was basically a kind of abstract interpretation. Unfortunately, that meant that it would forever be incomplete (abstract interpretation always has a tension between precision and decideability). This was the first bad omen. Nevertheless, I continued on for a while, hoping that I could come up with some sensible subset of the feature that would be complete enough to be useful, yet still decideable.

Bad omen number two was when I realized that a part of our system that had previously been ultra-lightweight and stateless would suddenly have to either maintain an unbounded amount of state or else sacrifice some degree of correctness. I explored a few alternatives for bounding the size of the state, but eventually decided that there really was no natural boundary.

Bad omen number three was when I realized that because the new feature was intricately involved with the concept of time, the rest of the system would have to be very careful with the parts of our system that did involve storing state. This is because storing information as state effectively delays the flow of time -- so you end up having to store some kind of meta-state as well. Furthermore, in some cases, the correct handling of state seemed to require arbitrarily complicated heuristics in order to achieve correctness.

At this point I stopped diving, and showed what I had so far to the rest of the group. They recoiled in horror, and the feature ended up shelved.

Why did this feature fail? I think that's an important question to ask. One of the other members of the team had said from the beginning that he thought the feature would turn out to be infeasible. Why had he recognized this so early, even though nobody else had?

The answer that I've been coming up with is that the feature was fundamentally declarative, rather than imperative. By that I mean that the feature amounted to announcing some fact to the computer, and expecting the computer to react "in any appropriate way". I contrast this with an imperative approach, where you come up with a well-defined mechanism, and you can immediately see which problems it will solve and which it won't.

Here's an example that might help explain the distinction. Suppose you announce to a Windows XP machine that "This file shall be read-only from now on". In response to this proclamation, you expect the machine to do things like disable selection of that file in the "Save" dialog, have notepad flush its in-memory buffers if it happens to have the file open for editing, mirror the file to multiple remote locations just in case the local copy gets corrupted, add it to the list of read-only files maintained on the company intranet, etc, etc.

In one sense, the statement that "this file shall be read-only from now on" is very well-defined and comprehensible. Humans may think they know all of the things that that should entail. But from the computer's perspective, the possibilities are endless. You essentially have to revisit each and every piece of software on the machine, and imagine how it might be affected. And some of those decisions might not have any clear and definitive answer other than "whatever I happen to want". In short, it pretty much requires some form of AI.

I think that software engineers may have a peculiar vulnerability to this kind of folly. We're used to thinking up requirements, and then expecting that there will be some kind of reasonable implementation path, even if it may be difficult and involved. But there really are two kinds of requirements: requirements about well-bounded mechanisms, and requirements about unbounded declarative invariants. "Go through these motions" versus "make sure P is true". Constructive proofs versus existential ones.

I'm beginning to think that one of the main skills of a good architect is to be able to recognize the difference between these two kinds of requirements, and avoid the ones that smell like AI, or like formal proofs of turing-complete systems. I think that I have a natural tendency to try to chase the declarative wild goose, and that that tendency has been responsible for the demise of several of my spare-time projects. I'd like to think that I'm starting to be able to recognize that tendency in myself, and hopefully I'll be able to start avoiding it in the future.