I've been interviewing a lot of people for the QA position at StreamBase recently. Here's a question that I've found to be very useful during a phone screen:
Suppose you have a simple command-line calculator program called "calc". The program takes an expression on the command line, and prints out the result. For example, if you type "calc 1+2", it will print "3", and then exit. Tell me how you would design a test harness for this program so that it's easy to add tests for new calculator features, such as support for division, or trigonometry functions.
I encourage you to think about how you would solve this yourself, before you continue reading.
The vast majority (more than 90%) of interviewees will come up with something like this:
Organize your tests as a list of triples. The first element in the triple is the first operand, the second element is the second operand, and the third is the result. For example, to test 1+2, your triple looks like ("1", "2", "3").
If I get this response, I ask them to explain how the test harness would know what operator to use -- should it use addition, subtraction, or what? (Remember, I made it very clear in the original question that the test harness should be easily extensible to new features.) If you're playing along, consider how you would answer that question before you continue reading.
More than 90% will respond with either 1) add another entry to the tuple to store the operator, or 2) create a separate table for each operator. One problem with both of those responses is that they don't work if I want to test expressions with multiple operators. So I next ask the interviewee how they would test "1+2*5". If you're following along at home, here's another time to think before you continue.
At this point, most people fumble for a bit. Half of the candidates will just give up, while the stronger ones will usually come up with something like: use a list instead of a fixed number of elements, and read them in order from left to right. So 1+2*5 becomes ["1", "+", "2", "*", "5"].
Unfortunately this solution still isn't general enough. So then I pose my next stumper: how would you test syntax errors like "1+" or "*3-"?
At this point, I would expect a good candidate to either completely rethink their approach, or to have avoided this dead end in the first place. Unfortunately, nobody who has started down this dead end has ever successfully made it back out within the ten minutes I allow for this part of the phone screen.
So now you may be wondering: what's the correct answer? I think the ideal answer should go something like this:
Organize your tests as a list of string pairs. The first element of each pair is an expression to run the program with, and the second element of the pair is the expected output. You might also want to extend the pair to a triple, where the third element stores the expected exit code from the program.
Note that this solution is simpler than the most common response, and yet more powerful. It doesn't try to impose any structure on the expression to be tested, nor on the response, so it's perfectly reasonable to have tests like the following:
("1+", "Error: expected number after +")
("x+y", "Error: unrecognized symbol 'x'")
("cos(0)*3.1", "3.1")
("---7", "-7")
So, now you know my best QA interview question. Use it wisely.
Posted on January 16, 2006 09:12 PM
More management articles
I'm not a QA engineer, but I've actually tried writing test suites for my code, and the pair of strings is the simplest most obvious approach to me, and I didn't even think of doing it any other way. And if you think about it it makes sense, if you treat the program as a function, with the commandline string being the argument and producing a string of output. The way you look at the problem gets you quite far on the way to a solution.
Posted by: crzwdjk at January 17, 2006 03:27 AMThis may be an instance of the generally observable pattern in business that people tend to create structures and bueraucracy [sp.] for no earthly good reason - or continue using it for long after it has lost its usefulness. I suspect that this problem starts long before the workplace, in education - with students incented to provide the answers they think their teachers want to hear - without understanding why.
Posted by: Robin Green at January 17, 2006 07:06 AMlol I was like where's the trick. I was thinking you have a text file of inputs 1 per line and a text file of outputs 1 per line and your harness runs against text file 1 and collects the output for posterity. you can do a diff against text file 2.
Then when you want to introduce a new operator, say a unary operator, then you can generate a new pair of text files that are a function of each of the previous text files. etc etc.
unclear how to address floating point instability, but you can hand wave for the purpose of an interview.
don't forget different kinds of NaN.
sweet.
Posted by: Ron at January 17, 2006 01:41 PMMy first thought was something like Fitnesse or doctest, only with plaintext instead of HTML or Python.
Posted by: Darius Bacon at January 18, 2006 02:57 PMI was looking for such tips as I have to take an interview for a QA candidate :P
Thank you for the post.
Posted by: Shaikh Sonny Aman at March 13, 2008 01:08 AM