Interview Question for QA Engineers

I've been interviewing a lot of people for the QA position at StreamBase recently. Here's a question that I've found to be very useful during a phone screen:

Suppose you have a simple command-line calculator program called "calc". The program takes an expression on the command line, and prints out the result. For example, if you type "calc 1+2", it will print "3", and then exit. Tell me how you would design a test harness for this program so that it's easy to add tests for new calculator features, such as support for division, or trigonometry functions.

I encourage you to think about how you would solve this yourself, before you continue reading.

. . . . . . . . . . . . . . .

The vast majority (more than 90%) of interviewees will come up with something like this:

Organize your tests as a list of triples. The first element in the triple is the first operand, the second element is the second operand, and the third is the result. For example, to test 1+2, your triple looks like ("1", "2", "3").

If I get this response, I ask them to explain how the test harness would know what operator to use -- should it use addition, subtraction, or what? (Remember, I made it very clear in the original question that the test harness should be easily extensible to new features.) If you're playing along, consider how you would answer that question before you continue reading.

. . . . . . . . . . . . . . .

More than 90% will respond with either 1) add another entry to the tuple to store the operator, or 2) create a separate table for each operator. One problem with both of those responses is that they don't work if I want to test expressions with multiple operators. So I next ask the interviewee how they would test "1+2*5". If you're following along at home, here's another time to think before you continue.

. . . . . . . . . . . . . . .

At this point, most people fumble for a bit. Half of the candidates will just give up, while the stronger ones will usually come up with something like: use a list instead of a fixed number of elements, and read them in order from left to right. So 1+2*5 becomes ["1", "+", "2", "*", "5"].

Unfortunately this solution still isn't general enough. So then I pose my next stumper: how would you test syntax errors like "1+" or "*3-"?

. . . . . . . . . . . . . . .

At this point, I would expect a good candidate to either completely rethink their approach, or to have avoided this dead end in the first place. Unfortunately, nobody who has started down this dead end has ever successfully made it back out within the ten minutes I allow for this part of the phone screen.

So now you may be wondering: what's the correct answer? I think the ideal answer should go something like this:

Organize your tests as a list of string pairs. The first element of each pair is an expression to run the program with, and the second element of the pair is the expected output. You might also want to extend the pair to a triple, where the third element stores the expected exit code from the program.

Note that this solution is simpler than the most common response, and yet more powerful. It doesn't try to impose any structure on the expression to be tested, nor on the response, so it's perfectly reasonable to have tests like the following:

("1+", "Error: expected number after +")
("x+y", "Error: unrecognized symbol 'x'")
("cos(0)*3.1", "3.1")
("---7", "-7")

So, now you know my best QA interview question. Use it wisely.

Posted on January 16, 2006 09:12 PM
More management articles

Comments

I'm not a QA engineer, but I've actually tried writing test suites for my code, and the pair of strings is the simplest most obvious approach to me, and I didn't even think of doing it any other way. And if you think about it it makes sense, if you treat the program as a function, with the commandline string being the argument and producing a string of output. The way you look at the problem gets you quite far on the way to a solution.

Posted by: crzwdjk at January 17, 2006 03:27 AM

This may be an instance of the generally observable pattern in business that people tend to create structures and bueraucracy [sp.] for no earthly good reason - or continue using it for long after it has lost its usefulness. I suspect that this problem starts long before the workplace, in education - with students incented to provide the answers they think their teachers want to hear - without understanding why.

Posted by: Robin Green at January 17, 2006 07:06 AM

lol I was like where's the trick. I was thinking you have a text file of inputs 1 per line and a text file of outputs 1 per line and your harness runs against text file 1 and collects the output for posterity. you can do a diff against text file 2.

Then when you want to introduce a new operator, say a unary operator, then you can generate a new pair of text files that are a function of each of the previous text files. etc etc.

unclear how to address floating point instability, but you can hand wave for the purpose of an interview.

don't forget different kinds of NaN.

sweet.

Posted by: Ron at January 17, 2006 01:41 PM

My first thought was something like Fitnesse or doctest, only with plaintext instead of HTML or Python.

Posted by: Darius Bacon at January 18, 2006 02:57 PM

I was looking for such tips as I have to take an interview for a QA candidate :P

Thank you for the post.

Posted by: Shaikh Sonny Aman at March 13, 2008 01:08 AM

Thanks for the post. This will help me for sure.

Posted by: free games at September 28, 2009 02:42 AM

My first thought was something like Fitnesse or doctest, only with plaintext instead of HTML or Python.

Posted by: auto insurance for college students at October 18, 2009 02:55 PM

As humans, we're simply born into one of these dog training classes: chief or follower. As these couple types are extremes of one another, you may simply be a chief, or a buff, tiny is there of any fashion in between these puppy training.Some might contest though, that you could be both at one time, the chief to numerous and a buff to one.There are more teach dog tricks to outlook out for to review if or not you're a chief or a follower. The bulk all over individual aspect is the abilities to generate determinations not simply for yourself, but in supplement for remnant as prolonged as dog behavior training.Take this covering for example; each post-graduate learner is rewarded a cubicle each, which doesn't draw seal for loose though. Most would just announce no just because it doesn't draw seal free to wholesale dog supplies. The actual justification behind their tenacity is the portion that none wants to act on away from the crowd.When one or couple of them creates a tenacity to receive one cubicle in any dog supplies, remnant in the past prolonged followed not far behind. One later another, the cubicles are being filled. Sadly, there just isn't passable room for everyone to wholesale pet supplies. Even as learners with very competent standings and privileges, not all are competent to take find how to lose weight and generate stern life correcting determinations on their own. A buff would perpetually have determinations organised for them, perpetually staring to lose weight to play-act first after the time arises. If you're one of those any person who essentially have a darling to move out with the accumulation, it's come seal time to commence correcting and take find fault for once. belly fat is a yardstick journalist for a many cluster of themes and subjects. He in supplement drafts on braun bathe and transform and braun bathe and charge.

Posted by: dogtraining at October 21, 2009 08:53 PM

These Women's Classsic Tall 5815 Boots in pink flower print can either be worn at their normal height if you want Ugg boots, but in order for you to create a more casual look to what you are wearing they can be pushed down Women's UGG Classic Tall 5815 Boots in Baked Clay

Posted by: ugg boots on sale at October 27, 2009 02:25 PM

Love this man. Great stuff, great prices, free shipping. Email him if you cant find what you are looking for and he will look for you. Good customer service.louis vuittonSomeone who is knowledgeable in louis vuitton.Another guy from hong Kong (Largest louis vuitton second hand city, I believe) he has great stuff at fair prices but his shipping is insane

Posted by: louis vuitton at October 29, 2009 08:12 AM

Love this man. Great stuff, great prices, free shipping. Email him if you cant find what you are looking for and he will look for you. Good customer service.louis vuittonSomeone who is knowledgeable in louis vuitton.Another guy from hong Kong (Largest louis vuitton second hand city, I believe) he has great stuff at fair prices but his shipping is insane

Posted by: louis vuitton at October 29, 2009 08:12 AM

Love this man. Great stuff, great prices, free shipping. Email him if you cant find what you are looking for and he will look for you. Good customer service.louis vuittonSomeone who is knowledgeable in louis vuitton.Another guy from hong Kong (Largest louis vuitton second hand city, I believe) he has great stuff at fair prices but his shipping is insane

Posted by: louis vuitton at October 29, 2009 08:12 AM

Good day! Thx for your great post and Im thinking about how to introduce my ugg boots to you cos Im not sure if u like this. Many people who live in the cold area like the ugg boots, especially the north-europe. Our uggshare.co.uk supply the cheapest classic cardy ugg boots, class tall ugg boots, classic short ugg boots and other series of uggs, hope my words didn’t trouble u, cy!

Posted by: ugg boots at November 5, 2009 03:40 AM

Eliminating the need for the extra trouble to wear ugg boots uk
ugg boots saleuggs, in the extreme sports, look for happiness in life, with UGG Classic boots and nike sb showed off your tall body, do not worry about being laughed at, dancing new style ugg australia and nike sb dunk . The flavor of life that is colorful, there are Christian Louboutincheap nike dunksUGG cardy boots、and nike sb for sale also used fear to creep away money from your pocket Mody ? Love beauty, you might even love UGG Classic tall bootsUGG Nightfall bootsUGG Classic short bootsnew nike sb。There are an infinite high power drivers of popular front ugg boots saleugg saleuggsnike dunkdiscount ugg bootsnike dunk sb, our website also have UGG Nightfall bootsbutton ugg bootsbailey button uggugg button bootsbailey button bootsnike dunk midnike dunk highChristian Louboutin Boots, pick your favorite, do not hesitate!

Posted by: ugg boots sale at November 12, 2009 03:44 AM

People all over the world know the abercrombie and fitch,but not everyone really knows how fashion the abercrombie is,hollister is the Legend maker. Everybody wears the hollister clothing would be the abercrombie mensand the abercrombie womens, if you want know you can search the Ruehl No.925 or abercrombie outlet in the www.google.com .

Posted by: anf at November 13, 2009 12:40 AM

Uggs on sale now.UGG Classic Cardy Boot makes me different form the other

girls. The UGG Bailey Button Boot is a good choice for female.

Posted by: ugg classic cardy boots at November 16, 2009 11:41 PM

Louis Vuitton , commonly referred to as Louis Vuitton Sneakers and Gucci Diaper Bags on Sale, or sometimes shortened to Canvas Handbags has become one of the most louis vuitton alexandra wallet Agendas luxurybrands Tiffany In Hongkong.

Posted by: 1 at November 17, 2009 01:41 AM

UGG Bailey Button bootsis a new style in 2009.The classic cardy uggs boots is another hot boots that worth of buying.And the classic tall ugg boots will make your winter amusing.And now uggs on sale,if you are looking for such a boot,the ugg boots is good choice this year.

Posted by: UGG Bailey Button Boots at November 17, 2009 02:05 AM

I love Women's UGG Classic Cardy Boot 5819 very much. And the new style UGG Mayfaire Boots is aslo a good choice this year.

Posted by: UGG Bailey Button Fancy Boots at November 17, 2009 03:13 AM

I think I will try to recommend this post to my friends and family, cuz it’s really helpful.

Thanks for this nice post, i like it very much, write this article more. uggs boots on sale now!Free shipping!


Posted by: uggs on sale at November 17, 2009 08:04 PM

lxjuan08ed hardyis a famous ed hardy store which sell directly ed hardy clothing, shoes, boots, swim suit and other cheap ed hardy.

Browse through our catalogue of hundreds of ed hardy clothing,Ed hardy Shirt,Ed hardy ... Buy Ed Hardy today for a gift, for a friend, or for yourself.

ED HARDY Online Shop,Ed Hardy Store by Christian Audigier,coach handbags,rolex watches.We supply cheapest Newest ed hardy discount 2009 Commodity New styles.


ed hardy sale

Posted by: ed hardy at November 18, 2009 08:15 PM

lxjuan08ed hardyis a famous ed hardy store which sell directly ed hardy clothing, shoes, boots, swim suit and other cheap ed hardy.

Browse through our catalogue of hundreds of ed hardy clothing,Ed hardy Shirt,Ed hardy ... Buy Ed Hardy today for a gift, for a friend, or for yourself.

ED HARDY Online Shop,Ed Hardy Store by Christian Audigier,coach handbags,rolex watches.We supply cheapest Newest ed hardy discount 2009 Commodity New styles.


ed hardy sale

Posted by: ed hardy at November 18, 2009 08:15 PM


I bought one kids

mayfaire-boots-c-99.html">ugg mayfaire boots

for my

daughter last week, Very fast shipping, exactly as described, great

value!!

Posted by: 3th at November 19, 2009 08:02 PM

We are the best online sales for the china wholesale . Here you can have a large of choices of kinds Ugg Boots,Converse Shoes,Timberland Boots,puma shoes,Nike Shox Shoes ,Nike Dunk SB Shoes,Nike Air Max,Links Of London,Tiffany Jewelry,Dior Handbags?,jimmy choo handbags ,Cartier Watches, 8GB Mp4 Players,Bluetooth Car DVDs. All our cheap online cheap goods are high quality and original packages, and best service. We offer our customers the best service, 7 days arrive at your door.Enjoy your easy and happy shopping with us.

Posted by: cheap goods sale. at November 19, 2009 10:02 PM

Do you like sports? If you like it,the NIke Air Max TN and Nike Air Max 360 is a good choice for you.And now the there is a good news,CheapNike Air Max running Shoes is waiting for you.I think The New Nike Air MAX 2009 Running Shoes makes you great comfortable.

Posted by: Nike Air Max shoes at November 20, 2009 03:29 AM

Burberry watch for sale
Bvlgari watch for sale

Posted by: chopard watch for sale at November 22, 2009 10:01 AM

Your article is write very well, I like it very much ~
ghd Hair Straightener
ghd Hair Straighteners
ghd straighteners
GHD
hair straighteners
bose in ear headphones
bose on ear headphones
bose headphones
nike sb dunks
nike dunks
I wish you have a wonderful day!Thank you.

Posted by: ghd straighteners at November 22, 2009 08:23 PM

Timeberland boots on sale now! Do you want moreTimeberland boots information?Just move your mouse and press it to our Timberland shoes online store.

Posted by: Timberland uk at November 22, 2009 11:14 PM

Laptop Battery Laptop Battery Laptop Batteries
Laptop Batteries discount laptop battery
discount laptop battery
notebook battery notebook battery
computer battery computer battery
replacement laptop battery replacement laptop battery
notebook batteries notebook batteries

Posted by: Laptop Battery at November 23, 2009 10:00 PM

chaoying nfl jerseys
Giants Jerseys
Patriots Jerseys
Packers Jerseys
These things are good, I hope you can continue to maintain!


Posted by: jerseysleague at November 29, 2009 03:44 AM

Paul Smith has been collaborating with a few different organizations recently and his latest venture sees him teaming up with Japanese bicycle saddle, Paul Smith bag, on a limited edition leather saddle. The release features Smith’s signature stripes where only 20 will be available at Paul Smith Clothing store in Japan and 8 at the Floral Street store in London. Get on it quick

Posted by: fafe at November 29, 2009 11:10 AM

Just wanted to say great job with the blog, today is my first visit here and I’ve enjoyed reading your posts so far
ugg bailey button
Wow, my ugg classic mini will not be coming off now! I’ve had them on for 12hrs strait and I do not want to take them off. Thanks for everything, well worth the wait.

Posted by: ugg bailey button boots at November 30, 2009 06:17 AM

Spring is near,every girl wants to be the bride in the special season.They are eager to put on beautiful Wedding Dresses or the Bridal gowns.During the day the Wedding gowns is the good choose,and the night,if you want radiant,you need the Evening gowns.
About the bridesmaid,they have to wear Bridesmaid Dresses in order to avoid grab limelight with the bride.And the Flower Girl has the Flower Girl Dresses,too.In the wedding,the Cocktail Dresses and the Evening Dresses is necessary,too!And remember,the Wedding Dress of the Bridal Dress must be the most glaring!

Posted by: weddingdressclub at December 1, 2009 12:28 AM

Gucci is famous as top quality, Louis Vuitton Tivoli PM luxurious, modern, and sexy luxury goods. Louis Vuitton Replica Tivoli PM It is a symbol of status and wealth. From time to time, Gucci belt is favored by most of people very much.As we know, Gucci belts stand for fashion trend,
Louis Vuitton Tivoli PM Replica popular elements, culture, conciseness gucci and noble. As a gentle and fashion business man or woman, Gucci belt is your best choice. Gucci commercial belts adopt real leathers, and the leathers have high hardness, toughness and scratch resistant. Belt buckles adopt high quality CTB alloy, the golden and silver color of them will never be faded. You like casual clothes and belts, because casual living Speedy 25 lifestyles focus on the simple pleasures of life. We believe that you will not hesitate to choose Gucci belt. Gucci casual belts own fashion and lovely style, bright color. The material of belt is high quality real leather or durable PU leather. Therefore, if you own this kind of Gucci casual belt, you will become a popular star.

Posted by: fanqin at December 1, 2009 12:50 AM

Thank you for the great job!In order to welcome to Christmas, recently,our company is on ugg boots promotional sales and the prices of all products such as ugg sundance II boots or ugg bailey button boots are lower than any time!And the biggest surprise is that you can buy one get one free,that is once you buy an ugg mayfaire boots , ugg coquette, or ugg classic crochet,you can get one gift for free.Don't hesitate, to get a free present, check it on http://www.uggsnowbootsbest.com/

Posted by: uggsnowbootsbest at December 1, 2009 04:06 AM

Good job!Thank you for your message!It help me so much! Christmas is coming,there are big sales promotions on http://www.nikeaf1jordanshoes.com/,ugg bailey button and ugg classic argyle knit boots on our site are lower than ever,you may be intersting in ugg mayfaire boots or ugg nightfall boots,ugg sundance II ,come on to choose your favorate goods!

Posted by: nikeaf1jordanshoes at December 1, 2009 04:07 AM

You should go to www.uggssky.com who want to buy cheap ugg boots ,classic tall boots,
short tall and bailey boots,buy youself or your friedns a pair of ugg boots for the christmas!

Posted by: UGG Bailey Button Boots at December 1, 2009 09:14 AM

ugg boots sale at a time when large heat has set off a winter 08/09 to Ewha.Outlines the

warm wool winter fashion degree to wool crocheted wool ugg boots uk to join after the

body does not stimulate your skin soft, soft comfort UP. Fashion can not without
ugg boots presence in Winter.and ugg boots have many series.such as:
ugg classic
UGG Bailey Button boots
ugg nightfall
ugg kids

Posted by: uggboots at December 2, 2009 07:48 PM
Post a comment









Remember info?




Prove you're human. Type "human":