The Value in Two Questions

Steam unfurled lazily from my tea in the quiet of agile alley.  A sudden eddy broke my concentration as did a subtle alert of an instant message.  Usually an IM from Steve, a tester on my team, arrived with a friendly salutation followed by some details around an unusual defect.  This time, she asked that I stop over.  I’m Reynolds McHumphrey – everyone calls me Mack.

Steve started as a business analyst out of school.  With an eye for detail, a mind for programming, and passion for technology, she moved into testing.  If she asked me to come upstairs to her cube, she needed a collaborative ear.  I recalled her team was working on closing costs transactions.

I walked up the stairs and through the windows I could see gray gathering in the west.  Leaves fluttered as a summer breeze picked up.  It’ll probably rain.  I arrived in Steve’s alley and saw John sharing some notes and diagrams with her.

“Hey Steve.  John.”

“Mack!  Hello!  John and I were reviewing his plans to migrate the automated regression tests.”

I looked at her trying to stifle surprise in my face.  “Oh?”

“Yeah”, she replied through a squint.

Skepticism had leaked in my voice.  The automated regression was just starting to demonstrate benefit.  It had its issues at first, but now we were relying on it to check hundreds of transactions with diverse inputs.  This allowed Steve and her team to delve into higher risk scenarios, collateral impact, and infrequent batch processes.  They had already found four defects that had been in the system as long as six months.  I turned to John.

“Mack”, he brimmed with impatience, “this new tool became available in Git about a month back.  It has native JSON libraries, improved performance, easier configuration – you know we’ve always wanted that, and lots of other cool stuff we can use.”

I kept my eyes on him while I took a sip of my tea wishing it was a stiff shot.  He was silent.  When it felt like the wake of his enthusiasm had crested and was returning to normal, he continued.

“Yeah.  So, we, me and the other automation guys, we reviewed the present automation, split up the functions and re-wrote it all.  I stopped over to see Steve so I could explain how to use it.”

Amidst the red flags waving around my head and dull thunder outside, I managed, “How do you mean ‘use it’?”

“Well, we want to deploy today it so you can start using it.  It’s really cool!  But it’s a little different in the start up.   Ahhh – and the reports.”

I looked at Steve.  Some hair had fallen over one eye, the other was looking at me, and she had a knowing smirk.  I looked back at John.  We were fortunate to have John and his team.  It required a bit of energy to help our business team appreciate the value of automation in testing.  They had concerns around the cost and maintenance, and rightly so.  Many experiments with automation often resolve to maintenance nightmares, technology-of-the-month trials, or flaky results.  I didn’t want our shop to follow in those industrial accidents.

We had described our implementation and maintenance plans, and how frequent use of automation could grow confidence in our products.  With the time saved, our testing teams could dive deep where we saw fit.  Steve’s team was the pilot for automation.

“John,” I started, “how much time did your team spend on this, ah, migration?”

Rain was hitting the roof – I may have left a car window open.

“Um, two of us worked about five or six days on it.  I worked on it at home a little, too.”

“I see.”  I held one arm in the other, pulled my hand across my mouth, rested my chin on my hand, looked at him intently, and let out a sigh.  He half smiled, looked at Steve and then at me.

Energy and initiative are rampant with John and his team.  Even while they developed the present automation, we did well to direct their energy without prescribing their work.  Since its introduction into our work stream, we haven’t been meeting as often.  Without a rudder, the automation team knew they “could” but neglected to ask themselves if they “should”.

“John, I appreciate your enthusiasm as always.”  He detected a “but”.  “But, we can’t deploy this new automation right now.”

“It’s so cool though and you’ll love it.”

“I’m sure.  Let me ask you, what is the added benefit to Steve’s team?”

His face betrayed confusion.  “Aaahh – ummm”.

“The present automation allows her team to investigate our more complex transactions.  It provides them the benefit of time.  And, we prioritize that time on complex and risky tests.  If we deploy new automation, her team may have to execute tests manually while you get it running.  We would lose not only the time spent in manual testing, but the time lost in what we could be testing.  That’s a big impact.”

The puzzle was taking shape for him.

“If all you have done is to provide a replacement, there is no net benefit to our testing effort.  That is, we would not get any more time to test our priorities for your effort.”

His expression was “Oh no”.  Or something like that.

“Mack, I – I’m sorry.  The tool just showed a lot of promise.  We even created automation for your other team.”

“Let’s talk about that.  We should also start meeting regularly.”

Epilogue
During our meetings, the testing team and automation team discussed methods of maintaining existing automation, and evaluating new tools.  Frequently, Mack led discussions around topics from the book “Specification By Example”.  He also encouraged the teams to answer two questions:  “Can we?” and “Should we?”.

The first question was easy but the second question prompted them to investigate how the testing team would benefit.  They decided that for the efforts of the automation team, the testing team should be able to, incrementally, expand their manual investigations.  Over time, the automation grew to provide not only more time in manual testing but it improved project pace and reduced project duration.

The Defect in the Delay

It felt like rain in agile alley.  The persistent din of frustration was lightly peppered with mildly satisfying bouts of usefulness.  Friday was as welcome as a third round at the Blue Bark Lounge.  I’m Reynolds McHumphrey – everyone calls me Mack.

“Good morning”, someone purred behind me, “maybe you’d like to walk me through this defect.”

Maybe. She was smart and I liked her but our project manager maintained curiosity at arm’s length. Asking about defects with all the smoothness of silk didn’t belay her purpose. My team had been testing the payment transaction for two weeks.  Between the web site performance and availability, a few of us were lucky to navigate all four pages.

When I checked the defects log this morning, three were new, five were being worked, and two were deferred. Of the three new defects, one was very severe.  I wonder if…

“We’re deploying to production in two weeks, Mack,” she started. My ears drank her declaration like old scotch. She turned towards me and her eyes narrowed slightly. “A new defect could pour cold water on those plans.”

The Testing team had very little time to evaluate the transaction so we focused on the high priority functions.  We reviewed that strategy with her last week.  Well, we were all in the same room anyway.

From the beginning, we’ve collaborated with her on how testing operates within the project.  Every now and then she became, well, concerned when project reality started winning the tug of war with the project schedule.

“When we discover something that looks odd or prevents normal behavior, we’ll open a defect,” I offered.  I suppose I knew the real question she was asking but I thought I’d start with a premise.  The fact that deployment was two days away was an artifact of an old estimate, and possibly a combination of poor implementation or lack of unit testing.  It often happens this way yet everyone seems surprised.  Or annoyed.

“I appreciate that, Mack,” there was a hint of gratitude in her voice, “We have limited time available for correcting defects,” her head tilted forward, “what are you expecting us to do?”

“I expect us to take a deep breath.” She saw a roadblock and I saw opportunity for both the project and for her. Her eyes narrowed slightly then softened. I sighed purposefully.

“Having the time to fix defects is your decision – let’s review them and consider the business risk if they’re not addressed.”

Her lips pursed and a suggested smile. “Getting business input will take time, Mack,” she sighed almost annoyed.

“It would if we need it,” I replied matching her tone.  “The defect description already contains a statement of business impact.”  Not only was my team a fine set of testers, they understood the transaction, and the business need for the transaction.

They could describe how the defect impacts the product from both a technical and business point of view.  With an awareness of their audience, the defect spoke to both developers and project managers.  In that manner, my team could continue to test while others considered the defect’s disposition.  We’ve written defects like this for the whole project.

“I recall you mentioning that – how does that help us?”

“It means we have the information we need – it means you and I can evaluate the risk now.”

Her eyes remained on me while she considered this. “Shouldn’t you be testing?”

“This is testing.”  I could have added that we found the defects by testing helping her to connect the dots.  Stating the obvious has its purpose but not this time.  She didn’t blink so I needed to walk her through my world.  Again.

“Part of our test planning was learning the transaction and the technology.  We did that weeks ago so when we write defects you and others can considered them quickly.”  I paused to help that idea bake.

“Especially in crunch time.  You should have only to consider the impact and decide what to do with the defect.  If it’s not clear, I can help”, I added while opening the defect description on my computer.

Her face relaxed – a step in the right direction.

“Steve opened this one this morning.  It looks like the payment balance function is off by a few cents.”  I glanced up.  “Over time, we could lose hundreds of dollars a month at present production volumes.”

“Show me where it says that,” she requested as she leaned over my desk squinting at my screen.  I highlighted the paragraph.  After a minute, she stood up still looking at my screen.  “We have two days. What is your next step?”

I don’t answer questions like this – I actually see it as a defect in her thinking.  It’s not my responsibility to determine when a product defect should be fixed.  I finally had context to help her see that it is both a significant business impact and, more importantly, her decision.  Maybe this time it’ll sink in.

“We found this error in a common usage scenario.  The bank account holder makes an on-line payment through their account, the account detects a low balance, and funds are transferred from their savings account which is also near the low balance threshold.  This triggers the balancing function, and the balancing function adds or subtracts a few cents in error.”

She nodded in understanding.

“The error is in the balancing function.  I reviewed the code.  The developer mixed data types resulting in a truncation.”

Her eyes moved from the screen to mine. “How do you know that?”

“Steve verified it with the developer and told me.  The developer thought it might take a half hour to correct.  Steve made suggestions for her unit tests.”

“Good work.” Mmmm – another scotch. “I’ll have them work the defect,” she said turning to leave. “Thanks Mack,” she threw over her shoulder.

Epilogue
While the developer made the correction, Steve found two other transactions that triggered the balancing function and exhibited the same behavior.  During the code review, they watched a demonstration of the new code through the unit test.  When it was elevated later that afternoon, Steve executed multiple scenarios demonstrating the resolution.

Writing Business Focused Defects
Mack and his team learned how to write better defects in the BBST Bug Advocacy course.  For more information, visit http://www.testingeducation.org/BBST/bugadvocacy/.