Saturday, November 12, 2005

Defect reporting etiquette.

This should go without saying, however sometimes a little reminder is needed.  First of all, I presume that most of the readers here are actual developers of some form or another.  Maybe you're developing the next “killer app,” maintaining an internal accounting system, or just plinking out code in your spare time.  For most of you, people besides yourself use your application to make their life a little easier.  Basically, we all understand that the developer is the producer and the user is the consumer.  For one segment of the overall market, those consumers are other developers just like yourself.

Now that we've established that we all share at least one thing in common, even besides Delphi, is that we're all software developers.  I think we can certainly empathize with one another on one thing; defect or bug reports.  I certainly think that there is a certain etiquette one must follow to create a good, clear, and concise defect report.  This is especially true when reporting a defect to a fellow developer.  This developer may be on your team, or some unknown developer that creates some set of components on which you rely.

First of all, make sure you are able to reproduce the bug.  This is the most basic requirement in order to increase the likelyhood that the defect will be fixed.  The next item is try to narrow the case down as much as possible and eliminate as many variables as possible.  This will serve to keep the developer working to fix the problem focused on actually further understanding the issue.  Otherwise you risk sending that developer on a wild-goose-chase, which only wastes time and further delays the final fix, if it is even possible.  Sometimes a bug report may contain so many steps or variables that getting down to the real, core problem becomes an excercise in frustration and futility.

Finally, put yourself in that developer's position.  Remember, you're a developer, so please afford them the same courtesy you would want when someone reports a defect in your code.  What information would you want in the report?  What is a good threshold for numbers of steps?  It is rarely helpful to just dump you're entire project as an attachment in order to demonstrate the bug.  Most of all, just think about the perfect bug report from your perspective and try and come as close to that ideal as you are able.

“Oh sure“, some may be saying, “then I'll never report a bug because I just don't have the time to mess with all that crap!”  I'm sorry, but everyone has the same number of hours in the day.  You're time is just as valuable as everyone else's.  In fact by not reporting a defect, you're actually hurting your own productivity!  No software product is going to be 100% free of defects (mainly because many item's design and operation may be considered a “defect”), they can only be made better than before.  That can't happen if the developers of said software are unaware of the problems.  They also cannot be expected to use the product in the same ways that everyone else does.  That's why there are field or beta test programs.  These are essentially, volunteers that spend time using the product as they would normall use it.

“What if I just can't seem to faithfully reproduce a bug?”  In the ideal world, every defect is reproducable.  You should strive to narrow down the steps to as few as possible, and to eliminate as many other variables as possible.  However, in reality, it is never that simple.   Sometimes an encountered defect is simply not reproducable on command.  You may be asking, “So if I can't reproduce a defect, do I bother even reporting it?”  The simple answer is, “Yes!”  It's true that a single, lone non-reproducable report has little, if any chance of being fixed.  But that is the beauty of a community defect reporting tool!  By simply reporting the error message, and all other pertinant data you actually have, even if that is not enough information to actually reproduce the problem, you are adding to a growing database of data points.  What if you run into a defect and spent 30 minutes trying to narrow the steps or even just reproduce the problem all to no avail?  So you file a report anyway with all the data you currently have about the problem.  Then someone else sees that report and immediatly is able to spot some commonality between your report and 15 other reports that seem to have the same unreproducable issue.  Your defect report may have been the final bit of information in a long chain of data and finally forms a complete picture of the whole problem.

“Nobody can reproduce my defect report, but I have no trouble reproducing it! What do I do?”  This is another common scenario that sometimes discourages and frustrates people.  Don't fret!  We still want to have these kinds of reports.  Provide as much relevent information as possible, because just like the case in the previous paragraph, your defect may be the last key in a long standing quest to finally get to the bottom of a very complicated issue.  Ultimately, the developers and quality assurance team responsible for fixing and testing a certain defect must be able to reproduce the problem.  This makes sense, right?  I mean how else can you be sure you've actually fixed a defect if you cannot compare the behavior before and after making a change to the code?  You also need these steps and reproducability in order to plug this test case into a whole set of regression test suites.  How else can you make sure that a once a defect is fixed that it actually remains fixed and doesn't somehow re-occur in a later build?

“I don't have time to actually give you good steps and a good description of the problem.   It's not my job to be QA!  I'll just tell you that there is a problem in general terms and you guys can try and figure out the defect and fix it from there.”  Fair enough.  This is similar to the first scenario I presented, but there's some more things we need to cover.  Everyone's time is valuable and everyone's circumstances are different.  However, I do need to point out that if this is your normal modus operandi, there is something you need to be aware of.  Only you are the foremost authority on exactly how you work and what steps you took to encounter a defect.  It is very difficult for the QA staff to extrapolate enough information to form a decent defect report.  Also, unless clear and concise steps are given, how can we even be sure that we've actually found the defect you're reporting?  What if a completely different defect is found when trying to narrow the steps and that is attributed to the original reporter's report?  Sure, we've found and possibly fixed a defect, but how did that help the original reporter?  Their defect is still there even though we've now closed out the report as fixed.  That's not very helpful for all involved, is it?

I hope folks understand that by following a few simple rules and creating a defect report that you would like to get is actually you contributing to your own success.  In this one case I have to say that karma is the best way to describe this process.  By paying it forward and actually becoming a part of the solution, it will all eventually come back to you.  Even though you may think your contribution is tiny and insignificant, the fact of the matter is that your contribution when compounded with everyone else's can pay back in large dividends.  Quality Central is your friend!

17 comments:

  1. Latelly you don't speak to much... but when you speak, it sounds like the evangelium :-) Ok.. you are right... I made some of the common mistakes you comment here in your post... I will try to correct myself as doing this is better for all of us.

    ReplyDelete
  2. Of course, even when you follow these guidelines it sometimes gets you nowhere. For example, look at Quality Central #8866. i think it fits the bill as a good quality report. Definitely reproducable and well described.


    I posted it, waited 2 months for it to be "opened internally", then waited 6 months for a "resolution" or "won't do". no communication with me occurred. No explanation. No results.


    Very discouraging.

    ReplyDelete
  3. Hi Allen,


    Have you looked at Quality Central #13652 ?

    I guess that I have respected all points :


    - defect is well discribed

    - defect is always reproducable

    - demo project is boundled


    I post this QC nearly 6 month ago.... and it is STILL reported... as Doug said it is very discouraging !




    ReplyDelete
  4. Take a look at this doc, it describes the bug reporting process in such a nice way :-)


    http://www.chiark.greenend.org.uk/~sgtatham/bugs.html


    ReplyDelete
  5. Very nice post :) However, If I would be creating bug reports for your product, when would I find time to actually do my own work? Nice bug reports are great!! But to rely that people will donate their own time and money freely to help Borland show more profit, becaues it saved on Q&A, is like trying to earn money on open source. The same way as there is curtosy from the customers point of view, there is also the curtosy from the Borlands point of view. Dont try to push the development workload for the product you are selling to the customer, who allready payed for it. And about karmic implications: You are creating a debt and implicitely forcing the customers to report bugs that they need to have fixed, because otherwise they would feel let down due the trust they have put in Borland. Is it good karma for customers to report more bugs of more quality and sacrifying their own work? There must be balance in everything.

    ReplyDelete
  6. This shows a level of customer disresepct that it ain't funny anymore. Delphi is incredibly buggy. Customers report problems. You should be HAPPY AND THANKFUL to those and treat them with special care, even if the reports aren't perfect. These are your top customers, those that are willing to stick with your product regardless of the bugs. They could as well walk away from your product. And guess what, yelling at them that they should follow some rules that make your life easier is not treating them with care. Boy, I am just glad that Chrome exists and there is an alternative Pascal for .Net around now (oh, and will be interesting whether your are self confident enough to leave a mentioning of Chrome on your page :)

    ReplyDelete
  7. If we are talking about betas, then your requirements seems to be OK.

    But if it is release, then be happy that people do not as money back as it suppose to be!

    It's just a Borland luck that Delphi users is very patient people...

    ReplyDelete
  8. This explains why Delphi quality is getting worse by every version Borland releases. There seems to be no sense of customer service whatsoever. Obviously you expect your customers to do the job for you.

    I have been running Delphi 2005 for almost a year now and I must say it is one of the buggiest softwares I've ever used. I've also tried to report bugs in Quality Central, with all the information you requested, but not even the simplest bug has been fixed or even commented by Borland.

    Get real!

    ReplyDelete
  9. Ulrick --


    I'm having a hard time reconciling your comments with


    http://qc.borland.com/wc/qcmain.aspx?search=1&proj=10&rib=10.0.


    and


    http://qc.borland.com/wc/qcmain.aspx?search=1&proj=270&rib=10.0.


    any suggestions on how I might reconcile them?


    Nick

    ReplyDelete
  10. I noticed that a lot of the closed reports in QC are fixed in version 10.something, does that mean D2006? Those bug fixes should of course be available free of charge to all D2005 customers. It's also sad to see the large amount of bugs that you let through in the first place...

    ReplyDelete
  11. Actually Nick, I think your two links illustrate the point nicely. Let's refine a little shall we?


    http://qc.borland.com/wc/qcmain.aspx?search=1&stat=20&res=10&proj=10&vers=9.0 is a link to a search for all Delphi 2005 for Win32 reports marked as Resolved, Fixed. Count them all - it shouldn't take long.


    http://qc.borland.com/wc/qcmain.aspx?search=1&stat=40&proj=10&vers=9.0


    So by now Delphi 2006 has been announced, and yet Delphi 2005 only has two marked as resolved, fixed, but 685 sitting on Reported.

    Truly commendable.


    The full breakdown is as follows:

    Reported: 685 - 41%

    Opened: 402 - 24%

    Closed: 477 - 29%

    Withdrawn: 38 - 2%

    Resolved: 57 - 3%


    So in all, Borland has not done anything with 41% of the reports - some of which are old, some of which are new. They have only resolved 3% and marked a good 29% as Closed! In fact, remove "Reported" reports from the calculation, and nearly half the reports have been closed! Borland customers must be such idiots to report rubbish half the time!


    I agree that one should strive to be as complete and accurate as possible when submitting a report, but to be fair: I have countless times spent a day or so searching for a difficult to reproduce bug because my customers kept getting plagued by it.


    Borland OTOH seems to think that their customers should spend a day or two hunting their bugs. I get paid to do my own work - I cannot waste hours at a time doing theirs.


    Ironically, I find when using JCL's stack trace that I can very often go to the exact sport where the bug occurred and fix it quickly. However, Borland has not *opened* even one of the reports created with their automatic error reporting. Surprise Surprise!

    ReplyDelete
  12. I would like to suggest that the automatic incident reporting feature in BDS 2006 be made non modal, so that menu items and other details could be checked.


    As it is, it pops up, doesn't allow me to check anything in BDS and then I have no other option but to send it to you.


    Perhaps another option of saving the issue to a file would give me the chance to try to repro those issues?

    ReplyDelete
  13. <rant>

    "because it saved on Q&A"


    Do people actually KNOW what it stands for when they use an acronym? Would this be "Quality & Assurance"? Which one do you do? Quality or Assurance? Clearly they are two different things because someone is saying "Q&A" as in "Questions and Answers".


    Sheesh..Q&A...makes my head explode...


    </rant>

    ReplyDelete
  14. <Rant #2>

    For those that are saying that Allen's post somehow constitutes disrespect. Let's get real and leave the land of Oz for the moment.


    First: Borland has admitted their mistakes with D2005 and pledged significantly higher quality on the next release, and have been publicly discussing what they've been doing to address quality issues. Continually flogging them for D2005 might be cathartic for you, but I would ask you...." to what end "? What is gained anymore by doing this?


    Second: I submitted bugs on VS2005 to Microsoft. They were significant and would prevent me from using the product. My replies from MS's bug team? "Good catch. We'll try to fix it in the next release. Sorry ". We're not talking about bugs submitted just before release, but a couple of MONTHS before release... and MS is basically saying "next time". And guess what, they WILL charge me for that. In an ideal world, we'd NEVER pay for a bug fix, but the reality of it is there's probably not a software package anywhere out there that you haven't paid for an upgrade that, in addition to new features, fixes some existing bugs or annoyances.


    Third: More and more software companies are realizing that the way to go is 'JIT Bug Feedback" e.g. the software itself sends a bug report back to the company when an error occurs. This is a GOOD thing. No matter how much you try, you're going to miss stuff. The thing is, the one thing that the JIT reporting tools cannot do well is give contextual information, steps, etc. If I get a "JIT" bug that comes up with a [send] button, I write down the steps as I best remember them. Trying to reproduce the bug FIRST is a waste of time, as the [send] button will then be gone and it may or may not reproduce easily, and if it doesn't, I've lost the call stack and bug details forever.


    I think at least when you get call stacks and stuff back you can create automation tools to pinpoint trouble areas of code by seeing the frequency of JIT reports that come in with that stack signature.


    Those that think that Allen is lecturing them or talking disrespectfully to them need to wise up (so I guess *I* am assuming the role of being disrespectful now). Allen is just doing the Jerry Maguire here: "Help ME help YOU" -- He is clearly concerned about quality, and he's just trying to grease the wheels of communication between customer and company. He doesn't control the software schedule. He doesn't control the price of the product, etc. He's an engineer talking to other engineers about what would be helpful to him (or you) to fix bugs. If nothing else,these will make it more likely that a bug fix will be reproduced and fixed for a service pack release. (again, something he may not have control over). If you want to rant about getting a bug fix in D2005 that is marked fix for the next version, talk to product management about it. Don't slam the engineer who is just trying to ensure as many bugs as possible get fixed within a certain window of time.


    </Rant #2>

    ReplyDelete
  15. Randy,


    First: This is one of the things that upsets me the most. Borland will leave D2005 the way it is, close to useless, and expect us to give them more money for a version they claim works. "to what end?" you ask, well, I was kind of hoping that we at least could get a patch that actually improved something, unlike the three released so far.


    Second: So you mean that the fact there are worse companies, makes Borlands ignorance OK?


    A piece of advice to Borland personnel, don't lecture your users. A lot of them are like me very frustrated. I've been using Delphi since version 1 and bought every version released so far. D2005 is by far the worst one so far.

    ReplyDelete
  16. After reading Cobus Kruger's search results in Quality Central above, I realized one of the two reports that has actually been fixed in D2005, was reported by me (a duplicate to #9663). So, since Borland fixed at least something in D2005 I take back some of my criticism.

    ReplyDelete
  17. >> First: This is one of the things that upsets me the most. Borland will leave D2005 the way it is, close to useless, and expect us to give them more money for a version they claim works.>>


    Has Borland stated there will be no more service packs for D2005? I wasn't aware of that. I also know that Allen personally did a lot of work to deliver even informal fixes and updates to D2005 users even as things were being done towards D2006


    > "to what end?" you ask, well, I was kind of hoping that we at least could get a patch that actually improved something, unlike the three released so far.>


    Sorry, but that's just crap. Yes, there are still problems with D2005, but MANY things did get addressed in the service packs and in Allen's informal work. The product runs much faster with FastMM and Allen's patches. I use it every day for production work and with each update it got considerably better. You would have much more credibility if you did not resort to grand, sweeping statements which most people would consider demonstrably false as stated. If you wish to make your point, don't overstate it for dramatic effect.


    > Second: So you mean that the fact there are worse companies, makes Borlands ignorance OK?


    ignorance?


    > A piece of advice to Borland personnel, don't lecture your users. A lot of them are like me very frustrated. I've been using Delphi since version 1 and bought every version released so far. D2005 is by far the worst one so far.>


    Again, you're shooting the wrong person here. Allen is trying to make the engineer <-> engineer feedback better, and you're throwing rocks at him for it. My suggestion would be to take your issues with regards to schedules and pricing to someone who actually controls that, starting with product management and going up from there. The only thing you'll accomplish by antagonizing Allen is seeing that he no longer feels free to talk openly to the community for fear that he's going to be held personally responsible for every business decision Borland makes.


    Randy

    ReplyDelete

Please keep your comments related to the post on which you are commenting. No spam, personal attacks, or general nastiness. I will be watching and will delete comments I find irrelevant, offensive and unnecessary.