From: Mark Gilbert (mg_abimail_at_yahoo.com)
Date: Thu Apr 01 2004 - 20:22:14 EST
An open (and long) letter to whoever does end up working within the
TaskForce. Some specific response and more generally useful information
from your friendly neighbourhood QA lead.
On Thu, 2004-04-01 at 17:37, Jan Christiansen wrote:
> What we need to do is, in simple words, to do a cleanup.
...
> 2) The orginal reporters do not always verify fixed bugs. If we can be really sure that a bug is fixed, we'll do the verify and get rid of those bugs. I've planned to do a bulk verify on dupes/wonfix/invalid (did jeg forget some?) before we start, to be sure, that all that those unecessary bugs are gone.
Please consult me before doing such bulk action. It's been done before
but only very carefully, using precisely worded comments and carefully
designed bug screens.
Additionally you'll need to take responsibility as qa contact for those
bugs and (this part is just a request) cc me.
Also note that it often isnt as tasking to actually verify such bugs as
it is for fixed bugs.
>
> 3) Now there should only be open bugs left. We should start with the unconfirmed ones. Try to reproduce them, then try even harder, and if we can't, then ask the reporter. I've scheduled a 3 week cleanup period, if we don't get in touch with the reporters within that period, we probably won't get an answer.
>
For 2 and 3 _please_ take a look at the FaqBugPolicy pages of the wiki.
There is important information and reasoning there. If you want to
change some parameters, I'm open for discussion, but any change needs to
be agreed upon and used consistently by all qa folk (lead and helpers of
all activity levels).
Also, it might not be mentioned in there, feel free to make use of the
needinfo keyword, as well as if needed _very_ short whiteboard codes.
> 4) Now there should be a lot of open confirmed bugs left. We'll make sure, that they still are valid. Now and then developers have forgotten to mark it as fixed, or it has been fixed together with another bug (side-effects). If a bug is still valid (which it usually is.....) then we may be able to enter some hints, traces or whatever might help later when someone has to fix it. Make sure, that it isn't a dupe!
>
Also make even more sure that you dont misidentify a bug as a dupe.
While we always try to cut down on clutter by seeking out dups, it would
be even worse to let bugs slip through the cracks by flagging legitimate
independent reports as dup.
> 5) We have to look at every bug regardless of OS. A Linux bug, might also be a Windows, in that case the OS should be ALL. We should also have a look at then severity (enhancement, trivial,.......). Change it to whatever suits (have a look at the "priority" url, we'll set the priority by the book, so to say) - but don't make blockers, make them criticals.
Yes, I have been trying for a while to (re)triage the bugs ranging from
the normal or major severity to the blocker severity to be more accurate
and meaningful (whether they need upgrading or downgrading in severity)
but it is hundreds of bugs and my available time diminishes each day.
Having helpers to take care of this for me, esp. upgrading the normal
and major bugs when appropriate would be awesome so I could focus on the
worst bugs and seeing about release relations.
Priorities are a great concept but this project does not use them so
much (for a number of reasons, I can explain for the curious). If that
can change, great. Feel free to prioritize bugs in addition to
modulating severity but be aware that I and the other core devs will be
paying most attention to severity until prioritization is consistent
across the board (which will be starting with you all).
> As an Abi core member, Mark Gilbert will have a look at all critical bugs and decide which to raise (and block the next release).
Yes, this will shift to be my top priority with respect to the various
QA tasks that I could be doing if I get enough help. The reason being
that it is the one which can be done by the least number of people.
I will be modulating and notating with respect to micro releases
(2.1.x), time frames, and well the upcoming minor release. The upcoming
minor (technical term denoting the second digit in the version, it is
actually a milestone) is 2.2, scheduled tentatively for late june at the
earliest due to the nauseating number of bugs and newly-introduced
instabilities, as well as the fact that many devs such as myself will
have more time to cram before the release then as opposed to earlier in
the spring.
I'll be upgrading (raising in severity) as well as downgrading (lowering
in severity). I'll basically equate severity with priority (priority
not necessarily meaning the actual bug field, which is relatively low
res, but the actual life priority) such that the most severe bugs are
the highest priority for release. I know the use of priority is a
little ambiguous, let me know if you dont quite understand.
NB: Additionally, as we get very close to 2.2, there is the possibility
of my causing divergences in severity in priority, but only if
absolutely necessary. I'm going to avoid this as much as possible, but
if it comes june and abi is truly stable and all but a couple
release-critical bugs are fixed and the remaining dont show any sign of
getting fixed, the other so-called cores and I may have to make a hard
decision to postpone bugs. But that contingency is a long way off,
let's think positive.
>
> We should always use the latest cvs-head build (either build it yourself or get it from pchasm) as it includes most features/bugs. But if you have time, you should also use the latest stable build (there might be an important difference...). You can find more good QA related stuff here, http://www.abisource.com/beta/
>
Righto. Now be very careful here: if a bug is filed against 2.0, we
cant close it just because it isnt in HEAD, since 2.0 is the production
release series. But you may want to consider 'Not2-2' in the whiteboard
or some other means of notation in addition to commenting to signify
that a bug does not exist in HEAD (or at least so we think).
Obviously we'll stop supporting 2.0-only bugs not too long after 2.2 is
out the door, not because we want to but because we simply have
remarkably too few developer resources to continue fixing bugs that have
already been fixed. Not an ideal situation but something we have to
live with. That said, we will not stop supporting them before then, and
bugs are still being fixed in 2.0 (in many cases before they are fixed
in HEAD).
>
> If you don't already know, a new mailinglist has been created, abiword-qa_at_abisource.com. The QA discussion list for the hardworking QA people. We plan to start using it, when we start the WTF, but it'll stay there afterwards for future QA work.
Yep. Dr. MG gives lecture series on qa tactics from time to time (-:
His methods of staying afloat in a sea of bugs with a raft of only a few
developers are world renowned (-: (yes I'm also often facetious)
>
> Schedule:
> Week 14: (this week) Contact all top candidates. CASE-001 starts (explained later).
> Week 15: Get more help, if we still need someone. Finish CASE-001. Primary assigned users, feel free to start
Note, 001 will take longer. I'm sorry. It'd be doable if this was my
full time job (or yours), but that just isn't the case.
> Week 16: Now we're all playing the cleanup game.
> Week 17: Still playing
> Week 18: (end of April). Let us be proud and show the rest of the world some results.
It isnt going to be quite so clear cut IMO. First, it will take
longer. Second, there is no final static result set, new bugs are filed
every day, old ones change, priorities modulate as releases approach,
etc. We'll need all the QA force we can muster to stick around as much
as possible (not to consume all your time but what little of it you can
spare). Third, there is life after 2.2 (-:
I know, I'm a spoilsport )=
But the results of the TaskForces are very important, and will be taken
seriously.
> Please have a look at the below described cases and then make a list (mail it to me) of the cases that you like (the most wanted first, if you can handle more than one, please tell me). If someone else has taken your favourite, I'll have a look at your choice no 2 and so forth. When selecting cases, please consider and make sure, that it's a subject that you can handle. Some final advice: A case with a small bugcount isn't always easier than the opposite.
>
Well, be realistic with respect to the amount of time you'll have to
spend. The components are grouped primarily for the purpose of division
to conquer, the actual components themselves are less relevant.
Important things:
1. it doesnt need to be one to a case.
2. these are a lot more work than you expect, especially when we get
to the bug localization phases (like comparing test cases, providing
backtraces, testing in different versions, etc)
3. Therefore try not to take on more than is reasonable. Frankly, 24
unique bugs is a lot of time spent triaging and localizing and
retesting and so forth. So dont take cases 7 and 11 just because you
figure they're the smallest it should be no problem. If anything that
just means those are the only two where it might be acceptable to have
only one to a case. If you do get through an entire case, localize and
retriage everything etc., then while mg is busy kissing your feet you
can certainly pick another to tackle (I don't think we'll run out of
bugs any time soon).
4. Communicate with your case partner(s) to work consistently and
avoid duplicating efforts.
5. Communicate your tactics to the qa list so that we can work more
quickly, efficiently, and effectively, and mg's job can be a lot
easier. This means things like letting us know what codes you use in
the whiteboard, what habits you form for testing things, whether or not
you make yourself qa-contact of the bugs you take charge of, etc. The
more we know about how you work and the more consistent we are
throughout the QA department, the more effective this taskforcing and
workload distribution will be (and most importantly, the least likely
it will be that bugs slip through the cracks)
> CASE-001: Initial cleanup of misplaced bugs
[cases follow]
Hmms. There are approximately 203 bugs unaccounted for on this list. I
guess that's mostly the second case? (of course there are a number of
new bugs since that query was conducted, but probably not 203)
>
> Any questions? ....ask me! We should agree 100% on what we are doing and how we do it. That's the way to be efficient and finish the job within the schedule. I hope, that you reached this last section without to much confusion, but you are probably a bit tired of reading :-)
>
Probably more tired of me (-: But absolutely correct, communication and
consistency amongst qa folk is key and will make this a lot better in
the aforementioned ways.
That's why the qa list was created, for qa tactic discussion.
Any and all users are highly welcomed. If this is as much a success as
we hope it is:
1) AbiWord 2.2 and beyond will be far more stable and robust than anyone
expected (and sooner)
2) the Windows port will be in much better shape than it ever has been
3) As if that wasn't enough, we may, erm, find additional ways to reward
the top QAers for their work. But of course we know that AbiWord is
what really matters or you wouldn't be on the list of bugzilla devotees.
Thank you all for your contribution!
Talk with you soon
-MG
This archive was generated by hypermail 2.1.4 : Thu Apr 01 2004 - 20:22:29 EST