Chris Pattinson

In my experience, determing how much testing time you need in a release is one of the toughest questions in software development planning.

How much time do you need to test a yet to be developed product, with nebulous features yet to exist, and also ensure your previous functionality still works at least as well as intended?

Balancing this, is there is usually a time sensitivity component to a release so the earlier to market, the better the product will do - unless it suffers from quality problems. So the team needs to determine the right balance of feature development and testing period to succeed.

There are many good sites that discuss this (for example - http://www.etestinghub.com/why_is_testing_necessary.php ), and several of the Embarcadero teams have comprehensive test estimation processes. However, the further out you make your estimates, the more likely it is off- and roadmaps generally ask for a team commitment to high level feature themes. Remember 64 bit? Cross platform? Those are very large features that need a significant amount of "R" in research and development to estimate properly.

Here is a simplified description of the approach QA uses:

• Evaluate previous project testing efforts in duration, coverage and results
• Review find and fix rates during the project - I’m looking for a spike after feature complete that eventually tapers off on severity as the project is closer to release. If it doesn’t taper off, then previous testing was too short. If it tapers off early, then we had some extra bake time and could have cut back.
• Measure bugs found during release and bugs found after release, and in what areas of the product
• Objectively assess where testing gaps may have existed
• Measure the complexity and risk of new features, and how much of the product they impact
• Review any outstanding bug backlog
• Determine any special quality goals from product management (IE - we’re going to bulletproof XYZ this release. What do we need to do that?)
• Determine schedule of an ‘end game’ phase based on the above including a beta test
• Track carefully testing progress as the project progresses via feature completion rate, test development and execution rate, bug find and fix rates (including how severe and where they are found) and beta tester feedback.
• Conduct a quality review regularly - we have a project dashboard full of quality and project related metrics to review how we’re doing every week.

Embarcadero leverages a modified Agile method for it’s releases - traditional Agile has all the testing in the one sprint, however I find that usually not enough - performance testing is a major effort, usability testing is a major effort, and the final regression test to ensure the entire product is in good shape takes more time then a single sprint can accomplish.

Hence I call it a ‘modified Agile’ process, which seems to work well - R&D wraps up features weeks to months prior to release (based on complexity of the product and changes), and then the team goes into polish mode - working with beta testers to improve usability, QA to run performance, stress, usability, install and acceptance tests, and verify bug fixes. Then once the product is locked down, a last testing pass is conducted to ensure the team knows exactly what they are shipping.

Seems like a lot of effort, doesn’t it? It really is.  It’s great when it all comes together, and the team KNOWS it’s shipping a great release.

Share This | Email this page to a friend

There is a great public review of RAD Studio 2010 on Dr Dobb’s.  After a release, I’m always looking forward to seeing if the community notices the feature and quality improvements made by the team - it looks like the team really scored based on the following review comments:

• http://www.ddj.com/windows/220001058

Also worth checking out this series of reviews on Digg.com walking through the product by a long time RAD Studio user:

• http://digg.com/programming/RAD_Studio_2010_Review_Part_1_The_Top_10

Also been keeping an eye on the newsgroups. So far a lot of positive buzz, it’s worth browsing around at :

• https://forums.embarcadero.com/forum.jspa?forumID=67

And the various technical forums for Delphi, C++Builder and Delphi Prism found in various forums: 

• https://forums.embarcadero.com/index.jspa?categoryID=1

I think we can all say with confidence that RAD Studio found a great home with Embarcadero. 

Share This | Email this page to a friend

One of the activities I’ve always loved with previous Borland and now Embarcadero is where development team members present and meet the development community.

We’ve been successfully using the web to host a conference which has high accessibility to ‘everyone’ around the world, and the fourth such conference is coming up soon.

You can read about it here:

• http://conferences.embarcadero.com/coderage
• http://conferences.embarcadero.com/coderage/sessions
• http://conferences.embarcadero.com/coderage/speakers

I’ll be doing a session talking about quality assurance processes including continuous improvement, reporting, Jira, beta testing, and test automation development during development cycles on Thursday, September 10th from 2pm to 2:45pm PDT. I’ll take questions after a general presentation and would be interested to know what you want to hear about - I can do more detailed sessions in the future.

There are a lot of sessions, so check them out and see which ones interest you.

Share This | Email this page to a friend

RAD Studio 2010, which incldes Delphi 2010, C++Builder 2010 and Delphi Prism , is on it’s way to general release soon.

http://www.embarcadero.com/rad-studio-2010/

This years release is packed with a lot of great features, and a focus on usability quality improvements. In addition, the team has moved the documentation to a new wiki format to help with faster updates, and allow the documentation team to rapidly react to questions and requests for content from the community. 

There is documentation in the previous H2 format also included in the product, of course.

From the quality side, I’ve been looking at what the team accomplished. We had a much larger beta program, leveraging the CenterCode Connect Beta software to help manage and run the beta (see http://www.centercode.com/ ) and the best volume and quality of feedback in the past ten years.

The metrics for this year come out to:

• 781 public bug reports Fixed (those from Quality Central) fixed since January 1, 2009 for Delphi
• 198 public bug reports Fixed (those from Quality Central) fixed since January 1, 2009 for C++Builder
• 2573 total Delphi bugs fixed since 1/1/09 including internal and plublic reported issues
• 730 total C++Builder bugs fixes since 1/1/09 including internal and public reported issues

Note: ‘Delphi’ is used for general IDE fixes which also apply to C++Builder

• Beta Quality survey responses showed 81.8% would recommend RAD Studio 2010 to a friend, 6.1% would not, and 12.1% were undecided.

Note: Updated Aug 24, 5pm - I’ve asked the beta participants to submit a final survey so will update as the results come in.

The team also worked hard on adding and addressing as many of the ‘little usability enhancements’ as they could work in, as well as reviewing the QC feature requests to hear what the community has been asking about.

The QA team has done a pretty good job keeping up with the Reported issues for recent releases. Considering QC was designed as a public self-service system, I’m very pleased with how beneficial it’s been to have QA review QC entries more directly. We’re going to look at dealing with more of the QC backlog in 2010 and ‘cleaning it up’ - a pretty significant task, but one worth doing IMO.

So RAD Studio 2010 achieved the quality goals we planned earlier in the year and has something for everyone. I hope you guys all really love it!

Regards,

Chris

Share This | Email this page to a friend

In the past couple months, I’ve been working with a team of 7 quality assurance managers to evaluate and determine a roadmap of QA improvements for their product teams. As part of this effort, I wanted to establish a series of guidelines to measure ‘where are we now?’ and to know ‘where to go next?’.

Being fairly familiar with CMM, ISO and IEEE standards of development, I looked to apply what I consider the most valuable in the the Agile methodologies Embarcadero uses. The result of this effort is a documented Quality Maturity Model. This model is broken into 8 categories:

1. Documentation Maturity
2. Project Management Maturity
3. Reporting Maturity
4. Defect Managemetn Maturity
5. Test Coverage Maturity
6. Test Automation Maturity
7. Beta Maturity
8. Training Maturity

Each of these categories has 5 levels. Everyone likes to ‘level up, right? Well, 6 if you count level 0 which means no part of that category has been defined or documented. Each level requires that the previous level of requirements are satisfied. So Level 3 would require that Level 1 and Level 2 are completed.

For example, the guidelines for Reporting maturity are the following:

Reporting Maturity

It’s a nice model, since it can leverage any other methodology and compare product teams in a large, complex organization.

And it worked really well - as a result of this model, the Quality Assurance Managers (QAM’s) set quarterly goals for Q2 2009, achieved over 80% of their goals, and are working towards an annual roadmap with a clear vision of what to do next.

Rather handy, especially for newer managers and to help get organizational support and visibility into ‘What the heck is it that QA teams do?’.

I’ll be speaking about this model in a session at CodeRage 4, you can find the details here:

• http://conferences.codegear.com/coderage

At CodeRage 4 I’ll use RAD Studio as an example of how this model was applied, and how it helps the teams stay focused on quality processes to ensure a quality release. 

Cheers,

Chris

Share This | Email this page to a friend

It’s worth mentioning that Embarcadero has a central Beta application page where you can apply to a number of beta’s of products in the works:

• http://www.embarcadero.com/products/beta_programs.php

Come take a look at some of the products under development (under NDA of course).

Share This | Email this page to a friend

David I. had an interesting question for the team - are we fixing enough of our older bugs in the product? Are we neglecting past issues reported by our customers?

As a resident quality champion, of course I want to know too

So I took a quick query into the bug tracking system, and looked at how many reports found earlier then Jan 1, 2009 were fixed this year. A quick, no fuss query though I made sure to only include bug reports (not feature requests) and resolutions related to bug fixing.

The result?

711 Delphi bugs found earlier then Jan 1, 2009 have been fixed so far this year, and the team is in middle of polish phase for the next release "Weaver".

The next question is how many came from customer reports? So I looked at the number coming from Quality Central - 222 of those 711 are Quality Central reports.  Not an insignificant number. A number (about half) of those were released in RAD Studio 2009 Updates 3 and 4 with the remainder going into Weaver. As I mentioned - more is being done as well.

Additional slices:

• 198 of the 711 were reported earlier then 2008
• 116 reported prior to 2007
• 91 reported prior to 2006

That’s good news, and a sign the team looks at older issues as well as new.

With 10 years of reports, there is a mixed bag of about 2000  public reports the team is constantly working on, and our beta feedback helps guide the release to ensure it’s a good one.

As always, we can improve- and need to to stay ahead of the pack.  So let me know if you have a QC report you feel is neglected. I’ll take a look at it.

Share This | Email this page to a friend

I’ve now had almost a decade in the Quality Assurance field of software development, and while all folks understand the importance of testing, the understanding and measuring of the real value of a quality assurance organization can be difficult to determine.

From my personal view, a well run quality organization achieves three majors goals:

• Visibility
• Control
• Reduces Development Debt incurred by Bugs

By visibility, the more mature the testing organization is, the faster and more effectively problems are found and the impact of problems are understood.

When a bug is reported, a mature organization can analyze the frequency and impact of such an issue, then once it’s analyzed create an effective automated test to ensure that once the issue is fixed, it stays fixed.

Visibility is more then just detecting the bug as well - the requirement for a feature may be ambiguous or incomplete leading to inefficiencies in development. And then other processes such as code reviews, project planning, and release processes could all have areas that if improved, not only ensure a product is shipped of good quality but that the process to do so is lean, mean and very effective - less wast of one of the most precious resources : Time.

Control is achieved by clearly defined milestones and acceptance criteria to validate that milestone. It’s easy to say you have your first Beta build with no qualifications, however if you state the beta build needs to have certain functionality and bug thresholds, then you can ensure the team aims for those goals and can keep a project ‘health’ in good shape.  You can also ensure that when you release a product to market, you clearly understand it’s state and can more confidently predict revenue.  Also, public beta’s advertise how your organization cares about a product, and customers can feel part of the development process - as well as help other customers learn and use the product.

QA in itself does not fix a product - however QA ENABLES the R&D team to fix a product and better meet a customer’s needs. And with highly visible test systems that run frequently, this adds huge value to R&D - the sooner a bug is detected the easier it is to fix. 

Now, the first two factors are hard to measure. Development Debt can be measured on the time it takes for developers to work on bug fixing during a product cycle. This work can be measured against customer satisfaction to understand the development cost of bugs to achieve certain satisfaction levels. A well run development process with mature quality organization can be truly agile and spend more time on features, and less on bug fixing since the cost of fixing a bug found quickly after introduced is hugely less then found later.

Take the simple scenario - an engineer modifies a feature and a manual tester looks at it, and certifies it’s ok. However a bug now occurs in another feature that depended on a property that was modified and this other feature wasn’t tested. Worst case, the customer finds the issue weeks or months later after many other code changes - the time to investigate, and the dependencies of OTHER code on this initial change would typically cause a lot of work and headches for both R&D and QA, ultimately making it hard to fix the issue without causing OTHER side-effects.

Now, if automated testing, or even a manual test that takes the depencies into account, finds the problem the next day - the fix is trivial.

Ultimately, R&D ‘feels’ less work was done, however if the issue was caught and fixed, the Development Debt is less for the next release, meaning more exciting features . It’s clear how this is a huge benefit of a well run QA organization.

This article generally supports these thoughts: http://searchsoftwarequality.techtarget.com/news/column/0,294698,sid92_gci1318196,00.html#

There are other advantages too - once you have visibility you can find opportunities for new features and understand what the customer is really asking for, not what you are assuming they are asking for. You may find that you thought customers always use your product in one way, or one feature that is really important to them. You may also find areas in the development process that if you improve, dramatically improve overall team productivity.

Share This | Email this page to a friend

I’ve been asked to take on a new role in Embarcadero as the Director of Quality Assurance. What a great opportunity to apply what was done for RAD Studio in the past few years for the other Embarcadero products!

In the past couple weeks, I travelled around several worldwide sites to meet the development teams, product managers, and senior management with a passion for their products. I’ve had the pleasure to take a deep dive into some really great products such as ER/Studio (wow, wish I had use that for a DB Architecture course I took a couple years ago *grin*!), Change Manager and DBArtisan.  Given the history that I started about 9 years ago as an International QA Engineer focused on Database testing, this is so right up my alley! And I’m loving it.

So now I’m setting up a Quality Leadership team, and helping to spread some of the great tools we’ve been using with RAD Studio products to help out the other teams with their test development and run, test reporting, and beta management. The existing Embarcadero products are already pretty good - I strongly recommend folks take a peek at ER/Studio if you do any sort of complex database design. I felt like I was 21 again delving into all the fun ways to define and then push live database architectural changes, and there is powerful built in scripting.

So say hello to the new Embarcadero Quality Champion - now for 23 products *big grin*!

Share This | Email this page to a friend

I’d like to give some props to Pawel for great work finding and summarizing excellent references for the latest Delphi 2009 and C++Buider 2009 products, including good starting places to understand the unicode changes.

http://blogs.codegear.com/pawelglowacki/2008/11/03/38527

He’s provided sections for Overview, New IDE features, Unicode support, E/R Studio information, New language features, changes in RTL/VCL., DataSnap and a whole lot more. Seems like ‘Must Read!’ to me for the best Delphi and C++Builder release so far

Share This | Email this page to a friend