Chris Pattinson

In the past month I’ve been helping a very large, worldwide distributed project (not RAD for those wondering - something new you’ll see soon) with several systems, focus on both quality and scope to a release date.  The team is passionate; it’s one of the things I love about working in Embarcadero engineering.  

In the last month, we saw an unusual spike in bug regressions. Regressions are where functionality has been tested and proven to work, stops working due to another change in the system. "Large" is quantified as over 10% of bugs found. Ideally regressions are a very small number, indicating a good quality process in development.  A large number of regressions is doubly concerning, since first QA has to find them, and then fixing them slows down the overall progress to release.  To discuss these issues, I held a tiger team session with the development leadership to understand the causes, and discuss what we could do in short and long term.

It was a really productive discussion, so I’ll be sharing some of the findings on my blog.  They seem like "common sense" best practices. Overall the team believes they will reduce regressions, and increase both quality and predictability to the release:

1. Communicate early and often between a new code owner and the previous code owner on new changes.  This was possibly the #1 source of regressions when someone was taking over an area of code and making changes, without the full context of why the code was designed in a certain way, or what other systems were impacted by the code.  Most modern source control management systems have some method to identify the person who last touched a portion of code, and armed with that information - a new change will be communicated to the original owner for review and make sure no hidden "gotchas".
2. Be disciplined. Make sure to run new code through unit tests on a CLEAN system (not the same system developed on), and in our case - run the integration smoketests. Just recently I had an email from a very senior engineer who was frankly amazed how many serious regressions in other systems were introduced by what "should" have been a safe code fix. Far far better for the project to catch these before even checking in to the development branch.
3. For this large, distributed team - establish as strong ownership of code as is practical. Document the areas and owners, both for R&D and QA. Make sure the owner/experts are consulted when working on code that may cross boundaries.  This is already helping, as it encourages point #1, and a unified table helps gives all team members a picture of who to talk to, to understand a system.  It seems basic, however when a new team is spun up and has presence worldwide, it makes a big difference. 
4. Periodic bug ownership review meeting with R&D leads. Here is where developers who after reviewing a bug (or partially fixing one with shared ownership) discuss the more complicated and cross system issues with their peers, and make sure the most appropriate people are looking at the right bugs.  At least weekly, though encouraged informally between owners that bugs cross multiple boundaries on.

These specific practices complement a number of other basic practices which are institutionalized at Embarcadero, such as:

• Daily automated functional tests (these found plenty of the regressions, quickly, which led to the topics above).
• Daily bug councils to triage incoming issues, help filter them to the right people.
• Daily collaboration between R&D and QA to ensure issues reported are well understood in terms of customer impact, frequency and breadth.

It would be interesting to hear other tips/tricks and suggestions from other developers to help improve defect prevention, not just detection. I think this is where teams can truly become agile - when they can self-improve to reduce the number of defects coming into the product.

Share This | Email this page to a friend

Ongoing goals of the QA organization are to:

1. Keep test cycles as quick as possible.
2. Drive towards high levels of test coverage.
3. Provide concise and informative reports on test results. 
4. Provide the most important feedback first.

While many product teams have a mature automated test system in place, there are several opportunities for improvement.  A couple years back, I defined the Embarcadero Quality Maturity Model to help provide a roadmap for improvement in automation. I’d love to see when a QA engineer sits down in the morning, they have in their email box a summary report of any new passes and failures.  Even better if the R&D engineer associated with such a test gets a quick summary report letting them know a test failed, and their QA was notified!

This week I sat down with Nevil Patel, one of the QA Architects at Embarcadero, to firm up the test goals and strategy for 2011. From those discussions, the following model was formed to help analyze the various components of the test system. The model is intended to be simple and high level enough to share with any technical professional:

Embarcadero Automated Test Model

The continuous integration system checks the quality and functionality of every code check-in, providing rapid feedback to the development team for each check-in.  An incremental build is made, unit tests run, and then a quick smoketest run.  

Source that passes this build is a candidate for the daily test, which typically runs overnight. While continuous integration has several advantages, I personally think knowing immediately when the build becomes "unhealthy" and quickly fixing it, keeps the momentum high in an Agile team.

Typically a daily build is made on a clean system every 24 hours.  Building clean ensures the build is 100% reproducible and any "tweaks" are captured properly. Once the build is completed, the daily tests start which executes a chain of broader and deeper tests.

First a smoketest is run, which is typically an hour or less test of basic functionality. This smoketest verifies the build is good for deeper testing. With a worldwide team, some teams are waiting for the results of the smoketest before continuing test development or exploratory testing, so we try to timebox it in a short time frame, yet enough to prove the build is good.

After the smoketest, additional functional tests are chained off. The model shows a very simplistic view - in reality, many test systems are brought up, hundreds in a 24 hour period. Tests are run first on a common platform (such as XP or Windows 7), then spread into additional test levels. The strategy being the most common test cases are run first, so that the team has rapid insight into any serious issues.

As tests are run, the results are sent to a centralized reporting system in real time.  Below are a few screenshots of an internally developed reporting system. The test reporting system centralizes the test results, and uses color to rapidly identify areas experiencing failures.

Image 1: Example of actual reporting system on RAD Studio.

Image 1 shows the results of testing against each daily build. The test results vary due to manual launching of specific suites on additional configurations and platforms, as well as test development in progress. This is normal during development, and tends to climax during the final weeks of a release, where the candidate build has tests run on a large variety of platforms and configurations (with RAD Studio XE, the number was over 500 000). Note that this system does not capture unit test results, which are reported in Hudson, the continuous integration part of the test system.

Image 2 - Detailed view of a specific build

Image 2 shows part of a more detailed view of the test results for one specific build. Note the tree control on the left, and the color coding on the right. Tests are weighed both by total failures in all test cases, and number of test suites that fail, to help give overall grading of a build.

Image 3 shows part of a detailed view of tests run in a specific functional area.

Image 3 was taken from tests in progress, and highlights how one test can be run several times on different platforms, languages, and product versions.  Usually a risk assessment is made by the QA engineer on the functional area to determine what variations of a test should be run to validate the feature works on all the supported platforms/languages.

A build compare tool has been developed by innovative QA engineers to help rapidly compare the test results between builds, and identify and differences in test results. Soon I’d like to use this tool to send an email report summarizing differences between daily runs to QA engineers, to save a lot of time going through test results.

Image 4 - Build Compare Tool Used with DBArtisan

Hopefully this helps give a basic understanding of the automated test system used at Embarcadero. I think this is a fairly valuable model to help a team architect a system to provide rapid, meaningful, feedback to the development team. 

The automated test system has been key to quality improvements related to several Embarcadero products such as RAD Studio and DBArtisan.  It gives great job satisfaction to hear customers say such great things about our products as found in the Embarcadero Quality Survey for RAD Studio XE .

I remember in 2000 when it took several weeks to complete a test pass on Delphi, and it was very difficult to get the "big picture" on what tests were run, and when. Now well over 100 000, closing quickly to 200 000, tests are run nightly on RAD Studio. The team knows what tests have been run, on what build, and ship with confidence in just a few days.  Good stuff!

Share This | Email this page to a friend

At Embarcadero, we take quality of our products VERY seriously, and are always seeking to focus efforts to providing the best tools to help you succeed in your work.  One of our methods to assess how we did is to collect information from our customers in an annual quality focused survey.

Earlier this week I sent out a Quality Survey to several thousand worldwide users of C++Builder XE, Delphi XE and RAD Studio XE.  I will be doing other products, such as ER/Studio and DBArtisan, in the coming weeks.  In just a couple days we’ve seen several hundred responses, and the responses have been very positive - Approximately 80% indicate meets or exceeds expectations, approximately 15% indicated minor issues, and a few percent indicated major issues. 

I designed the survey to ask questions on features, stability, performance and overall quality. Response choices are "Exceeds expectations", "Meets expectations", "Minor issues", or "Major issues".  Statistically, the results are the best I’ve seen in ten years - even Delphi 7.

Here are samples of responses to the question "What do you like about XE?" (NOTE: Edited for grammar and spelling. ):

• "The renewed emphasis on quality we were promised a year or two back really shows. I used to hold off migrating to new releases, waiting to see what nasties they included; since D2009 I have moved over to new releases as soon as possible."
• "It’s really [an] eXellent edition for Embarcadero. I LOVE Delphi. It is pure inspiration."
• "It is very powerful. Embarcadero has put a lot of work into Delphi and it shows. I am impressed."
• "From Delphi 2009 to XE everything is better and performance of Code Editor has dramatically improved. Developing is fun again… :)"
• "Lots of focus on quality and stability. Many quirks of previous versions that occur every single day, if not every single hour, were resolved."
• "That fact that, as a developer, its "got my back" and has for many many years now - with the exception of 64-bit."
• "It’s a very effective tool. RAD development is excellent."
• "I like Delphi XE very much. Its outstanding performance make makes my work pleasant. I can [quickly] achieve what I want to."
• "Only just switched to it as I waited until we were starting a new build phase of our own products, but so far I am very happy. Been a user since Delphi 1, and am happy to say that things are getting much better than they had been. Stability seems far better even than 2010, and speed increases are very, very welcome. Early days for me so far but please tell the team thanks."
• "The whole experience is so much like it should be !"
• "I’ve had every version of Delphi - many have had fantastic improvements but this is best yet."
• "I have been using Delphi since 2006 release and quality has been improving all the time. XE is the best release I have used."
• "Faster and more reliable than previous versions - even than Delphi 2010 which was a major milestone."
• "It is very nice to work with it. Best Delphi yet. Delphi is still the best tool I know."
• "The best improvement is in quality. I can comfortably code all day in Delphi again. And it’s faster on hardware available at time of release than any version I’ve had since Delphi 1. (Today, that’s a quad core beastie). About this survey: I needed another category for "Do not use, but plan to". There’s a lot of new stuff in XE since 2010 that I won’t get to for months (e.g. AQTime, FinalBuilder, etc).The effort put into quality has paid off. Keep it up."
• "Generally fast, stable. A definite improvement over previous versions."

There are many similar comments. The data shows that efforts such as leveraging the Quality Maturity Model and Quality Central initiative are really making a difference.

We also had questions on what users would like to see improved, and asked what Quality Central reports  we should be looking at. Overall what we heard lines up with what we’re already doing, so it was a great validation of our focus for 2011.

One area of concern in the past couple years is documentation quality.  A collaborative plan was decided between documentation and R&D to have R&D focus on API documentation, and the documentation team on examples and demos. The processes for defining and owning documentation work have been improved, and we expect this to help with future documentation work. Another documentation update is being wrapped up in the next few weeks.

During RAD Studio XE development and beta testing, the team went through over seven thousand Quality Central reports. Besides the survey, Quality Central is a great place to record any issues or suggestions for the team to look at.  A question was also included in the survey asking for any specific QC numbers we should look at first. These will go into a list the team will look at and ensure are prioritized based on severity, frequency, risk and effort.

If you are an existing XE customer and did NOT receive the survey, there could be several reasons. First check your spam filters. Also, I avoided sending it to anyone who opted out of email or newsletter communications from Embarcadero. If you would like to participate, feel free to contact me at chris.pattinson@embarcadero.com and request an invitation to the survey. We’d love to hear from you. We literally look at the responses from every single survey.

Regards,

Chris

Share This | Email this page to a friend

Recently I’ve been reading up on posts from various QA boards and newsgroups, and often I see newer Software QA engineers making comments about automated testing and wondering why management is pushing so hard for "more automation".

I’d like to share a few thoughts from my own experience as to what’s really behind such a statement, since understanding the benefits and goals, as well as understanding the challenges, is essential to any automated test initiative.

Big benefits of automation

1) Automation reduces test cycle time.

Reducing the gap between test cycles adds huge value to development, and is a fundamental requirement for any successful Agile project. Consider the difference when QA tests a build one day old compared to a build one month old. When QA finds issues, the amount of effort by R&D to identify the root cause of a regression, and unravel other dependent code changes to the source of a bug, is exponentially larger the longer the problematic code was introduced. If testing finds a bug in source changes made the day before, the fix is trivial and safer.

Having many and short test cycles incrementally checking the health of a build also helps ensure the confidence that the build will be ready on the date communicated to other stakeholders - customers, marketing and sales. The team KNOWS the state of their product, all the time, and this is a really good thing.

2) Sustainable testing over many cycles.

When a new project starts, QA often can rapidly keep up with R&D through manual testing since there are only a few functions and platforms to test. Products can grow very rapidly, and the number of test cases only gets larger and large, and the number of platforms to test and customer configurations grows as the product matures. For simplicity sake let’s say a team has 10 R&D and 5 QA engineers to work on a new product line. QA defines and runs 50 test cases a week. The first 6 weeks this means 300 test cases, after 50 weeks this is 2500 test cases. After three years we’d see almost 8000 test cases. This is not sustainable with the initial resources, and while teams grow as revenue grows, rarely do they grow at this sort of scale. Let’s say a QA engineer can run 200 test cases a week, and then you have 8 weeks to do one complete test cycle of the product. This usually means only the highest priority test cases are run during development, with one or two detailed passes as the project progresses, and the risk is huge that many issues are found late, R&D may not be familiar with such old code, and it can be very expensive to fix safely. If you had automated testing for those test cases, it’s pretty easy to find and revert a code change that caused a problem.

3) Rapid support for new platforms.

When a new operating system or database server comes out, using existing automated tests can give you very quick feedback on the behavior of existing software with the new platform. After a little tweaking to the RAD Studio set of over 100 000 functional tests, they were all run on the new Windows 7 OS in just a few days. For DBArtisan existing tests that exercise database features can be run on a new version and immediately inform the team on any changes in behavior with previous functionality. This type of rapid feedback can really help a team estimate the effort and scope to support a new platform, as well as ensure previous functionality keeps working with new systems.

The big challenges with automation

1) Automated testing takes a sizeable amount of time and effort to get started.

Command line testing is easier then UI testing, however the UI testing is what captures the end user experience best and helps ensure the whole system behaves as expect.

Often a small number of engineers (typically R&D and QA) create the test framework, and to a product owner this can feel like zero gain work. Development of the test framework can take several months to get a basic set of tests going, especially for a new project where the UI keeps changing.

A new test tool for a QA team may take training and some time to learn and become productive with. Here is where it really takes a dedicated effort to build a strong foundation that will make it easy in the long term to code, run, maintain and debug tests. When I was the QA manager with RAD several years ago, I told the team we were going to cut down manual testing to 20% and focus efforts on automation. It was a scary and tough decision, but really paid off. The team now has over 250 000 automated tests and a mature test system. Product quality seems at an all-time high, and the team truly feels Agile as new failures are caught and corrected very quickly.

Some test tools with recording technology can look very appealing - however make sure the recorder creates test that survive simple UI changes or new builds of a product. Otherwise the team will feel they are spending a lot of time maintaining existing tests and frustrated with test framework failures, not real product failures.

RAD Studio team (Steve Trefethen if memory serves) enhanced Zombie with a model generator tool to help sync up the test automation models with UI changes. The team gained a huge increase in productivity as maintenance work that accounted for 20%+ QA time nearly vanished. This also meant the test results were finding real failures so the confidence in the test results was increased. Far fewer false failures to distract the team with!

2) Keeping the push on automation while other project pressures come in.

It’s easy to stop and revert to manual testing behaviors, especially when a release date looms and the team feels they benefit the most from a manual test pass. What is important to consider is how much is a week or two of manual testing REALLY going to buy you, compared to the momentum from another week or two of automated testing that can be mostly re-used for every future release, and help reduce the cost of fixing? It is very important to work with the project core team and set goals and expectations of test coverage, and how much investment in automation. Other managers may not really understand the long term benefits of test automation, and when there is even a little struggle or setback, may declare it as a reason to stop. I think it’s wise to treat a new automation initiative as a new feature in the product - define the requirements, estimate the work, design the architecture, have it reviewed by the technical leads, prototype to gain a small success and prove the strategy works, then expand from a solid base.

Anyone else had experience with a new automated testing initiative? Or as a developer, what are your thoughts on automating the testing of features that you work on?

Share This | Email this page to a friend

Tis the season to don my chef hat and bake up a storm.  At this time of year I’m usually doing at least 4 types of cookies, and typically a pumpkin cheesecake for the family. This morning while driving in, I started to think how the art of baking cookies relates to software development. Strange, but true

Here are the thoughts:

1) The business requirement is the basic type and amount of cookie. What cookie type appeals to the most? How many people are getting the cookie? Functional requirements are what tastes and ingredients help increase appetite and appeal? Do you need to consider any allergies to nuts or other ingredients? Or any considerations about amount of sugar or salt?

2) The design is the recipe. Is it a design that is simple and easy, or complicated with lots of ingredients and steps? Does the design call for complicated or unusual steps or equipment?

3) The development is mixing and executing on the design. You can still get creative here and substitute certain ingredients (switch shortening for butter, or butter for margarine, or change types of flour, or add more or less salt or sugar), while taking a risk doing so. Experience pays off here, just like in software development, and technically you can do prototyping especially with cookies - make a small or large cookie first and measure the bake time to get it just how you want it.

4) Testing is after the baking, and involves checking for firmness, consistency, taste, size, texture and appearance. Does it taste and look great? Does it stay together when transported?

5) Deployment is taking it to your consumer. Friends, family or work-mates in many cases. You may want to wrap it up in a nice tin, dish or cookie plate. You may want to mix several cookies together. Or perhaps add some garnishing both for taste and presentation. (Note - this might be ‘tested’ again by my wife who can tell me if it looks good or has more suggestions to improve. )

6) Post project review - get the feedback from those eating your cookies and determine how you want to do the next batch, next time. Maybe they were a little too sweet, or too small, or if you did several types you found one or two were voraciously consumed.

So yes, there are a lot of similarities to think about when comparing software development to cookies, and what to consider for a really good batch of cookies. Now I’ll be thinking of some delicious chocolate chip oatmeal, snicker doodle or gingerbread cookies when working with the team during the ‘bake time’ of a release.

And as it turns out, I just brought in a batch of snicker doodles for the team to enjoy today.

Happy holidays!

Share This | Email this page to a friend

Recently I’ve been joining QA and Test related groups found on LinkedIn to try and extend my view of "what’s going on" in the rest of the industry.  One is simply called "Software Testing and QA" and has had some really interested content in the past month.

One of them is this post:

• http://blog.softwaretestingclub.com/2010/12/a-tester-is-for-life-not-just-for-christmas-2/

The blog post provides an EBook with a number of testers answering questions about their experiences in the QA profession such as favorite books and authors, biggest accomplishments and challenges, views on certification and why they love software testing.  It is a good read, done for charity, and at the minimum contains some great reading references for anyone interested to learn more about software testing. Enjoy!

Share This | Email this page to a friend

Every so often, I am asked how Embarcadero leverages the many popular standards such as ISO, IEEE, CMMI and Six Sigma. These are really great resources, and I’d like to spend a few minutes introducing them, and talking about how we apply the concepts and principles at Embarcadero.

IEEE promotes advancement in worldwide technology, ISO provides over 18 000 international standards used in a variety of industries, and Six Sigma and CMMI are frameworks for continuous improvement.

QA teams such as RAD Studio based several of their QA related processes and documents from IEEE standards such as :

• IEEE Std 1058-1998 Software Project Management Plans
• IEEE Std 1074-1997 Developing Software Life Cycle Processes
• IEEE Std 730-1998  Software Quality Assurance Plans
• IEEE 829-1998 Software Test Documentation
• IEEE Std 830-1998 Recommmended Practice for Software Requirements Specifications
• IEEE Std 1219-1998 Software Maintenance

ISO 9001 - 2008 describes the quality management process for a project.  It is a great reference for planning, executing, reporting and improving quality processes.

ISO 9004 - 2009 describes the quality management process for an organization. It is a great reference to build and develop a potent quality organization.

CMMI and Six Sigma are continuous improvement programs. CMMI focuses on levels of processes in terms of definition, repeatability and self-improvements. Six Sigma focuses on customers, processes and reducing the source of failures. I think both add huge value over time by reducing ‘waste’ incurred in software development and through optimizing processes, really accelerate productivity.

The Embarcadero Quality Maturity model leverages from all of these, to help create a framework to measure how mature a project is in defining, executing, monitoring and continually improving their quality management practices.  A video presenting this model can be found here - http://channel-e.embarcadero.com/index.php?option=com_jvideodirect&x=1&v=uwjfk5qm4WkM6

Here are a few links I’ve used in the past to help provide an overview of the standards:

• IEEE 829 - http://www.coleyconsulting.co.uk/IEEE829.htm
• ISO 9001 - 2008 :http://www.praxiom.com/iso-9001.htm
• ISO 9004 - 2009: http://www.praxiom.com/iso-9004.htm
• CMMI - http://www.whittingtonassociates.com/v2/resources/articles/sei_cmm.shtml
• CMMI - http://www.cmmifaq.info/
• Six Sigma - http://www.tech-faq.com/six-sigma.html

Each of these standards and processes takes a significant investment in time and effort. The payoff includes increased customer satisfaction, reduced waste, and confidence in release schedules.

 At Embarcadero we have a dedicated QA organization. The entire management team participates in our quality planning, execution, monitoring and improvement into the project teams. Our quality assurance managers are key members of project core teams, tasked with gathering and analyzing data to determine how the product is shaping up to customer expectations. As the project requirements are being defined they are reviewed and ‘tested’ by the team, as a poor requirement has a huge cost downstream into the project.  Each R&D engineer knows they have a QA engineer ‘partner’ to help review the conformance of a feature with the use cases defined in the requirements, and QA engineers work with beta testers and customers to help understand any issues encountered in the field.

Quality is about a lot more than just testing. Quality is layered in from the beginning in requirements definition and project planning, all the way to deployment.  Once an issue is identified, then identifying the source of problem and preventing serious issues from every happening again is a major benefit of a mature quality system.

As you go through these standards you’ll notice a big focus on continuous improvement, self-assessment, and introspection. Quality is so important to Embarcadero that engineering principle #2 is all about understanding the customers, and engineering principle #6 is all about continuous improvement and introspection.

To me "good quality" ultimately means satisfied customers, satisfied employees, and continually improving skills and capacity.

Share This | Email this page to a friend

Possibly one of the less known "really cool" features in Delphi XE and RAD XE are the built in audits and metrics allowing a developer to do code analysis.

There are at least couple great sources describing this -

1.  RAD Studio XE and Code Reviews: Moving from Subjective to Objective in Seconds!

   [Doc Link]
   

2. And a video by Mike Rozlog (44 minutes - worth it!)

   [Video Link] 

As a quality champion, I really love how this helps a project team create and maintain code for very large projects. One of the advantages of the built-in audits in metrics in RAD is how you can enable/disable the metrics most important to you, and it makes it super easy to hide any false positives or to focus on the code analysis most important to your team.

Delphi XE comes with over 200 audits and 80 static code analysis metrics (10 of each in Professional. This makes the Enterprise and Architect versions appealing).

I suggest checking it out - static code analysis tools complement code reviews and put quantative analysis in to an otherwise qualitative process.

Share This | Email this page to a friend

Michael Froh, one of the R&D managers at Embarcadero suggested a fairly new book called "Making Software" a couple weeks ago. He knows I’m quite interested in continuous improvement, and even more so - studies and data on what other companies have found worked for them.

Like component re-use, learning from the work of others and modifying to what works for your teams and projects is a useful method to moving forward.  Not all ideas work for every company, however in my experience learning from others successes (and mistakes) is always a good thing.

"Making Software" is really a collection of 30 software development stories from some familiar faces.  For example - Steve McConnel of Code Complete (another favorite from years ago) and Barry Boehm who sort of the father of software development practices and standards. The book includes studies on optimizing architectural time based on project size, how effective is unit testing , and strategies to detecting and then preventing sources of defects (possibly the topic I value the most).

A link to the book can be found here - http://oreilly.com/catalog/9780596808303 .

I downloaded the electronic version, and found the following chapters very interesting:

• Chapter 3 : What We Can Learn from Systematic Reviews
• Chapter 8 : Beyond Lines of Code : Do We Need More Complexity Metrics?
• Chapter 9: An Automated Fault Prediction System
• Chapter 10: Architecting: How Much and When?
• Chapter 12: How Effective is Test-Driven Development?
• Chapter 18: Modern Code Review
• Chapter 25: Where Do Most Software Flaws Come From?
• Chapter 27: Mining Your Own Evidence

Several of the others are still very interesting, though I tend to favor analysis and metrics with sufficient detail explaining how the data was gathered, and any deviations.  Chapter 12 was a bit weak, more or less suggesting "try test driven development and keep using it if it works for you".

Some interesting conclusions raised included:

• Typically 20% of the changed code causes about 80% of defects. (Identify that 20% and focus on it to optimize quality efforts…)
• For projects with over a million lines of code, about 30% of project time invested in requirements and design gives the most optimal return. (At a Software Test and Performance Conference in 2007, weak requirements management was cited as #1 reason projects fail.)
• Specific suggestions how to mine subversion data to help identify areas of highest change traffic (attributed to highest risk areas of code where defects are most likely)

Overall, I thought it was a good read and suggest to anyone working in software engineering. Thanks Michael for the suggestion.

Share This | Email this page to a friend

Looking through our testing reference materials, I realized it’s hard to get a picture of all the activities that go into the release certification process.  I put together the following diagram showing primary categories of testing, to help in the high level planning for a release.

(Click on the image to zoom in to a better quality view)

The tasks flow from the left to right - first the build must pass unit tests or typically the build is ‘broken’. Next an automated smoketest is run to validate integration of the build worked and the build is at least testable before QA spends time on a test pass. Automated functional tests are then run, or manual if it’s a new feature still having the test cases fleshed out and developed.

Automated functional testing is leveraged to run on various product SKU’s (Trial, Architect, Enterprise, Profressional for example) and various operating systems (XP, Vista, Windows 7, etc…).  Test also run on the localized product and include international enabling tests.  In parallel, automated bug regression tests validate bugs once fixed are still fixed.  Everything to this point is part of the regression test that verifies functionality that used to work still works.

As a point of reference, RAD XE had over 140 000 automated tests run in one evening, which gives the team good insight into the fundamental status of the build.  A few days later the release build had a total of 280 000 tests run against it to validate the various locale builds, and SKU’s.

User acceptance testing is next, with a big focus on beta testing. Passing marks such as "85% or better ready to ship" and "80% or better recommend to a friend" are usually set.  The QA team also tests various user scenarios around new features, general usability and polish, the functionality of 3rd party apps (though typically we contract our 3rd parties for this testing - we still do smoke level testing just in case). Exploratory testing is run to find any holes in the test system. Exploratory testing includes things the typical user won’t do to help expose any error messages or stability issues in the product. (A fun part of the testing work?)

Install testing is in parallel with user acceptance testing. The product is installed with other releases, uninstalled and verified certain settings are maintained, and checks licensing works such as upgrading trials to full versions of products.

For a release build, and several times during a project cycle, the team will conduct performance tests. These tests include stability (checking for memory leaks or A/V’s from repeated operations), scalability (large file processing, many users at once, many applications at once), and time and resources to complete common user interactions.

As you can see, it’s a lot of activity and the challenges the QA engineers, managers and leads face is how to balance their energy and efforts to cover all these requirements. Leveraging automation to ‘write once, run many’ is definitely a major asset for Embarcadero QA teams.

I found this diagram helpful when reviewing high level test plans, and making sure all the bases are covered. I’ll be next looking into breaking down each of the major areas of testing into the next level - functional testing for example covers a broad set of test types.

Share This | Email this page to a friend