It’s worth mentioning that Embarcadero has a central Beta application page where you can apply to a number of beta’s of products in the works:
Come take a look at some of the products under development (under NDA of course).
David I. had an interesting question for the team - are we fixing enough of our older bugs in the product? Are we neglecting past issues reported by our customers?
As a resident quality champion, of course I want to know too 
So I took a quick query into the bug tracking system, and looked at how many reports found earlier then Jan 1, 2009 were fixed this year. A quick, no fuss query though I made sure to only include bug reports (not feature requests) and resolutions related to bug fixing.
The result?
711 Delphi bugs found earlier then Jan 1, 2009 have been fixed so far this year, and the team is in middle of polish phase for the next release "Weaver".
The next question is how many came from customer reports? So I looked at the number coming from Quality Central - 222 of those 711 are Quality Central reports. Not an insignificant number. A number (about half) of those were released in RAD Studio 2009 Updates 3 and 4 with the remainder going into Weaver. As I mentioned - more is being done as well.
Additional slices:
That’s good news, and a sign the team looks at older issues as well as new.
With 10 years of reports, there is a mixed bag of about 2000 public reports the team is constantly working on, and our beta feedback helps guide the release to ensure it’s a good one.
As always, we can improve- and need to to stay ahead of the pack. So let me know if you have a QC report you feel is neglected. I’ll take a look at it.
I’ve now had almost a decade in the Quality Assurance field of software development, and while all folks understand the importance of testing, the understanding and measuring of the real value of a quality assurance organization can be difficult to determine.
From my personal view, a well run quality organization achieves three majors goals:
By visibility, the more mature the testing organization is, the faster and more effectively problems are found and the impact of problems are understood.
When a bug is reported, a mature organization can analyze the frequency and impact of such an issue, then once it’s analyzed create an effective automated test to ensure that once the issue is fixed, it stays fixed.
Visibility is more then just detecting the bug as well - the requirement for a feature may be ambiguous or incomplete leading to inefficiencies in development. And then other processes such as code reviews, project planning, and release processes could all have areas that if improved, not only ensure a product is shipped of good quality but that the process to do so is lean, mean and very effective - less wast of one of the most precious resources : Time.
Control is achieved by clearly defined milestones and acceptance criteria to validate that milestone. It’s easy to say you have your first Beta build with no qualifications, however if you state the beta build needs to have certain functionality and bug thresholds, then you can ensure the team aims for those goals and can keep a project ‘health’ in good shape. You can also ensure that when you release a product to market, you clearly understand it’s state and can more confidently predict revenue. Also, public beta’s advertise how your organization cares about a product, and customers can feel part of the development process - as well as help other customers learn and use the product.
QA in itself does not fix a product - however QA ENABLES the R&D team to fix a product and better meet a customer’s needs. And with highly visible test systems that run frequently, this adds huge value to R&D - the sooner a bug is detected the easier it is to fix.
Now, the first two factors are hard to measure. Development Debt can be measured on the time it takes for developers to work on bug fixing during a product cycle. This work can be measured against customer satisfaction to understand the development cost of bugs to achieve certain satisfaction levels. A well run development process with mature quality organization can be truly agile and spend more time on features, and less on bug fixing since the cost of fixing a bug found quickly after introduced is hugely less then found later.
Take the simple scenario - an engineer modifies a feature and a manual tester looks at it, and certifies it’s ok. However a bug now occurs in another feature that depended on a property that was modified and this other feature wasn’t tested. Worst case, the customer finds the issue weeks or months later after many other code changes - the time to investigate, and the dependencies of OTHER code on this initial change would typically cause a lot of work and headches for both R&D and QA, ultimately making it hard to fix the issue without causing OTHER side-effects.
Now, if automated testing, or even a manual test that takes the depencies into account, finds the problem the next day - the fix is trivial.
Ultimately, R&D ‘feels’ less work was done, however if the issue was caught and fixed, the Development Debt is less for the next release, meaning more exciting features . It’s clear how this is a huge benefit of a well run QA organization.
This article generally supports these thoughts: http://searchsoftwarequality.techtarget.com/news/column/0,294698,sid92_gci1318196,00.html#
There are other advantages too - once you have visibility you can find opportunities for new features and understand what the customer is really asking for, not what you are assuming they are asking for. You may find that you thought customers always use your product in one way, or one feature that is really important to them. You may also find areas in the development process that if you improve, dramatically improve overall team productivity.
I’ve been asked to take on a new role in Embarcadero as the Director of Quality Assurance. What a great opportunity to apply what was done for RAD Studio in the past few years for the other Embarcadero products!
In the past couple weeks, I travelled around several worldwide sites to meet the development teams, product managers, and senior management with a passion for their products. I’ve had the pleasure to take a deep dive into some really great products such as ER/Studio (wow, wish I had use that for a DB Architecture course I took a couple years ago *grin*!), Change Manager and DBArtisan. Given the history that I started about 9 years ago as an International QA Engineer focused on Database testing, this is so right up my alley! And I’m loving it.
So now I’m setting up a Quality Leadership team, and helping to spread some of the great tools we’ve been using with RAD Studio products to help out the other teams with their test development and run, test reporting, and beta management. The existing Embarcadero products are already pretty good - I strongly recommend folks take a peek at ER/Studio if you do any sort of complex database design. I felt like I was 21 again delving into all the fun ways to define and then push live database architectural changes, and there is powerful built in scripting.
So say hello to the new Embarcadero Quality Champion - now for 23 products *big grin*!
I’d like to give some props to Pawel for great work finding and summarizing excellent references for the latest Delphi 2009 and C++Buider 2009 products, including good starting places to understand the unicode changes.
http://blogs.codegear.com/pawelglowacki/2008/11/03/38527
He’s provided sections for Overview, New IDE features, Unicode support, E/R Studio information, New language features, changes in RTL/VCL., DataSnap and a whole lot more. Seems like ‘Must Read!’ to me for the best Delphi and C++Builder release so far
Hi all,
We’ve started the Delphi Prism Beta, which is testing of our new Delphi.NET solution. It’s pretty cool, I’m not supposed to say more until Nick officially announces the details.
However if you are interested in testing out fully functional beta builds, go here to apply:
https://beta.embarcadero.com/callout/default.html?callid=6E2829BA76664D57ABD8F26046781EA5
We’ll be adding folks periodically from those that apply. Anyone who has an interest in using Delphi syntax in Windows .NET applications should check it out and give us some feedback on what they think. And Delphi users who want to see if they can develop easily with .NET should also take a look.
Cheers,
Chris
I’ve been quiet on my blog for about a year, that was a result of work to transition the team from Borland to CodeGear then CodeGear to Embarcadero, work on improving our beta tests, and championing product quality. I kept telling myself I should be blogging more, which I’m sure is also true
The team did several ‘cool things’ for quality this year. We went through the ENTIRE bug backlog, reprioritized and retested literally thousands of reports, and identified the highest severity reports we wanted to fix for the Delphi 2009 and C++Builder 2009 release.
Then the team settled in to a focused period of feature development and bug fixing, and the biggest feature work revolved around the latest Unicode feature, Generics, the new Ribbon Controls and DataSnap. There were a few dozen other features put into the mix as well and we struck a generally good balance of features and fixing.
What really helped the quality this release was full management support. Of course revenue is a pressure, but our new management team as a longer term outlook - they wanted a release for more then just a quarter. So when our field test responses were not looking as good as Rad Studio 2007, we extended the schedule for several more weeks of focused bug fixing - and I think the results paid off.
Can we do more? Heck yes. And we’ll work on regular updates to both product and documentation, and keep an eye on what you folks are using most and the most serious issues to fix first.
We’re also totally revamping our beta test, I’ll blog more about that in a week or two since we’ll have our next Delphi .NET product to give this new system (CenterCode Connect) a spin.
Back on topic - I’ve posted fix lists of older Quality Central issues fixed as part of ‘Tiburon’ - the Delphi 2009 and C++ Builder 2009 project. You can find these articles here:
Delphi 2009 - http://dn.codegear.com/article/38714
C++ Builder 2009 - http://dn.codegear.com/article/38715
Over 4000 fixes were made, though of course some of those were regressions introduced during development, but it’s still a darn solid number. And the bug find rate was much lower in the final weeks - the usual sign during software development testing is completed and you are ready to ship. We had a good understanding of what we were shipping this time, and already a list of more bugs to fix to take ‘great’ to ‘Wow! Amazing!’.
I’m going to be honest - Documentation isn’t up a level of great quality yet. Delphi 7 help was awesome, and really set a good bar. We’re building back from some business decisions in the past that really cut the quality out from the Documentation since D8, and it’s been a hard climb.
That being said, Documentation/Help *is* definitely better then RAD Studio 2007 with over two hundred fixes between Delphi and C++, especially in terms of speed and number of examples, yet not the great reference experience we want it to be. The team will keep working away on regular updates, and are looking to increase the number of staff to be able to do more in less time. The R&D team, Nick, Alisdair and even QA such as David Dean have been stepping in to provide webinars and videos to help and I highly suggest a browse of development network in the product communities to see what I’m talking about.
For RadStudio we revamped our test automation tools, and we really leveraged those this year - about 20 000 test cases run against every build, with nice graphical reporting broken down by functional area let us ensure we constantly moved forward with product quality. Now we plan to extend those tests to run against our localized product to ensure our international customers get high quality releases with proper functionality. The international QA teams work really hard and do a good job - but it’s hard to cover that much of a product in manual testing every day!
Overall, quality in the product is easier to track, and test coverage is increasing. I highly recommend tools from Automated QA for profiling and determing testing code coverage. I’m sure Mark Edington would agree that AQTime was directly responsible for identifying some really nasty performance issues and crash issues in the IDE, leading to D2009 being far more stable then previous releases.
Hope you enjoy the results of this years work, and feel free to send me an email directly at cpattinson@codegear.com if you run into any bug that you think is bad and we should be looking at ASAP. Of course, I’d appreciate a Quality Central report and good steps
Regards,
Chris
Since we’ve just released the December 2007 update for RAD Studio 2007, Delphi 2007 and C++Builder 2007, this seems like an appropriate time to reflect on this years work. (see http://dn.codegear.com/article/37350 for details on the update).
First a few statistics: Since January 1, 2007 a total of 7006 defects have been verified by QA as ‘Fixed’ or ‘Retest’ between Delphi, Delphi.NET and C++Builder. This excludes duplicate reports, test case errors, or ’as designed’ reports. That’s a whole lot of product loving, in my opinion. 1196 of these reports are from Quality Central, which personally I’m glad to see the community feedback being used to help prioritize bug fixing efforts.
The team also released three major products, each having as much testing and fix time as main development - Delphi 2007, C++ Builder 2007 and RAD Studio 2007. Each release provided additional fixes to the previous, though it certainly made for a complex install environment.So there is where we are planning to focus next. Install needs work, and though that may not directly help current customers, we want a reliable, efficient and simple installer to get our products up and running on your systems in as efficient a manner as possible. Stay tuned for more information, since I expect to put a call out for field testers in Q1 2008 for helping ‘beat up’ new version(s) of the installer to see how much we can improve it in terms of performance and reliability.
2008 will be an interesting year - Unicode is on the roadmap, and work has already started. We’ll also be looking for users with complex projects to join the field test to exercise their projects against the next release. Odds are we’ll be simpler with the install (no multiple small releases - a pity in one way since we got tons of core fixes that way, but on the other hand made for very complex upgrades/patches!), yet nothing is finalized yet. I’d like to thank the CodeGear development team for putting a really big focus on quality this year. We put considerably more ‘bake time’ into the past few releases then in the previous years, and I strongly believe this helped both the products and the developers who use them. And I’d really like to thank the field testers and field test marshal’s who were vocal in a very constructive manner, and effectively used quality central to report issues, and help prioritize them.
We made some improvements already to Quality Central (such as increasing maximum number of votes considerably, and allowing users to vote on hundreds of reports!) and much more on the way. We’re working to centralize our bug reporting to Quality Central instead of using RAID (kudos to RAID to serving us so well for so long) mainly to avoid the team having to look in two places for information, and two systems makes it too easy to let things slip between the cracks. Internally, I have quite a few plans, and the QA team is already working very hard on them - running automated tests on localized builds (already run against German a few months ago, now need them automated daily), a quality dashboard with test results, test coverage and performance metrics, and integrating more real world projects into our test system. Reminds me, I’d like to thank several of our long term customers for submitting their business source code demonstrating performance and stability problems, leading to significant fixes in performance and nailing some nasty memory leaks.
Let’s do more of that in 2008, and make it even a better year.
Quality Central is one of the best methods to formally communicate with our customers.
One interesting statistic - 10472 reports have been pushed into RAID (internal bug tracking system) from QC since inception, and ‘Closed’. Closed can mean several things, so taking a slice of ‘Fixed’ and ‘Retest’ ones I see 5335. That seems pretty good.
On the other hand, we still have several thousand open and reported issues. And many of these have 0 votes, and not all product areas have QA or Sysops reviewing them on a periodic basis (since we’re all human, and folks tend to focus on the areas they are best at.)
Internally, we’re still continuing the plan to move our internal bug system into QC, so that the internal engineers can work with the one system and make the use of time much more efficient - especially when researching for similar or duplicated issues, or cleaning up ‘old’ areas that haven’t had much attention, or have been gathering reports on deprecated technology.
Of the Open issues in Quality Central, we’re looking to clean up as much as possible during the next year. This will take time, on average it takes 5 minutes of a bug council’s time to review one report. 3077 reports are ‘Open’ from Quality Central under the Delphi project (this includes internal field test reports). 2347 are ‘unique’ reports with the others being duplicates, and 511 of those are Feature Requests.
The numbers may seem big, but in the context that over 10 000 have been addressed and 2358 of those since 1/1/2006 - it seems doable.
So then it comes to priorities- we want to tackle the best ‘bang for the buck’ issues. I’ve proposed in borland.public.bdn.qualitycentral some changes to maximum voting and weighing, to encourage more Sysop participation. We still have a large backlog of Reported, though we’re are finally making slow progress knocking that down.
Some interesting ideas and points have been raised. The goal is to make it clearer to the development team which issues are being run into by several people. I see a focus too much on the voting of one individual, which is fine if weighing is given slightly to an expert, but I personally believe it is critical to determine the exposure of a problem, to help prioritize it. Now, that also means QC itself has to be friendly and usable by the community - plus have value and a feeling that time spent voting and putting in detailed reports is well worth the effort.
So for those visiting today, leave a comment on what you like/dislike/want/need/hate/love about Quality Central and what CodeGear behaviors you’d like to see demonstrated.
I’ll start with some examples:
*Feedback on reports from CodeGear in a timely manner
*Action on serious reports in form of patches or hotfixes in a timely manner
*Action on top voted feature requests
*Dislike - seperation of internal and external bug tracking systems
Every release, I get those release jitters when putting out that gold image to manufacturing. RAD Studio 2007 has been out for about a month and a half now, and looking in Quality Central and at various forums - overall feedback has been positive.
A few folks have run into issues with the installer - some history to this is we originally scoped out the technology for a major suite release in June, and late 2006 we decided to react to public survey feedback and release the highest demand personality first as Delphi 2007 instead.
That was cool, Delphi 2007 was released in February and feedback was quite strong. We aimed for backwards compatibility as much as possible and included the launch of a new installer.
Later in May we released C++Builder 2007, which was also provided major upgrade to Delphi 2007 users. Here was where we identified and fixed several issues in the Delphi 2007 installers. This process of iterative releases was new, the QA team was learning all the installation scenarios and one challenge was reacting to the fact what we shipped in Delphi 2007 impacted the upgrade experience for C++ Builder 2007.
What was a major positive was we had another general round of bug fixes for Delphi 2007 users.
So then came along the completion of the series - RAD Studio 2007, which added Delphi.NET, Blackfish SQL and ASP.NET 2.0. AND - another round of bug fixes. It has ‘everything’ including Delphi 2007 and C++Builder 2007.
In 2007 we’ve had over 6 500 bug fixes between the releases (over 1000 from QC), which is a new record for the product. We made progress with the help system, restored lost documentation and fixing many F1 links in the IDE. The core binaries are in the best shape they have ever been, which explains why folks have called RADStudio 2007 (or Delphi 2007 Update 3) the ‘Delphi has come back!’ release.
Now, I’m still a critical kind of guy who believes being realistic is important. We’re not done yet. To me this is just a good start - there is a lot more to do on the Doc side (some examples are back, not all), and the installer is dealing with a very complex set of scenarios. That relates to our product complexity and some internal processes that could be improved to make things less painful/difficult all around.
It’s easy to ‘blame’ InstallAware - but seriously, it’s difficult for any installer technology to deal with all the complexity we are putting on it. Some of the issues are on our side. Others are on the InstallAware side and they have been very helpful.
I’ve been spending dozens of hours talking and helping various customers with install issues. It’s less then when I did work on Kylix - now THAT was complex :) I think it’s confusing to customers as to how to apply an update, and when things go wrong - what to do.
The worst thing to do when the Update 3 fails, is grab your original D2007/C++Builder 2007 DVDs. That appears to be where a few folks run into trouble. The best thing to do there is download the ZIP file from http://cc.codegear.com/item/25014 into a clean directory, then install via the setup.exe including in the .zip. As soon as you have those files, no longer use the original DVD/version 1 install files. I’ve helped point several folks to this method, and it appears to address the main remaining ‘confusion’. (And hey - we’re taking responsibility for making this confusing and looking to fix it)
A couple other common issues are ones detailed in this blog. Odds are if you are reading this post - you’ve read those. Support is great at dealing with any install problems, and are being great about letting me know of any new issues.
Internal QA and the field tests did not run into the problems of ‘wrong install procedure’ since they were educated to the required process. I consider that a lesson for the future - do an open beta test for future releases and catch usability/first experience issues before going live.
Regardless, I invite anyone who is still running into any install issues to send me an email (cpattinson@codegear.com). I also keep an eye on the public newsgroup borland.public.install.delphi .
Consider this an open thread for quality discussions - we DO take quality extremely seriously. If you have a defect or issue, log it in Quality Central , then feel free to talk about it here. We have great feedback in quality central, and are always reviewing and prioritizing for future work. This includes feature requests and fixes (hey - you noticed the top voted request, Unicode, is on our roadmap, right?)