Looking Past “The Project”
With Open-Source Tools
Carl Nagle
SAS Institute, Inc.
Project Manager: SAFSDEV
1.0 Introduction
Some readers may know the SAFSDEV project was not always called “SAFSDEV”. The original effort, beginning way back in 1998 using Rational Software, was later affectionately known as “The Data-Driven Engine”; or more problematically, “The DDE”.
The SAFSDEV project, now on SourceForge, contains shared, public implementations of keyword-driven automation architectures like those advocated for many years. As is rampantly commonplace among our industry conventions, this type of automation methodology has been blessed with several different names: Data-Driven, Keyword-Driven, Action-Based, etc. Even our own documentation has used each of these different terms at different times.
As I mentioned, the first “SAFS engine” was implemented for what is now IBM Rational Robot. That is still considered to be our most full-featured framework, and the reference implementation. A complimentary engine supporting IBM Rational XDE Tester has also been developed and released. Additional work is currently underway to support standalone use of the XDE Tester engine, as well as future engines based on open-source technologies like Abbot, JFCUnit, or others we deem desirable and doable.
An engine has also been developed for Mercury Interactive WinRunner. It was originally translated by John Crunk, but John is now stepping into a new position using the Rational engine. A team in India, sponsored by Novell, has stepped forward to take up the WinRunner effort. This is a testament to one of the benefits provided by open-source solutions. The WinRunner engine will grow and remain publicly available through the SAFSDEV project.
Since 1999, I have often been asked the How? and Why? concerning the existence of the SAFSDEV project. I hope to answer some of those curious and critical questions here.
In addition, I want to make the reader think beyond “what is needed right now”; and to recognize that “what is needed right now” is often indicative of issues we will continue to encounter in future endeavors. Solve these issues once using broad-based solutions, and we solve them for all future encounters.
2.0 Looking beyond “The Project”
I am still amazed how often I see or hear an organization espouse “reuse” for the code used to develop shipping product; yet the test organization is allowed total freedom to do whatever they want—“reuse” is often not even in the plan vocabulary.
Is your automated test effort really unique?
As an often-encountered example, large corporations set out evaluating the various commercial automation tools available. This is a very valid and important step in any planned automation effort.
The company will bring in eager technical sales staff from the various vendors, watch demonstrations, and even perform some fairly thorough internal evaluation of each candidate tool. Eventually, the company will choose a vendor, and place an order for product licensing, maintenance contracts, and even onsite training. Unfortunately, from that point forward, the plan often degenerates terribly!
The tools and training get distributed throughout the company into various test departments—each working separately on their own projects. None of these test projects appear to have anything in common. The applications appear to be different. They solve different business problems, or satisfy different customer needs. And each project may have its own, individual schedule of deadlines to meet.
Do you really think your problems are novel?
Yet, nearly every one of these test departments begins to separately code functionally identical common libraries. They each make routines for setting up the test environment. They each make routines for accessing the operating system interface or API. They each make nearly identical file-handling routines, string utilities, database access routines—the list of code duplication is disheartening!
Now, it may be that some of us have avoided this type of isolation. Perhaps we are the only department working on automation in this company. Perhaps we have tried to collaborate across departments to build some of these common libraries and share them.
But what about other automation issues? Are we collaborating and sharing those, too?
For example, when we have difficulty working with a particular component or issue, who solves that problem? How is the problem resolution shared across departments? Often we find that when each separate project hits a problem—the same problem—each works independently to resolve it.
We are all trying to solve the same problems!
Then there is another, more subtle form of duplication--the myriad script fragments often used to capture and reuse small pieces of client functionality. This is most prevalent in the GUI testing environment.
These script fragments surface as a large number of innocuous scripts or function libraries that hold all the different things you might ever do with the tested application. There will be a small “reusable” script fragment coded to login to the application. Another one captured to enter some piece of client data. Then there are tens to hundreds more for other parts of the application. While these are steps in the right direction--away from fully recorded user scenarios--they still contain a vast amount of code duplication.
As mentioned, a complex application can end up with several hundred of these “highly reusable” script fragments. A large portion of these scripts contains identical code--the manipulation of various application components like text fields, buttons, lists, trees, and tables. Each must be developed, debugged, and made robust.
The problem? These scripts have codified an application-specific solution to working with these common components—they will only work on this one application. Yet, the only thing unique about much of this duplicated code is the information that says, “which component, using what data”! And, as should be evident, when you move on to the next project or application, you must do this all over again. Yikes!
Do you really want to do this all over again?
Generally, a finite set of common components makes up our infinite set of unique applications. We need to recognize that, and use it to our advantage. We can develop, debug, and make a robust set of libraries to deal with these common components. These libraries, however, are not application-specific. They will work on many other applications. We simply have to pass these new application-independent libraries the application-specific information they need: “which component, using what data”.
3.0 Looking beyond “The Enterprise”
“Go Make ReUsable Frameworks!”
That has been the battle cry of so many influential voices in public outlets and publications. Year after year the books, articles, seminars, and conferences reinforced our own experience that keyword-driven frameworks were viable solutions to many of the automation problems we all faced. And, for all those years it has been ours to do. It has been ours to design, ours to develop, ours to fund. Now, just go out and do it.
And, good luck!
Way back in 1999, I had conversations with several of those influential voices. One of them made a very good point: the team working on a 3 to 6 month release schedule will not have time to spend developing a keyword-driven architecture. This person also mentioned projects with only 3 weeks of test development time! Such a project could not even afford the time to consider such things!
I could only agree, initially. But I countered,
“What if the framework was already in-place?”
There was thoughtful silence. An in-place framework was not an option at the time. It was hypothetical.
At SAS, we had our own similar resource issues back in 1998. We all know that resources will always be an issue. Time and resources in automation are two of the primary obstacles that must be overcome. Yet, our own past experience with isolated automation frameworks--including several resource-starved failures--showed us we needed a shared, reusable architecture; and we needed it badly.
So, dealing with the resource issue, one manager with vision fought and won approval for one position. She knew it was not really enough, but it was one more than zero. The goal was to move forward on long-term plans for a keyword-driven framework. Having sufficient automation experience, I won the job; but together we knew we needed more help.
Rather than trying to do this isolated inside SAS, we decided to share the framework with others wanting to go down the same path. And, in doing so, we hoped to gain external development resources we knew we needed. Besides, if we fully intended to exploit code provided by others, we most certainly should return the favor.
Fortunately, many outside SAS had been hearing the same call to keyword-driven frameworks. And, just as it makes little sense to duplicate individual functions and libraries within the enterprise; it makes little sense for each individual enterprise to expend effort building similar, non-commercial frameworks. More importantly, sharing the effort and expense allows us to move forward. The alternative is often that we get nowhere at all.
Our decision to share the framework “beyond the enterprise” has since provided much more value than we could have even imagined back then.
4.0 Open-Source: More Than Meets the Eye
One of the most wonderful observations made by nearly every code junky is that there is a great deal of free code waltzing around the planet. Some of it is good. Some of it is outstanding--better than any sold commercially. Some of it is gibberish typed by monkeys. Still, all the various flavors, both good and bad, are free.
I am not going to get into distinctions between “free” and “open-source”. For our purposes, we are going to use the definitions where they meet. The code in question is free, and it is open-source. The compiled, executable tools are free. The code is freely modifiable. The user is free to examine every line of code involved leaving nothing open to guesswork. For better or for worse, the code is open to public scrutiny. You generally cannot get that from commercial solutions.
I am also not going into the true cost of free software. I have provided one or more links at the end for your reading enjoyment. So let us move on.
There is an enormous amount of this stuff out there. And if something is close, but does not exactly fit the need, we at least have the code foundation on which to build our own personal version of whatever it is we do need.
To some, that is all the open-source phenomena represents. And that is so shortsighted!
In the past we have seen messages from people having the strongest desire to leverage our open-source tools. Yet, management forbids these people to do so! The argument from management, invariably, is that the open-source offering has too many uncertainties. They argue that the best bet for more certain success is to strictly use in-house resources developing in-house solutions.
My own experience, and the experience I have heard from others, suggests these managers have stacked the odds against them. I have heard the stories where such internal projects struggled or flourished for a few months, and then they died away. Sometimes, the project might have succeeded in building a good framework, but few internal customers would choose to use it. Over time, with few customers, such projects are abandoned.
In other scenarios, the framework architect—the one or few that really knew the nuts and bolts of the implementation—they had moved on to other projects, or other companies. With internal expertise on the framework gone, and there being no other source of support on such a proprietary venture, the projects fall into disarray, and are often abandoned.
I can even attest to this with my own experience at SAS on the SAFSDEV project itself.
The only thing that kept the SAFSDEV project alive beyond the first 2 years was the growing user base outside of SAS. Like many companies; our individual testers, departments, and divisions were free to embrace or ignore this project. In the beginning, there were less than a handful of internal testers even having the time or inclination to evaluate what we were offering! There were more people using the framework outside SAS than we had using it internally. However, that external user base was growing and adding new features and functions all the time. I became the glue that kept the project focused, and on-track; growing the feature set for when the larger SAS community was ready to embrace it.
Then we encountered what should have been the ultimate framework killer. Following a corporate-wide restructuring, I found myself “under new management”. For a year I had to abandon my own full-time support of the framework! I was assigned to more high-profile performance testing. Yet, because the project was open-source—because there was an active community of users—the SAFSDEV project continued to grow even though I could only support it part-time. Others outside the SAS community provided the continuity that I could not.
Happily for me, that time has passed. Another manager with vision has stepped forward to fund my full-time support. The usage inside SAS is now growing tremendously, and we have dedicated personnel for complimentary tools and training that leverage and build upon using the SAFSDEV framework. And still, other external companies and individuals have stepped forward as needed to provide critical development resources. Consequently, we have been moving forward with new features and future plans much faster than anticipated!
As this should demonstrate, open-source initiatives can offer much more than just free tools and sourcecode that can be customized. Good open-source offerings generally have a global following. There is an active community of users and developers often eager to help us succeed with the project.
The keen observer will recognize that there are several huge benefits a good open-source project allows us to tap into:
- External knowledge base
- External technical support
- Viable means of continuity
- External Development Resources
5.0SAFSDEV, STAF, GNU, and You
SAFSDEV, STAF, and the GNU projects represent just the tip of this mammoth iceberg. The point I want to make is that we do not have to resign ourselves to starting from scratch, rolling our own, or even “doing without”.
If we can realize that our problems are not unique, then there is almost certainly something out in the open-source community that can be leveraged. When we do find something, we have the bonus opportunity to participate. We can even help define the paths that some of these projects take! And if we do not find a compatible solution, we do not need to be frustrated. We can start building and sharing a solution with other interested parties.
In 1998 SAS wanted a keyword-driven testing framework. We wanted to enable non-technical testers to write automated tests without learning the sordid details of the underlying automation tool. And we wanted to be able to migrate these same tests to different automation tools on different platforms as needed.
We now have many of these things in place. And with the help of an active open-source community, we are well on our way to all of them.
We started with a framework entirely exclusive to IBM Rational Robot. Still, there were enough interested external parties that this grew. John Crunk later came along and translated the Robot implementation into a Mercury Interactive WinRunner implementation.
Various other individuals have come along seeking similar engines for QARun, TestPartner, SilkTest, and others. Often times, these people go off in seclusion and work on these translations all by themselves. I find that unfortunate, but I have no control over their decisions, or the decisions of their managers.
For example, through private contacts, I know there is a Compuware TestPartner engine out there. But it is stuck behind a corporate firewall somewhere. Consequently, that company is stuck solely supporting and maintaining it. That is unfortunate for them, and it is unfortunate for other TestPartner users that have already demonstrated the desire to help enhance and maintain it.
In 2003, SAFSDEV launched an endeavor to more fully develop tool-independent infrastructure. The STAF project on SourceForge provides outstanding middleware glue that is multi-platform, and multi-language. With it, we have succeeded in our goal of providing independent SAFS framework tools that allow different SAFS engines and clients to share data, share resources, and coordinate their testing effort. In conjunction with this, we developed two new SAFS engines: (1) for IBM Rational XDE Tester, and (2) an independent (free/open-source) Java-based engine.