I have always been uncomfortable with the notion of "discount usability engineering." I am a strong believer in the old adage that "you get what you pay for", and I believe this lesson needs to be learned yet again by the Web development industry. As a consultant, I have in fact adapted my methods to fit very tight Web development timeframes, but I have always known - and tried to explain to my clients - that there is a cost to using less reliable, less rigorous methods, and that that cost is increased risk of product failure.
Initially, Web sites were functionally very simple compared to most traditional software applications, and so the fact that they typically took 8-12 weeks to develop, as compared to months or even years for traditional software applications, made some sense. Now, however, Web sites and applications have gotten more and more complex, and in many cases are much like traditional applications that happen to be implemented on a browser platform. The industry needs to adapt its notion of reasonable, feasible and effective timeframes (and budgets) for developing complex Web-based applications, which simply are not the same as simple content-only Web sites, and this includes adapting its notion of what kind of UE methods should be invested in.
A recent case study from my consulting practice will serve to illustrate the importance of rigorous UE methods during Web development.
A contract development team was building a web site for a client organization. The web site was to include up-to-date drug information and was intended to be used by physicians as a substitute for the standard desk references they currently use to look up such drug information as side effects, interactions, appropriate uses, and data from clinical trials. The business model for the site was an advertising model: physicians would visit the site regularly because more current information would be available, and that information would be more readily findable on the site, relative to published desk references. Pharmaceutical companies would buy advertising for their drug products on the site because the visitors to the site (physicians) represent their target market. Regular and increasing traffic due to repeat visitors, and new visitors joining based on word-of-mouth amongst physicians, would drive up the value of advertising, generating a profit for the client.
The development team I worked with generated a prototype design that the client would use to pursue venture capital to support the full-blown development and initial launch and maintenance of the site. The client paid for this prototype development. As the Usability Engineer on this project, I had to use some "discount usability engineering" methods in order to fit the timeframe of the project, which was only 8 weeks. I was able to do some very quick-and-dirty task analyses using "discount" methods, but there was no time for me to incorporate the data into the initial design, which was done without my input. I focused on designing and running a formal usability test (NOT a "discount usability engineering" technique, but a more traditional and rigorous technique), figuring that even though the initial design was not driven by requirements generated from the task analysis data, and the requirements data I was able to gather was sparse, the rough data that I did have would at least help me interpret the results of the testing and make reasonably effective redesign recommendations.
Eight physicians were paid to fly in to the development center and participate in the test as test users. Several basic search tasks were designed for the physicians to perform. They were pointed to the prototype's Home Page, and left on their own to try to successfully find the drug information that was requested in the first test task.
Within 45 seconds of starting their first search task, seven out of the eight physicians in effect gave up, and announced, unsolicited, that the site was unusable and if it were a real site, they would abandon it at that point and never return.
Clearly, if the site had launched as it was prior to this test, not only would an optimal ROI not have been realized, but in fact, the site would have failed all together and a complete loss of the clients' investment -not to mention any venture capital - would have resulted. If 7/8ths of all visitors never returned, enough traffic would not have been generated to have motivated advertisers to buy advertising. The entire investment would have been lost.
Instead, the test users were asked to continue with the entire test protocol, and the data generated revealed insights into the problem that was a show-stopper on the first task, as well as other problems revealed in other test tasks. The site was redesigned to eliminate the identified problems.
In this case study, clearly the usability test - NOT a discount usability engineering method - was worth the investment. In addition it can be argued that if the client had been willing to invest (with both time and money) in more rigorous task analysis and design methods PRIOR to the usability test (rather than the "discount usability engineering" task analysis and design techniques that were used), the problems found in the usability test could probably have been avoided in the first place, much of the redesign and re-implementation effort could have been saved, and more usability problems could have been uncovered and solved prior to launch.
While this is but one case study, I have found over and over again in my consulting practice that rigorous UE techniques during development pay off by saving time and money down the road.
In his recent Taskz column called "Is a high priced usability 'Guru' a good investment?", Charles Mauro describes high-priced "spot consulting" by so-called usability "Gurus" which "generally give clients the impression that, with a day or two of consulting a month, "usability' could somehow be 'injected' into web development teams." He points out that such "Gurus", who may charge as much as "tens of thousands of dollars a day", are "unlikely to have suitable understanding of your customer profile and their critical cognitive structures . . ." and also that "actual solutions come from the creation of well-balanced and strategically minded development groups that understand and buy into UCD" (user centered design.)
In effect, when you hire a "Guru" for "spot consulting", you are buying discount usability engineering methods at what are certainly not discounted - but in fact inflated - prices.
The very fact that such Gurus are commanding - and getting - such high consulting fees is actually heartening to me, because it suggests that there are high level executives out there who realize the value of usability and are willing to invest large sums of money in it. However, for "tens of thousands of dollars", you should - and could - be getting much more than "spot consulting" for "a day or two . . . a month." You should be getting an intense program of usability engineering methods that will reliably increase product usability, decrease product risk, transfer usability engineering skills to your internal staff, and pay for itself in bottom line benefits.
In a December 2000 report by Forrester Research, Inc., called "Scenario Design" (their term for usability engineering), it is pointed out that:
"Executives Must Buy Into Realistic Development Time Lines and Budgets
The mad Internet rush of the late 1990's produced the slipshod experiences that we see today. As firms move forward, they must shed their misplaced fascination with first-mover advantage in favor of lasting strategies that lean on quality of experience.
· Even single-channel initiatives will take eight to 12 months. The time required to conduct field research, interpret the gathered information, and formulate implementation specs for a new Web-based application will take four to six months. To prototype, build, and launch the effort will take another four to six months. This period will lengthen as the number of scenarios involved rises.
· These projects will cost at least $1.5 million in outside help. Firms will turn to eCommerce integrators and user experience specialists for the hard-to-find-experts, technical expertise, and collaborative methodologies required to conduct Scenario Design. Hiring these outside resources can be costly, with run rates from $150K to $200k per month. This expenditure is in addition to the cost of internal resources, such as project owners responsible for the effort's overall success and IT resources handling integrations with legacy systems."
I
agree that 8-12 months is a more realistic timeframe to develop a usable
web site or application that will provide a decent ROI. And, if this is
the overall project timeframe, there is enough time for UE specialists
to use traditional - rather than "discount" - usability engineering methods
to more reliably ensure product usability, a major contributor to ROI.
Depending on the complexity of your software product, somewhere between
$100,000 - $250,000 should buy you a rigorous and thorough UE program.
This is a small fraction of the $1.5 million estimated by Forrester for
all the outside help you will need.
In sum, my opinion - based on over 20 years experience - is that very
fast, very cheap UE methods will not sufficiently reduce project risk
and insure ROI. On the other hand, paying tens of thousands of dollars
for "spot consulting" by "Gurus" will not either. Be prepared to invest
time and money in UE on your development project, but do it smartly -
with a highly structured, time-intensive program of proven methods and
techniques by qualified experts.
 |
Cost-Justifying Usability Randolph G. Bias & Deborah J. Mayhew, Academic Press, Inc., 1994. |
Deborah
J. Mayhew, Ph.D.
Biography
October 1, 2001
Email
the editor with any questions
or comments.
_______________________________________________
Annotated Bibliography
Mayhew,
Deborah J., The Usability Engineering Lifecycle, Morgan Kaufmann
Publishers, 1999 (a detailed description of the kind of rigorous UE program
referred to in the article above)
Bias, Randolph G. and Mayhew, Deborah J., Editors, Cost-Justifying
Usability, Academic Press, 1994 (a detailed description of how to
cost-justify any usability engineering program)
Ratner, Julie, Editor, Human Factors and Web Development, 2nd Edition,
Lawrence Erlbaum Associates, due out in 2002 (parts of this article -
as well as much else of interest to anyone concerned with the usability
of Web sites - will appear in this forthcoming book)
Jacko, Julie and Sears, Andrew, Editors, The Handbook of Human-Computer
Interaction, Lawrence Erlbaum Associates, due out in 2002 (parts of
this article - as well as much else of interest to anyone concerned with
the usability of software products - will appear in this forthcoming book)
