 |
Measuring
the Value of Usability Engineering
Practitioners
in the field of usability engineering refer to the demonstration of the
value of usability engineering as the "Holy Grail" of the field. Such
an allusion implies that this proof is something that may or may not exist,
but that would be the treasure of a lifetime. If proof of the value of
usability is so desirable, why don't usability engineers spend more time
measuring it? Here are seven reasons I have observed why usability engineers
don't measure the value of usability efforts, rebuttals to each argument,
and suggestions for how to remedy them. At the conclusion of this discussion,
I will identify and examine a commonality among the reasons that may be
the single most problematic barrier to the measurement of the value of
usability.
Reason
#7: There are enough clients willing to invest in usability engineering
without requiring it be justified it in terms of value.
Rebuttal:
Although the usability engineering profession has gotten by to date without
having to prove the value of its services, the tide is turning. Given
the current economic downturn, especially within the IT industry, usability
engineers are now being asked to prove their value, especially in terms
of dollars and cents.
Remedy:
The resolution to this situation is difficult, and multi-faceted. First,
usability professionals should be trained in the techniques of cost-justification/ROI
calculation. The ability to predict and measure the dollar value of a
usability engineering effort should be a standard component of the usability
engineer's panoply of tools. Second, client expectations should be raised
to a new level. They should expect to be provided with such analyses.
Although it will initially place an additional burden on usability engineers,
the long-term benefits will be a better-respected discipline. Third, the
state of the art of the usability/user-centered design process should
evolve to include more appropriate measures of the value of the approach,
with an emphasis on market-based measures. No longer can the usability
profession afford to remain among the outer circles of the IT industry.
It must continue to integrate with technology, marketing, and other core
components of product development, and be measured by meaningful market-based
criteria.
Reason
#6: Measurement begets a fear of lack of positive results in the measurement
of value.
Rebuttal:
Fear of measurement is eminently human. It is inherently frightening to
put one's skills and abilities to the test. However, with regard to increasing
the credibility of usability engineering, the benefits of increased accountability
will certainly outweigh the costs. Further, measurement should not be
a threat if it is matched with reasonable client expectations and appropriate
success criteria.
Remedy:
In putting itself to the test, the usability engineering profession is
asked to take difficult steps that will ultimately result in an improved
perception of the discipline. If the profession wishes to grow, usability
practitioners must move toward greater accountability, and hence, must
measure outcome. It will become incumbent on the usability engineer to
not only learn how to measure results, but to learn to set client expectations
appropriately. User-centered design, although a low-risk approach, cannot
deliver ironclad guarantees of total product success. Clients must be
made aware of the scope and limitations of the approach; result should
be viewed in this light. Success criteria such as financial benefits,
as well as other measures of value (depending upon the nature of the goals
set at the start of the project) should be appropriate and reasonable.
Reason
#5: The ideal way to prove the value of usability engineering is to
perform a classic experiment in which the same product is designed and
developed twice: once with, and once without a usability engineering component.
Many usability engineers haven't been trained to do this, don't need to
do this to practice usability engineering, typically don't have time to
do this, and likely couldn't get anyone to pay for doing this.
Rebuttal:
Although taking such an approach to product design and development is
unlikely in actual practice, there is no reason why such inquiries should
not be undertaken, perhaps under the umbrella of academia. The results
of such studies, taken in aggregate, would likely produce critical insights
into the role of usability engineering in product design and development.
Remedy:
The usability profession should forge stronger ties with academia. Typically,
collaboration between industry and academia has proven to be beneficial
to both entities. Including basic research skills, including experimental
design and data analysis, in the formal training curricula of usability
professionals would produce potentially valuable insights for industry,
as well as training and experience for usability engineering students
who could then apply this training in real-world job settings.
Reason
#4: Even if one were to conduct a classic experiment, there are too
many variables to consider and measure. How can you isolate usability
and attribute value to it?
Rebuttal:
Imagine if researchers in medicine, education, the social sciences, and
other fields measuring human behavior and responses to intervention, threw
up their hands at their inability to measure results because there are
too many variables. Yes, measuring the effects of multiple variables is
difficult. But it is not impossible.
Remedy:
In the behavioral sciences, measuring complex constellations of variables
is addressed by multivariate data analytic methods. Usability engineers
need to make such techniques part of their arsenal. These methods and
suggestions for where to learn about them are discussed below.
Reason
#3: The value of usability has already been demonstrated. Studies
from universities, professional organizations, practicing professionals,
and companies that sell usability services frequently publish data that
offer proof of the importance of usability. It's not necessary to do this
again.
Rebuttal:
The IT field, technology products, and customer expectations are constantly
evolving. Assumptions of the value of usability from the past may or may
not apply to a new product in a new context. Further, even if the value
of usability has been proven, the message does not seem to be reaching
all the right people. I would hazard a guess that the vast majority of
IT managers are not convinced of the value of usability. All too frequently,
usability professionals must first sell the concept of usability prior
to being given a chance to demonstrate it. Imagine an architect, prior
to every assignment, having to convince a potential client of the value
of architectural design in building a home. In some cases, a project may
reach completion, with the client questioning the methods and conclusions
of the usability engineering effort.
Remedy:
A demonstration of the value of usability should be an integral aspect
of every project. Measuring results against stated objectives should be
routine. Until the time that usability engineering is as well accepted
as architecture and other established fields, it is the responsibility
of the usability professional to provide the data that will lead it to
be so.
Reason
#2: It's not necessary to demonstrate empirically the value of usability
engineering; it's common sense.
Rebuttal:
Common sense is one of the most overrated aspects of product design. Common
sense isn't as common as we'd like it to be. For example, if all it takes
is common sense to design a successful user experience, why is there so
much poor design out there? The value of usability is also not common
sense. An example: Many usability professionals complain about the poor
usability of a popular desktop application suite. Common sense would dictate
that this product should be a marketplace flop. Yet, this suite has locked
in a huge market share. The value of usability engineering and, indeed,
usability itself is far from common sense.
Remedy:
Usability professionals should be first and foremost, empiricists. The
field should be primarily data driven. Common sense has heretofore not
been quantifiable; therefore, other measures should be used to prove that
usability is a valuable product attribute. Subjective measures such as
user satisfaction are very useful, but should be supplemented with objective,
behavior-based measures such as time on task, error rates, and requests
for help. Other measures beyond usability should also be taken (see Reason
#1, below). In addition to usability professionals striving to become
empiricists and seeking training in the requisite skills, clients should
also be aware of the need to demonstrate empirically the value of usability.
Clients should expect that usability professionals will be trained in
empirical methods. Universities and professional organizations should
offer such training. Further, usability engineers should also be certified.
Certification will result in a selective advantage for qualified practitioners,
and allow consumers of usability engineering efforts to consequently feel
more secure in their investment. Finally, processes should be documented
that highlight and facilitate the data-driven aspects of usability engineering.
Reason
#1: The wrong attribute is being measured. The ultimate measure of
the success of a product is whether it meets its goals as stated at the
inception of the project. And the greatest contributor to that success
is not usability per se, but rather, the total customer experience. The
total customer experience consists of everything the user sees, hears,
and touches with regard to a product. It includes everything from the
technical "guts" of the system, to the ease of use of its user interface,
the product's usefulness, the aesthetics of its visual design, the perceived
user value as communicated in the marketing messages, and everything in-between.
It is not possible to measure such a broad concept.
Rebuttal:
Usability is a critical component of the total customer experience, and
can and should be measured. However, the total customer experience itself
should likewise be measured.
Remedy:
Usability engineers should augment typical usability data analysis methods
with multivariate analytical techniques (such as regression analysis,
cluster analysis, and meta analysis). Multivariate techniques can measure
key variables contributing to product success with regard to their individual
contributions and their combined effect. The total customer experience
is only as strong as its weakest link. A single weak component of the
experience can overcome the positive effects of all other components and
result in product failure. Conversely, though less frequently, a single
superior component of the total customer experience can compensate for
all other weak components. In either case, more robust multivariate techniques,
along with a focus on marketplace success as a measure of overall success,
can help identify which components - usability among them - cause a product
to either succeed or fail from a financial standpoint.
Reflecting
on these reasons, it seems that there is a common thread that runs through
virtually all of them. In general, the goals of usability engineering
seem to be at cross-purposes with those of the business world. Usability
engineering focuses on human behavior. Human-computer interaction research
and theory are the raisons d'être of the approach, whose goals are to
improve the experience of the user with the technology. Contrast this
with the "bottom line," ROI-driven goals of the business world. The primary
emphasis is not on the customer experience per se, but rather, on profit.
One would hope that a superior user experience would correlate with profit.
However, although the goals of usability engineering and the business
world should and often do coincide, each discipline brings with it its
own way of defining and approaching the measurement of value. Further,
in practice, each discipline is typically woefully uninformed of how the
other functions. Consequently, each measures value differently, and independent
of a consideration of the other's contribution to that value.
How,
then, can the usability engineering and business worlds be brought in
line with one another? At this point of the evolution of the IT industry,
a common definition of "value" needs to arise. Each discipline must learn
more about how the other functions. There must be a merging of the goals
of usability engineering and the business world. From the perspective
of the field of usability engineering, through increased awareness by
practitioners and clients; training for practitioners; and new and more
stringent, business-oriented standards of practice, the measurement of
the value of usability engineering will result in more credibility for
the profession, increased value to clients, and a demonstration that usability
is a critical component of product success. From a business perspective,
there needs to be an increased understanding of usability engineering
and the total customer experience, and their contributions to product
success. Those members of the traditional IT business community - project
managers, market intelligence professionals, programmers, other technicians
- need to become aware of the goals, methods, and contributions of usability
engineering to product success. Through increased cooperation between
the two disciplines, "value" will become a shared goal, with a common
definition and measurement that will drive both disciplines toward product
success.
Carol
Righi, Ph.D.
Biography
June 17, 2002
Email the editor with
any questions or comments. |
|
|
|
|
