Context-Driven Assessment of Commercial Web Sites

1.   Introduction

While each segment of the World Wide Web has its own needs and expectations of what websites should do, it is generally agreed upon that web usability is important. Web usability, as defined by Gartner Group, is “the measure of how easily, efficiently, effectively and satisfactorily users can accomplish tasks toward specific goals.” Web usability becomes extremely important for those companies that want to use the web as a source of competitive advantage. If a company plans to add the web to its list of strategic and competitive tools, the user experience must be taken into consideration when designing and building the website.

More generally, what is important is not just the usability of a Web site, but its effectiveness in achieving its intended purpose. In this paper, we address the problem of developing a Web Site Assessment Framework. We start in section 2 by evaluating some existing approaches to Web site evaluation. In section 3, we present our framework, and then in subsequent sections, explain the components of the framework.

2.   Review of Existing Web Site Assessment Approaches

The issue of usability of computer-based systems has been addressed by a variety of researchers over the years. Much of this work is represented by the ideas in various excellent books on the subject, including ^{[Nielsen, 1993]} and [Schneiderman, 1998]. The focus of such work has been mainly on the effectiveness of the human-computer interface. In this context, the usability of Web sites is significantly affected by factors such as the inherent hyperlinked interactivity of Web pages, the fact that the Web uses a connectionless protocol, so each hyperlink on a Web page results in an entirely new connection being established to the corresponding server, and the inherently multimedia characteristics of most Web content. More recently, Nielsen has also developed guidelines for Web site design ^{[Nielsen, 2000],} and there have been several experimental studies, including ^{[Chan, 2001]} and ^{[Basu and Muylle, 1999]}.

While the human-computer interface literature is useful, Web site effectiveness has a much broader scope. A number of approaches are currently being used in industry and government specifically for Web site usability analysis. We briefly review these approaches, as well as some academic research projects on the subject.

The U.S. Federal government has recently documented two sets of similar criteria to be used when evaluating and building governmental websites ^{[McClure et al, 2001]}. Although not specifically a framework, the government's approach provides comprehensive lists of specifications to be validated and included in a website with the primary purpose being ease-of-use and readability for users.

One set of criteria titled, Criteria for the Evaluation of Federal Websites, indicates the breadth of the criteria and warns the website developer to appropriately apply the criteria based on website purpose. The purposes indicated include: educational, entertainment, commercial, and services provision. Website developers are urged to design the navigation and content of the site based on the purpose of the site as well as the site's intended audience. Each set of criteria are fairly similar and aim to appropriately orient the user to the mission, purpose, services provided and scope of the website. As perhaps assumed, each set of criteria mandates compliance to Federal rulings such as COPA laws to protect children online and accessibility for all people, including those with disabilities and those with older, less sophisticated technology. Appropriate disclaimers regarding privacy, as well as the responsibility of external content accessible via provided site links, are specified to protect government agencies and appropriately warn users. The specific criteria for each set of government criteria are as follows:

Set # 1: Evaluation Criteria for Federal Websites

• Content
• Search Functionality
• Ease of Use
• Privacy and Security
• Design
• U.S. Policy Compliance
• Accessibility

Set #2: U.S. Government Printing Office Framework

• Consistent look, feel, tone and navigation
• Compliant with Federal policies and regulations
• Accessible
• Accommodate lowest level of software version and user's technical skill
• Provide on and offline support mechanisms

The government criterion focuses on simplicity, both with technologies employed and content. The websites essentially are targeted at every U.S. resident, therefore must cater to the least sophisticated users. Equally stressed is the need for consistency across all government websites. Several lists of standard technologies, aesthetics and content sourcing are provided to aid the developer in consistent design. User support is also stressed in this framework, not so much from a design and navigation perspective, but from off and online customer support capabilities.

The criteria detailed in Set#1 and #2 offer qualitative guidelines and attempt to ensure that all members of the public can access the information without difficulty. Both sets of frameworks are challenged with providing measurable or quantitative performance metrics or specifications. Developers are offered a series of rules or checklists of specifications rather than objective, measurable goals to maximize the user's utility. While the user experience is acknowledged, the approach advised is “simple equals better”. Simplicity of design may or may not actually help the user locate information. Well-designed websites should guide users to their desired location rather than providing a roadmap and hope they can self navigate. Consistency of design also has favorable intentions, but the danger of stressing consistency between Federal websites is potentially design mediocrity. Values such as, “good”, “clear” and “appropriate” rely on the developer's judgment rather than specific measurements, colors, or navigation and link designs.

The “Gartner 15” ^{[Spieler, 2001]} provides a relatively comprehensive framework for evaluating a transaction website of a retailer and the communication website of a government. The Gartner Group designed the Gartner 15 originally as an analytical framework for the retail web community and then later modified these fifteen usability elements to also apply when evaluating the usability of government websites. The Gartner 15 addresses more issues than most frameworks by dividing usability into fifteen components as listed in Table 1. By utilizing specific components and scaling systems with distinct points for scoring at defined levels of performance, the Gartner 15 is a quantifiable, explicit metric to use and provides an objective rating or ranking to websites. Gartner's usability frameworks for retail and government differ only slightly and would be more useful if presented in a single format. Although the measures address usability from the front-end perspective, they fail to consider other front-end components such as aesthetics. Gartner's Web Metrics ^{[Buytendijk and Janowski, 2001]} are distinct from the Gartner 15. The Web Metrics framework offers six perspectives of measurement when used in conjunction and when leveraged offer a highly effective customer-centric web site. Unlike other frameworks, Gartner's Web Metrics views the website as an extremely visible storefront that should look and feel consistent with all other channels and processes of the business. The framework authors stress maximizing user utility as the web is viewed as another customer channel. Websites are similar to a call center or physical store except that the web channel is easier for customers to become disoriented and retreat. Creating an effective e-channel is possible by analyzing several criteria gleaned from multiple sources, such server log files, customer input and comments from the field. Three sets of metrics are gathered and analyzed including:

1. Hard metrics — those with objective statistics that can be gleaned from Web server log files
2. Derived metrics — conclusions and assumptions derived from customer data
3. Soft metrics — subjective and qualitative measurements that aid in interpreting effectiveness

Although useful and insightful, none of these frameworks is both comprehensive and quantitative in measurement. When evaluating a website, an objective method or detailed goal/threshold is necessary to quantify improvement and compare results to other sites. The existing frameworks neglect elements of cross-functional applicability based on website purpose. After surveying the current usability needs, it is apparent that there is the need for a robust framework that is also a comprehensive model for all websites.

An approach to examine the extent to which commercial Web sites support the key online processes necessary in electronic commerce is presented in ^{[Basu and Muylle, 2002]}. In this approach, the particular online processes that need to be supported by a Web site supporting electronic commerce are used as a basis for site evaluation, and used to estimate the extent to which different industries have adopted electronic commerce.

3.   Proposed Framework for Measuring Web Usability

Our proposed framework consists of three dimensions, as shown in Figure 1. These consist of the e-business contexts of the site, the online processes that have to be supported, and the factors that define usability. These are orthogonal dimensions, in the sense that at the simplest level, a measure applies to a specific combination of values from all three dimensions. In this section we examine each of the dimensions in turn, and then show how these are operationalized.

3.1 The Business Context

The ICDT framework ^{[Angehrn, 1997]} explains the virtual marketspace and what comprises it. A firm's website can have activities in some or all of these virtual marketspaces. The current usability frameworks ignore the need to address usability for websites with different levels of functionality in the ICDT framework. A website whose purpose is only to provide information would have different usability considerations than a website whose purpose is to execute business transactions. The framework segments the virtual marketspace into four distinct areas:

• Virtual Information Presence (VIS)
• Virtual Communication Presence (VCS)
• Virtual Distribution Presence (VDS)
• Virtual Transaction Presence (VTS)

The Virtual Information Space consists of the channels by which companies can provide information about themselves, their products, and their services. This could be a company's Internet site or Intranet site that has the ability to provide rich information with a global reach. As an information channel, this is the simplest of Internet-based activities that a company can undertake to: (1) Provide basic information that customers would need, such as product information, company contact information, areas that the company services, and what visitors can do at the site (such as browsing the product catalog, etc.) (2) Build the firm's credibility by having a professional-looking website, which provides information that will build customer confidence, thus enhancing the firm's credibility. Providing information such as a company history, favorable press releases and photographs of the company's facilities can go a long way in reducing customer fears of dealing with a dubious operation. (3) Provide information that will move visitors further along the sales cycle, such as contact information for the nearest brick and mortar retailer, comparisons with competitive offerings or third-party evaluations of the company's offering. The information channel need not be restricted to servicing customers but can also be used by a company to serve the needs of other stakeholders, such as:

• prospective employees
• investors
• supply chain partners
• the media
• the company's own employees

The Virtual Communication Presence is a two-way exchange unlike the information provision in VIS. This communication channel could be through the use of e-mail, chat rooms or any other interactive form. This space allows for companies to exchange information with the various stakeholders in their business: their suppliers, customers, and strategic allies. The simplest use of VCS is to allow customers to ask questions or provide feedback to a company, usually through e-mail. It is important to look at this market space as an extension of the Information Space, where the company is taking information one-step further and allowing interaction. Communication Presence can enhance an Information Presence website. For example, Information Space assists in the creation of a knowledge base that will provide customers information on new uses for products while the Communication Space facilitates a response. This could serve as a means of generating demand for a company's offerings, cultivate customer loyalty, and further reinforce a company's positioning as a destination site.

The Virtual Distribution Space is a distribution channel, which companies can use to distribute their products and services. Distribution is especially relevant for products and services without a physical component, such as music, software, technical support, financial services, etc. The distribution channel can be directed towards customers, media, investors, as well as to the company's own employees. VDS would be most effective when the site includes VIS and VCS.

The Virtual Transaction Space provides a channel to initiate and execute business transactions. However, the first three spaces need to exist in order to make this space truly valuable to a company. A site essentially needs to provide the product or service information followed by the means to communicate with the site and the channel to distribute the product or service and finally the means to carry out the business transaction.

3.2 The Process Dimension

Once the firm decides on the context of its Web site, it is necessary to examine the various processes that have to be used within that context. We identify these processes in terms of the Electronic Commerce Architecture (ECA) defined in ^{[Basu and Muylle, 1999]}. The ECA (Electronic Commerce Architecture) consists of three levels of services: network services, commerce services, and content services. Network services are services that support the network from closed (proprietary, VAN) to open (Internet) networks through which EC is implemented. Commerce services relate to commerce processes in EC, which includes all activities that each party undertakes to complete a transaction, such as search, valuation, authentication, payment and logistics. Content services are services that are specific to a particular industry or domain. It is important to address each of these levels from the user's perspective when evaluating web usability. An effective framework must apply usability metrics that address these levels of the ECA architecture when it is applicable based on the purpose of the website. We briefly describe the commerce processes below.

Search functionality is the extent of support the site provides for searching products online and the extent of support for searching the seller online. The usability of the site will depend on how effective the Search functionality is able to connect users to the information they seek.

Valuation is the process of price discovery and negotiation. This functionality is the extent to which price information is displayed and the extent of support for price determination mechanisms.

Authentication is the process of ensuring the authenticity of the parties involved, as well as the quality of the products and services being transacted. Authentication establishes the identity of one party to another and requires the establishment of trust on both sides. Usability will be greatly affected if neither the buyer nor the seller can establish their credibility.

Payment is the process of defining the payment process, its issues and payment instruments. Payment support could be online or offline, and the extent of support provided by the channel determines its usability. Depending on the extent of online payment support, e-wallets, shopping cart mechanisms, online credit payments, smart card systems, e-cash and payment processes through trusted third parties can all be embraced. Offline payment support refers to check, cash, wire, and offline credit card payments.

Logistics is the process of completing procurement, which is preparing and delivering goods and services. This support includes offline delivery, identification of the nearest point of sale, provision of basic order status as well as online tracking functionalities. The extent of logistics support offered by the site both online and offline will determine the extent of usability.

Customization is the process of tailoring all the above processes to the needs and preferences of specific users. It impacts all the other commerce processes as well as content services within individual industries.

Similarly, there are additional basic usability factors that are relevant. These, like customization, impact the other commerce processes as well as content processes.

3.3 Specific Factors

Once a website's purpose has been defined and the supported processes identified, we can assess what components of the model are most important in evaluating the effectiveness of the website. Based on an analysis of the existing frameworks and their limitations, we have identified factors that are important for Web site assessment. These are described below, organized by the process that each supports.

Search Factors

• Accessibility from External Search Engines: This evaluates the extent to which generic portals and search engines such as Yahoo.com, Altavista and Google direct visitors to the site.
• Simplicity of Location: Clearly, website usability cannot be assessed if the user cannot find the site. Both for retail and government websites, the URL must be logical, such as using the name of the company, or easily located via popular search engines.
• Navigation: The site should be designed such that navigation is intuitive and requires minimal user skill. Links that assist with navigation such as “Return to Top” and “Homepage,” should be provided wherever appropriate. Site maps are also helpful to navigation and web usability. An important aspect of navigation to maximize usability is making sure the user can find what they are looking for in three clicks or less.
• Internal Search: Internal Search pertains to the extent of support offered by the site for searching its site, products, and services. All websites must offer a search engine feature that allows users to search the site. The most common search engines are in the form of a link or a search box. A well designed website should allow the user to successfully search for what they want and to find it as quickly as possible. Search should be easily available from every page on the site.
• Link Control: Link control involves whether a user can return to the original website easily when the website has collaborated with another company's website. All links should be active, appropriate, and give warnings if the link leads to large documents or images.

Authentication Factors

• Third Party Validation: The extent to which the site provides evidence of endorsement or approval by objective and informed third parties, which include experts, authorities and virtual communities.
• Information Clarity and Detail: Information clarity rates a site on whether or not content is clearly written and easy to comprehend. Information Clarity is subjective by nature and may be evaluated differently for a website based on whether the user is an expert versus novice or a first time visitor versus repeat visitor. Since site content is a major basis for establishing the credibility and quality of both vendor and product, clarity is a significant consideration in authentication.
• Advertising: Whether or not ads are necessary for the website depends on the business model of the website. They can provide indirect authentication for the site, based on its selection by advertisers. However, the concept of the advertising component is the fewer ads the better for maintaining usability of the website and avoiding ad overload. Usability is degraded if the content is difficult to reach due to the download of large ads or too many ads.
• Product Information: This component refers to the availability of the description and specific features and attributes of a particular product, which supports product authentication. If a website is designed to sell products, the more information potential consumers have regarding the product, the more likely they will make a purchase.

Valuation and Price Discovery Factors

• Pricing Information: Pricing information for all products on a website should be listed up front and easy to see. The price shown should contain all charges associated with the product excluding tax and shipping charges. The absence of price listings may encourage website abandonment or at least consumer frustration. Pricing information is similar to product information regarding its importance to website type.
• Support of Pricing Mechanisms: The extent to which the site supports not only posted prices, but also various dynamic pricing mechanisms such as negotiated prices, volume discounts and auctions.

Payment Factors

• Payment Options: Usability is enhanced when users know not only what they have to pay for the product but also what they have to pay for shipping, taxes or any other applicable charges. Usability is also enhanced with the extent of online payment support offered by the site. Online payment could be through e-wallets, credit card, smart card systems, e-cash, and payment processes through trusted third parties.
• Security and Privacy: Websites must consider security and privacy concerns, or risk losing customer confidence and patronage. The relevant factors include authentication, privacy, integrity and non-repudiation.

Logistics Factors

• Efficiency: Website efficiency is essentially measured using statistical metrics available via software measurement packages. Metrics to be captured include the number of visitors per hour, abandonment rate, the number of errors, buildup time per page, transaction/page load times, peak load effects and capacity utilization. These depend upon the back-end processes that include logistics support.
• Accessibility and Availability: Websites should be operational 7 days a week, 24 hours a day. Support functions, such as email, call centers and chat options ought to be available $24\times 7$, or at minimum, clearly state hours of operations and time zone. These indicate the level of logistical support for the Web front-end.
• Multi-channel Access: Websites that provide and communicate multiple avenues, or channels for communication and transactions, enhance overall usability. Available channels include options such as call center support, physical mailing address, e-mail, chat rooms, fax, CTI (Computer Telephony Integration) and/or VOIP (Voice Over Internet Protocol).
• Inventory Availability: Inventory availability refers to the ability of a website to inform a potential buyer that an item is not currently available. Websites can perform this task using several different methods. This is most effective when a website is able to operate in real time or near real time.
• Shipping Options: Websites must also consider the buyer's needs in product delivery. Different options and applicable rates should be easy to find. Prices associated with the different channels should also be available. Total order cost should be immediately updated in the event that the method of shipping changes before the transaction is complete. Any charges that apply to packaging methods, such as fragile or gift wrapping, should also be explained. Additionally, it is important to explain each delivery method and include estimated time until the product arrives.
• Form Processing: Form processing refers to the online access of forms, the form processing procedures, and the return processing procedures. Usability benefits are derived from providing the option of accessing the forms online, the ease with they can be accessed, and the availability of the different processing procedures.

Customization Factors

• Repeat Benefits: Repeat-customer benefits such as profiles, purchasing lists and shopping benefits improve the usability of transactional retail sites. By providing ongoing and updated information on a website, users also have a benefit or reason to repeatedly use the website. Providing these optional benefits to consumers encourages the repeated use and overall usability of the website.
• Match with Target Audience: Websites targeted to specific audiences must be uniquely tailored to fit the users' requirements whether by language, currency, shipping practices and/or cultural requirements. Clear statements communicating website purpose and intended target audience must be readily available to all website visitors. Regardless of the website's purpose, the appropriateness of language, tone, content and processes must cater to the target audience to be deemed useable and effective.

Usability Factors

• Integration: It is important that a website both clarify its scope and satisfy that scope properly. Many websites, such as governments and corporate intranets with heavy informational focuses, should be concerned with portal integration. Information from independent agencies and departments should be correctly integrated without any conflicting information for the website.
• Consistency: Consistency is an important aspect for all of the usability frameworks critiqued. A website should be concerned with consistency on many levels of the website, such as information, integration, style and design. It is important that all information on promotions, customers, etc. is consistent throughout the website.
• Aesthetics: Aesthetics encompasses the look and feel and the design of the site. The most important components of the look and feel of a website are Color, Font, Multimedia and Screen Resolution. When comparing the aesthetics between websites, each website should be evaluated for aesthetics using the same base client machine configuration defined by the company.

3.4 Using the Framework

The significance or weight of each factor depends upon both the business context, and the particular process that it is intended to support. Thus, the framework is operationalized by developing a set of weights for each factor within each process, and then factoring in the significance of that process within each specific business context.

4.   Conclusion

Web site effectiveness is not just a desirable quality; it is a necessary quality for companies who want to succeed online. We have proposed a framework that is both meaningful and comprehensive. It is also more comprehensive than existing frameworks, and based on a progressive contextualization of usability criteria that makes it potentially more effective.

It is important to note that the objective of such frameworks need not be to develop a single generic model for all possible Web sites. In other words, it is possible that different organizations will use different weights across the framework to reflect its own priorities. However, the set of factors, and the three-dimensional framework within which the factors are set, remains stable across all settings. This has been empirically validated through a focus group of Web users, and we are currently working on more structured validation processes.

Figure 1:Figure 1:

Figure 2:Figure 2:

Figure 3:Figure 3: