Adventures in Beta Testing CD-ROMs
This paper discusses the development of a CD-ROM from the point of view of a beta tester. It covers the preparation for testing, product installation, and the various commitments and responsibilities. It also emphasizes how the tester's experiences and comments affect product development.
Before the testing period begins, the vendor contacts the test sites to determine whether or not they meet the minimum hardware and software requirements to work with the system. This includes determining the type of computer used, the maximum memory capacity, the version of the operating system, and the types of peripherals available.
The type of computer is particularly important in the IBM environment because not all clones operate the same. Some vendors may decide to support only certain brands of machines rather than try to accommodate all clones.
The programmers at EBSCO tested The Serials Directory on IBM PCs and ATs. When I installed it on my XT, it operated only with greatly reduced functionality. The problem turned out to be an uninitialized variable that was easily fixed.
The memory capacity is important because the upgrades of the software tend to require increasing amounts of memory. Products like BiblioFile that began testing at 256K RAM soon required 512K. Many have since moved to 640K. Several products require hard disks with a minimum capacity of 10 megabytes (MB). Some require a substantial portion of this memory for proper operation.
A product like The Serials Directory requires 560 K of random access memory after loading DOS. This means that it consumes almost every bit of available RAM. Users will have to unload any memory-resident programs before loading EBSCO-CD. They may also have to modify their CONFIG.SYS files to decrease the number of files and buffers to the minimum levels specified. ..Omit?? The User's Manual makes no mention of these requirements; but the installation program does.
In addition, the operating files consume almost three megabytes of hard disk space. The history subdirectory which stores the files of all the queries also continues to grow with use. The system manager will want to monitor this directory and possibly clean it from time to time. One must also remember to install each update of the quarterly CD-ROM. On the other hand, Baker & Taylor's product, currently in field testing, requires 8 MB of hard disk space to store data files which get updated on a weekly basis.
The version of the operating system determines the fullness of functionality of commands. Some earlier versions are not compatible with later versions. Later versions also provide newer commands not present in earlier versions or fuller functionality that certain applications depend on. An increasing number of CD-ROMs require the use of Microsoft's CD-ROM Extensions which, in turn, requires DOS 3.1 or greater.
CD-ROM products require certain peripherals like the CD-ROM drive. They may recommend other optional equipment like modems and printers. While most products support most CD-ROM drives, some applications may require a particular brand of drive or a specific interface card.
Sometimes vendors may provide the test site with some or all of the necessary equipment to do the testing. This is more likely to occur if the product requires special equipment that may not yet have widespread use or an established market base. It may also occur if existing systems require only minor upgrades to accommodate the test product. Some vendors may let the testers keep some or all of the equipment at the end of the testing period.
Whatever arrangements vendors and testers agree to are usually spelled out in a contract signed by both parties before the testing begins. This usually includes specification of the duration of the testing period, what the vendor will supply, what the tester will supply, and the responsibilities of the tester. It may also include special provisions such as non-disclosure agreements or a prohibition against writing reviews based on prototype versions. Sometimes the agreement may specify some sort of compensation such as a complimentary one-year subscription to the product.
A complimentary subscription lets the tester see the results of his or her work. Even if the tester decides not to purchase the product, he or she can see how it develops and incorporates suggested modifications. The vendor also benefits if the tester continues to use the product and recommends improvements even after the conclusion of the testing period.
Many CD-ROMs come with a program to create the necessary directories and subdirectories and to copy the appropriate files to them. It may reside on the CD-ROM disc or it may come on a floppy disk along with other operating programs. The installation or setup program usually consists of a compiled file or a batch file that calls up a compiled file. Rarely can the user read it to determine what it will do upon execution. Setup programs which permit reader access in prototype versions often get distributed in compiled form when the product goes to market.
Besides creating the directories and subdirectories and copying files to them, the installation program often modifies the AUTOEXEC.BAT and CONFIG.SYS files to accommodate the application. While this is fine for dedicated systems, users who want to run more than one application on the same hardware may encounter problems because changing these two files may interfere with other software that these files usually control.
For example, a product like The Serials Directory/EBSCO CD-ROM that requires virtually all available RAM keeps the number of buffers to a minimum. The prototype version used four. The current version uses ten. Since each buffer sets aside 512 K of memory, a user who follows this configuration may not have enough memory available to operate a word processor if it requires 20 buffers. Likewise, a configuration of 20 buffers will prevent loading The Serials Directory/EBSCO CD-ROM.
The tester will want to guard against destroying any files during the installation process. It is a good idea to make backups before making any changes--especially if one isn't sure if the files can be recovered. Installation programs often perform this task; but a wise user protects himself in case one program doesn't.
One alternative would have the installation program create batch files for the user to invoke upon changing applications. This option would provide more user flexibility at the expense of ease-of-use which explains why it has not been implemented. It would require some knowledge of DOS commands to manage and use these programs effectively.
Computer Knowledge Helpful
While beta testing does not necessarily presuppose any knowledge of computer hardware or software, the more one knows about these matters, the easier and more enjoyable will be the experience. It is usually more important to have a tester who is very familiar with the daily operations that the CD-ROM application will address.
It helps to have a modicum of computer knowledge however. The tester usually installs the application on the basis of printed instructions or steps outlined in the manual. Often, user manuals of prototype systems are very brief and incomplete. Others, like EBSCO's, are quite detailed. In fact, EBSCO's completed version did not differ much from the test version.
Sometimes, the vendor's technical staff will assist with installation by telephone. Also, all troubleshooting is done over the phone.
Buyers of turnkey systems expect them to work properly right from the moment of installation. Beta testers, on the other hand, should expect problems. They should even seek them. Their responsibility is to try to ferret out the bugs and to suggest improvements prior to marketing the product. In this sense, beta testing is an adventure.
While testers may sometimes find their work frustrating, they usually consider it an exhilarating experience because they participate in the creative process of product development. As a potential customer, the tester also has a lot of input in refining a product to his or her requirements. The consumer drives the CD-ROM market just like any other market. The vendor will usually try to comply with market requests or needs if it will help sell the product.
However, modifications often require tradeoffs. They may require extensive programming resources that the vendor may not be willing or able to accommodate. Making some enhancements may provide attractive features but may not help sell a product. Certain features may require more programming costs that the incremental benefit does not justify. Others may demand too much memory.
Including certain features such as Boolean searching to satisfy one class of users such as experience searchers or librarians may present difficulties to novices. Programming, editorial, and marketing staff need to resolve which features to include and exclude as well as what tradeoffs they need to make to accommodate certain features and requests.
RESPONSIBILITIES AND COMMITMENTS
The beta tester examines the product, first of all, to determine that it performs as the producer says it should. This involves reading the manual and testing some of the strategies to get a feel for the system. The first thing one notices upon entering an application is the user interface. This includes whether it is menu-driven or command driven. If it is command driven, how easy are the commands to learn and use? If it is menu-driven, are the options well-organized? Is it easy to navigate around the system? Are the menus and screens clearly readable and understandable? What kinds of special features does the product include? What does the database contain and how current is it?
Normal Working Conditions
The tester then uses the product in everyday operation to give the system a fair test under normal working conditions. This involves extra work because one should not abandon current operating procedures during a testing period.
The tester tries to find typical work problems to solve with the product. One also tries to thwart it to see how it responds under worst-case scenarios. This can include doing long, complicated, time-consuming searches that require all of an application's resources. It also includes trying to think of ways novices might abuse the system. For example, if a product accepts input from ASCII files, what happens if somebody imports non-ASCII data? Some products show more resilience than others to such "problems".
Testers also try to find difficult problems to solve with the system. They look for problems for which solutions may not be available from other sources or which would be very time consuming to solve with other tools. It also involves comparing results with the various alternatives.
Beta testers should also try to push the technology or the application to its limits. They should try to find new uses and applications for products. While the programmers have a set of functions that they want the product to perform, users often approach tasks with different assumptions or have different needs that they attempt to satisfy with the product.
One challenge a tester faces involves determining lacunae in the documentation. Some products present more of a challenge than others because they come with very incomplete information. While testers should identify those omissions, their responsibility does not involve developing the manuals or instructional aids.
When the documentation presents a relatively complete picture of the product, testers often just need to make editorial suggestions. They usually depend on serendipity or happenstance to find features not described in the manuals or tutorials. They may have to call technical support to find out how to perform a particular task. They can spend lots of time on the phone resolving problems.
Testers should keep logs of their activities so they can retrace their steps to recreate a particular problem if the need arises. This usually takes the form of a written log. However, some vendors, like EBSCO, incorporate a software program in their testing modules to track keystrokes. In the event of a problem or system failure, the tester just sends those particular files to the vendor for examination and reconstruction of the problem. This decreases the telecommunications time and costs and lets the vendor examine the problem clearly without any ambiguities or confusion arising from trying to remember the particulars that led to the problem. The technicians don't have to try to guess what happened or negotiate complicated procedures with a computer novice.
In developing a system, programmers have a set of assumptions that do not always match user expectations. The assumptions determine the flowcharting, the menu layouts, and their interrelationships. This, in turn, affects ease of use. For example, the developers of BiblioFile envisioned their product as a tool for current cataloging and placed the indices on the most recent disc. Those who wanted to use it for retrospective conversion had to keep swapping discs--a very time consuming process.
The disc swapping eventually resulted in a problem of open files and apparent data loss. The Library Corporation corrected it in a couple of working days. The solution resulted in the "queue" which lets the operator store identification keys for up to 99 records before having to change discs. This single feature permitted doubling daily work output.
Sometimes users find other applications that the programmers did not envision. For example, though The Serials Directory was not designed as a collection development tool, it could certainly serve such a purpose. The Local Titles feature aimed to limit a long list of citations to only those titles held by the library. However, by selecting a particular index from the Index and Abstracts authorities list and limiting it to local titles, a user can easily determine whether or not to cancel certain titles. Librarians could also use this feature to compare local holdings against indexed titles to quickly determine invoice payments when publishers charge sliding scales based on subscription levels.
Testers and vendors usually keep close communication ties during the test period and sometimes afterwards. When testers find "bugs" in a product, vendors usually solve the problems quickly. We mentioned how quickly The Library Corporation developed the "queue".
The first version of The Serials Directory/EBSCO CD-ROM had a problem in the export format for full CONSER records. The program mis-read the directory structure and truncated five bytes and the end of field delimiter from each field. This corrupted the file and prevented its importation into a local acquisitions system. Upon identifying the cause, we spoke with EBSCO's technical staff. As they were preparing the tapes to send out for mastering the next day, they corrected the problem immediately and incorporated the modifications on the new disc.
On another occasion, we mentioned in passing that it would be nice to have a feature in The Serials Directory/EBSCO CD-ROM that would accept a file of periodical titles created on a word processor. The next day, to our surprise, we received a diskette by Federal Express that provided just such a utility. Although the program required inputting the International Standard Serial Numbers (ISSN) as the identifying key, this process took far less time than searching each title on the CD-ROM and saving it individually.
Testing agreements usually specify that the testers cannot write about the product on the basis of the prototype because the market version may differ substantially from the test version. However, upon completion, testers usually have an advantage over other reviewers. They have already climbed the learning curve. They have become intimately familiar with the product as they have seen it develop over several versions. Consequently, they can review it more quickly and thoroughly for publication.
They may also use their knowledge and experience as the basis for a conference paper. This benefits the individual who gets professional recognition. It also benefits the vendor who gets market exposure. Audiences usually give more credibility to independent endorsements of products than they do to company representatives. On occasion, a vendor might even request a beta tester to make such a presentation on his behalf to maximize audience exposure and to cultivate positive response from peer endorsement.
We have examined several facets of the beta testing process from preparation and installation through use in the daily workflow to the market version. We looked at the tester's responsibilities and commitments, the expectations placed upon him or her, and the relationship between the tester and the vendor.
While testers expect to encounter difficulties and frustrations, they must keep an open mind and a flexible attitude. If their duties may seem burdensome at times they can consider the contributions they make to their colleagues in assisting in the development of products that facilitate their work. In addition, they can reap many benefits both personal and professional.