9 User Feedback ................ 163

Chapter 9 User Feedback

In this chapter we will present the feedback that we received from the subjects who participated in the user studies. We recorded their spontaneous comments during the experiments and we asked them after each of the four sets of queries if they had any comments or observations (we presented the subjects first with a training set and then with the set of test queries for both the recognition and the generation task). These comments are summarized in the second column of the feedback table, provided they are different from answers found elsewhere in the feedback table. At the end of the experiments we asked a set of three questions, which are displayed in the other column headings, and the answers are summarized in the appropriate columns.

The feedback is instructive in several ways. First, the overall response to the InfoCrystal was positive, ranging from "The InfoCrystal was absolutely clear. ... It was much, much easier with the InfoCrystal. ... I felt confident with the InfoCrystal," to "the InfoCrystal was actually not that bad, even usable." Second, the feedback touched on and clearly articulated features that need to be made available to users. We purposely chose to have the subjects use a version of the InfoCrystal that did not have all the enhancing features that we have implemented or that we plan to develop, because we wanted to see how well they could use the InfoCrystal in its most basic and primitive form. In particular, the following feedback is worth highlighting:

Queries Easy to Formulate with the InfoCrystal: Several subjects observed that the InfoCrystal is better suited for certain queries, such as queries of the type "at least, exactly, or at most M out of N features". The experimental results support this observation.

· Inverse Relationship between Boolean and InfoCrystal Queries: One of the subjects, in particular, articulated that there is a "confusing" relationship between the cognitive effort required to formulate certain queries. Simple Boolean queries that predominately use the OR operator to coordinate components often require users to select many icons in the InfoCrystal mode. A Boolean query with a simple or repetitive structure can be easy to grasp because it reduces into a simple pattern, whereas it requires users to examine and select many of the interior icons in an InfoCrystal. Once users have more experience with the InfoCrystal representation, they will be able recognize how certain selection patterns are related to Boolean statements. Having a Boolean feedback window will help users to learn this (see also Hybrid Interface bullet). Further, we can assist users so that they can select groups of interior icons by interacting with the border or criterion icons and use the InfoCrystal as a Boolean Calculator (see section 4.2.2).

· Combinatorial Explosion: Many of the subjects commented that InfoCrystal queries with two or three features were straightforward to interpret and use, but that InfoCrystal queries with four features were overwhelming at first. This should come as no surprise because the number of interior icons grows exponentially as the number of features increases.
In a similar vein, a subject articulated that the InfoCrystal in its most basic form requires users to examine all icons, especially if we have a complex information need. This is an expensive and cognitively demanding operation, especially when the number of features considered at same time is more than three.

· Translation of Complex Queries in Stages: Some of the subjects would have liked to have been able to translate complex queries and perform the corresponding selections in the InfoCrystal in stages. This request touches on the issue of being able to save partial results in the InfoCrystal framework and then to combine these partial results to arrive at the final selection pattern. We address this issue in sections 4.2.1 and 4.2.2.

· How to juggle the different possibilities at the same time ? One subject commented that the OR operator and especially the OR combined with the NOT operator was difficult to translate into the InfoCrystal. The OR implies that there are multiple ways to satisfy the information need. The InfoCrystal in its basic form requires users to think about the different possibilities simultaneously, whereas it would be easier for users to consider each of these possibilities one at a time or in stages. In section 4.2.2 we describe how the InfoCrystal has been or can be extended to address this issue.

· Hybrid Interface: Several subjects concluded that it would be advantageous to have a hybrid retrieval interface, where they could use a textual Boolean and the InfoCrystal interface to formulate queries. Similarly, subjects noted that it would have been helpful if there had been a feedback window that showed the Boolean query that was equivalent to the current selection pattern of the interior icons. We could easily provide this type of feedback, but we wanted to test how well novice users could use the InfoCrystal without it.

· List Interface: One of the subjects suggested that it could be beneficial to see a list of the data items that are retrieved by the selected icons. By examining individual list items users could determine if they are retrieving the information that they are looking for. We have implemented ranked-list interface, but, as we have stated before, we wanted to see how well the subjects could use the InfoCrystal in its most basic and primitive form.

Table 9.1: summarizes the user feedback received during the evaluation experiments and in response to specific questions. The user population consisted of 4 women & 6 men; all had at least a College Education and all had some experience with Boolean Retrieval. Legend: IC = InfoCrystal.

Table 9.1 (cont.): summarizes the user feedback received during the evaluation experiments and in response to specific questions. The user population consisted of 4 women & 6 men; all had at least a College Education and all had some experience with Boolean Retrieval. Legend: IC = InfoCrystal.