1
|
- Housekeeping
- Evaluations
- Morse, E., Lewis, M., and Olsen, K. (2002) Testing Visual
Information Retrieval Methodologies Case Study: Comparative Analysis of
Textual, Icon Graphical and 'Spring' Displays Journal of the American
Society for Information Science and Technology (JASIST) PDF
- Reiterer H., Mußler G., Mann T.: Visual Information Retrieval for the
WWW, in: Smith M.J. et al. (eds.), Usability Evaluation and Interface
Design, Lawrence Erlbaum, 2001 PDF
- searchCrystal Studies
|
2
|
- Motivate domain choice.
- Perform task and need analysis.
- Describe design approach and information visualization principles used.
- Develop prototype.
- Have an "domain expert" use the prototype and provide
feedback.
- Class Presentation
You have 15 min. to describe task analysis and your design
approach.
Demonstrate your prototype.
Report on the "domain expert" feedback.
- Create Report
20 to 25 pages, written as a standard paper è 10pt, double-spaced
Provide screenshots of prototype and explain design
approach.
Include URL of prototype.
- Hand-in
Hardcopy of report.
Post report online and send instructor an email with the URL.
|
3
|
- Many Tools Proposed
- Few Tested and Often Inconclusive / Fare Poorly
- Simplify Evaluation
- è Focus on Method (instead
of implementation)
- è Only Static Aspects
- POI = Point of Interest Visualizations
- Glyph = Graphical Entity
- Conveys data values via attributes such as shape, size, color
|
4
|
|
5
|
|
6
|
|
7
|
|
8
|
|
9
|
|
10
|
|
11
|
|
12
|
|
13
|
|
14
|
|
15
|
|
16
|
|
17
|
|
18
|
|
19
|
- Validate Design Approach
- How does Overlap between Results Actually Correlate with Relevance?
- User Study
|
20
|
- Method
- Use Ad-hoc track data for TREC 3, 6, 7, 8
- Systems search the SAME Database
- Automatic Short Runs
- 50 Topics and 1,000 Documents per topic è 50,000 documents
- Retrieval systems can submit multiple runs
- è Select Best Run based
Mean Average Precision
- TREC 3 19 systems 928,709 documents
found
- TREC 6 24 systems 1,192,557 documents found
- TREC 7 28 systems 1,327,166 documents found
- TREC 8 35 systems 1,723,929 documents found
- Compute Average by summing over all 50 topics and divide by 50
|
21
|
|
22
|
- Compute overlap structure between top 50 search results
of 35 random groupings of 5 retrieval systems for 50 topics.
|
23
|
- How does Overlap between Search Results
Correlate with Relevance?
- Authority Effect – the more systems that find a document, the greater
the probability that it is relevant
- Ranking Effect – the higher up a document in a ranked list and the more
systems that find it, the greater the probability of its relevance
- Validates searchCrystal’s Design Approach
- searchCrystal Visualizes Authority & Ranking Effects
- searchCrystal can Guide User’s Exploration
Toward Relevant Documents
|
24
|
- Validate Design Approach
- How does Overlap between Results Actually Correlate with Relevance?
- User Study
- http://www.scils.rutgers.edu/~aspoerri/study/UserStudy.swf
|
25
|
|
26
|
|
27
|
- Nine undergraduates.
- Short Introduction and No Training.
- Randomized presentation order of data sets and display type.
- Subject selects ten document;
- Visual feedback about correct top 10
- http://www.scils.rutgers.edu/~aspoerri/study/UserStudy.swf
- Test for Cluster Bull’s Eye and RankSpiral displays:
- 1) How well can novices use visual cues to find the documents that are
most likely to be relevant?
- 2) Performance difference in terms of effectiveness and/or efficiency?
- 3) How much document’s distance from the display center will interfere
with the size coding used to encode its probability of being relevant
|
28
|
- Hypothesis 1: “Novices can perform the task.”
- Error is minimal for the top 7 documents and increases rapidly after the
top 7 documents for both displays.
- Novice users can use the Cluster Bulls-Eye and RankSpiral displays to
select highly relevant documents, especially the top 7 documents.
- Hypothesis 2: “RankSpiral outperforms Cluster Bulls-Eye.”
- 8 of the 9 subjects performed the task faster using the RankSpiral.
- Average time difference was 7.89 seconds.
- The one-sided T-test value is 0.033, which is significant at the 0.05
level.
- 7 out of 9 subjects performed the task more effectively using the
RankSpiral.
- Average “relevance score” difference is 0.034.
- The one-sided T-test value is 0.037, which is significant at the 0.05
level.
- Hypothesis 3: “Distance from center dominant cue.”
|
29
|
- Relax searchCrystal’s design principles?
- Mapping documents found by the same number of engines into the same
concentric ring.
- Option: Distance and Size encode likelihood that a document is relevant.
- Internet search results:
- Concentric rings are of value,
because it is much harder to estimate a document’s probability
of being relevant.
|
30
|
|
31
|
|
32
|
- Authority & Ranking Effects
- Comparing Results of All Retrieval Systems at once
- Comparing Results of Random Subsets of Five Systems
- è Validating
searchCrystal’s Design Principles
- User Study
- Identify Top 10 Docs in Cluster Bull’s Eye and RankSpiral
- Novice Users can use the two searchCrystal displays
- Statistical Difference between two displays
- Distance from center is dominant visual feature
|
33
|
- Please read the two papers published by me in First Monday:
http://www.firstmonday.org/ISSUES/issue12_4/
- Approach
- 1 Visualize Popular Wikipedia
Pages
- Overlap between 100 Most Visited Pages on Wikipedia
for September 2006 to January 2007
- Information Visualization helps to gain quick insights
- 2 Categorize Popular Wikipedia
Pages
- 3 Examine Popular Search Queries
- 4 Determine Search Result
Position of Popular Wikipedia pages
- 5 Implications
|