CSCW Evaluation

CS 294-7: CSCW
November 25, 1997

Methods for the Study of Groups

McGrath, Joseph E.

• Every method has built-in flaws, and advantages
• Consistent outcomes across multiple studies using multiple methods -> confidence in that outcome
• Key question: Is the evidence from this study consistent with the evidence from other studies on the same problem that used different methods?

Central Ideas

1. methods enable and also limit evidence
2. all methods flawed, but all are valuable
3. different flaws in different methods can be counteracted by using multiple methods
4. choose methods so that each one's strengths offset the weaknesses of other methods

Research tries to maximize:

1. Generalizability of evidence over lots of people (sample surveys and formal theory best)
2. precision of measurement of people's behaviours (lab experiments best)
3. realism of experimental situation (context) (field studies best)

Not possible to maximize all three things at once!

Basic research questions:

• How often does X occur? (baserate question)
• Are X and Y related? (relational question)
• Is there systematic covariation of the values of X and Y? (correlational question)
• Is Y present when X is present (absent)? (comparison/difference question)

Not possible to control all factors (X). To do a true experiment, must randomize across all the factors one is not controlling.

• True experiments can reveal much about causal relations between variables
• reduce possible scope of study
• reduce realism of context

Four types of validity

• Internal validity (true experiments)
• Statistical validity
• Confidence with which one finds a real difference between X and not X cases
• When results significant, researcher may conclude that chance was not the only operator
• may not conclude that researcher's hypothesis produced the significant results unless most rival hypotheses are eliminated
• Construct validity - good mapping of experiment to theory and concepts
• External validity - basis for estimating how results would hold up in other situations

Increasing internal and statistical validity threatens external validity (but not necessarily the converse).

Four final points (a trifle obvious)

1. results depend on methods (of course!)
2. all methods have limitations, hence any one set of results is limited
3. Not possible to maximize all desirable features of method in any one study; there are trade-offs involved
4. Each study - each set of results - must be interpreted in relation to other sets of evidence bearing on the same questions

The Impact of Technological Support on Groups: An Assessment of the Empirical Research

Pinsonneault, Alain and Kraemer, Kenneth L.

A collective analysis of the empirical results gathered from research studies on the effects of technological support systems on groups...

In theory, technological support facilitates group processes by enhancing group capabilities by:

• enhancing group capabilities
• removing barriers to group interaction
• improving the group in its task
• building, reinforcing social values of members through successful task performance

Develop a framework for analyzing impact of tech. supp. systems on groups.

• contextual variables
• which influence the group process
• which influence the task-related outcomes
• which influence the group-related outcomes

Gather research findings from a variety of impact studies and apply to this framework.
    => Recognized inconsistencies in the results, regarding group members' confidence in and satisfaction with decisions.
        => Resolved these inconsistencies by categorizing the studies according to the type of tech. support system used:

• GDSS (Group Decision Support Systems) = systems for structuring, supporting decision processes (of individual and/or group) through models
• choice models
• judgment refinements
• decision conference
• nominal group technique
• GCSS (Group Communication Support Systems) = systems for reducing communication barriers in groups, esp. if members are trying to do more than just communicating
• teleconferencing
• email
• local group networks

Given these categories of tech. support systems, how do their results compare?
 

 Discussion of these results: Why different results for GDSS vs. GCSS?

1. GCSS does not fulfill users' expectations of a technological support system => dissatisfaction w/group process => dissatisfaction w/decision => lower confidence in decision
2. GCSS offers perceived benefits to groups in early stage of development, while groups in a more advanced stage of development don't need the support and thus, don't perceive any benefits to using the system
3. Both GCSS and GDSS help focus on task, but since GCSS increases personally-oriented communication => lowers confidence of group members => decreases satisfaction w/decision => decreases satisfaction w/group process

?s:
    Authors suggest that, given that groups using GCSS experience decrese in confidence and satisfaction, the future of GCSS may be threatened.
    Do you think this is a reasonable conclusion?
    I wasn't under impression that GCSS's were designed specifically for decision-making, so I wasn't surprised that, when used for decision making, they exhibited weaknesses that the GDSS's, which were designed for that purpose, did not.  Given the example applications, I don't see GCSS's as being threatened at all.

Does "roomware" matter?

Streitz, Norbert A., Rexroth, Petra, and Holmer, Torsten.

Do CSCW products really help? Is more necessarily better?

Compare three meeting situations - same problem (Family TV channel), same number of participants (4), very similar room (OCEAN-Lab), same software (DOLPHIN)

1. networked workstations only (WS)
2. Liveboard only (LB) (well, pencil + paper, too)
3. workstations and Liveboard, all networked together (WS+LB)

Measured:

• overall quality (experts judged)
• quantitative measures (number of various actions, number of ideas generated)
• speaking time for each group member
• time spend in different cooperation modes
• basically boiled down to time spent working alone, in various subgroups, or as a group
• duration of individual work

Results - WS+LB beats WS, which beats LB, at:

• final document quality
• number of ideas generated (a lot of ideas generated on paper with LB never made it to final document)
• Do more ideas necessarily mean better final product?
• number of content operations
• amount of time group members spent working individually (in parallel)

Conclusions

• WS+LB better than LB or WS alone
• having networked individual workstations AND a shared Liveboard best
• more individual work, in parallel with subgroup and group work, leads to better final product

A few more notes on the subject of CSCW evaluation, collected from the in-class discussion...

Methods paper did a nice job of organizing the concepts in a useful fashion. Although the important ideas presented may appear common sense, they can be easily overlooked and go unrecognized by those of us who are still novices at accurately and effectively evaluating research studies.

How do we evaluate the impact of technological support used to facilitate groupwork? The second paper approached this by collectively analyzing empirical results of an array of research studies. Unfortunately, most of the studies used had collected their data using laboratory methods, which can be lacking in realism. We look forward to future research conducted in the form of field studies to invalidate or further validate the results described above. As far as the claim of success for GDSS's, could this, in part, simply be due to the structure they impose on the group process, and if so, couldn't this be easily achieved without the technology?

Common theme in today's papers: one way to evaluate the effectiveness of CSCW tools is to measure the quality of a group design or decision, both by the group members themselves and a third party. Question remains as to how to accurately measure this quality. Also, are the negative feelings described by groups who have used the tools to produce high-quality products/decisions mostly due to the newness of the technology?

Observation on possible bias injected into discussion of research results = is it fair to rely on studies of CSCW tools done by CSCW designers?

And finally, how can we evaluate the CSCW tools we used for this course? >From comments offered during class, at this point, although frustrations with the tools are plentiful, it does not appear that there exist fundamental problems with regard to using the technology in this fashion. A very large set of the frustration with the current state of the mash-tools lies in their lack of an adequate user-interface and the fact that, being that this is a research effort as opposed to a robust commercial-strength system, it is still undergoing development. Eventually, the class will write a paper discussing their collective experiences from this term. Also, more formal studies may be conducted after the interface has matured and some of the basic technological issues have been ironed out.