THE EVALUATION

We conducted two phases of evaluation work around the exhibits. The first was formative, where we trained museum staff about prototyping and different techniques they could use as they put prototypes on the floor to gather data about visitors' experiences with the exhibits. These techniques included naturalistic observations, mediated interviews, and exit interviews. We also conducted two site visits to each institution, studying the prototypes after museum staff had worked with visitors and refined them; we conducted staff reviews and offered our critiques of the exhibits.

The second phase of the evaluation of the exhibitions involved a limited summative study. Originally, we had proposed to evaluate the exhibits for the final time after they had begun to travel and were set up in non-originating settings. We decided to modify our timeline so that we could see the exhibitions in their home museum; in this way, fine-tuning could take place before they were sent out. Our budget for the summative study was limited, so ultimately, we were unable to gather as full a set of data on each exhibition as we would have liked. We did, however, gather sufficient data to offer final feedback to the museums, and to make some general comments in this report about the overall quality of the exhibitions. As we stated earlier in this report, our summative study involved tracking visitors, timing them at each exhibit and in the exhibition as a whole, as well as conducting exit interviews as visitors left the exhibitions. In addition, we observed and interviewed visitors around "problematic" components that seemed to need additional work.

In general, we tried to be consistent in our data gathering from one location to the next. However, there were some circumstances that influenced the amount of data we gathered at each location. For example, the three-day visit to Montshire took place during an unusually busy time, which probably influenced the overall usage times of the exhibition. In contrast, the weekend we visited the Catawba Science Center was unusually slow, so we were unable to gather as much tracking data as we would have liked. Similarly, when we visited Ann Arbor, a blizzard hit, limiting the number of visitors we were able to work with. All of these factors need to be kept in mind when interpreting usage data. We conducted an average of 20 trackings and an average of 15 exit interviews at each museum. We include in the appendix a set of tracking data for each museum.

Overall Quality of the Exhibitions

In general, while the five exhibitions are not of uniform quality, they are good small exhibitions. Almost all of the museums were able to create better exhibitions through the collaborative than they would have without it, and the better exhibitions in the set of five are of very high quality. The more stable institutions were not hampered by the collaborative experience, while the institutions that dealt with staff turnover and other institutional issues were supported in creating better quality exhibitions than they might have been able to otherwise.

In the rest of this section of the report, we will discuss the quality of the exhibitions across several key dimensions. These dimension include:

• General Factors Influencing the Overall Quality of the Exhibitions - What are the generic factors that influence the quality of the exhibits produced?
• Context of Invitation - What is the overall context of invitation to visitors? Is it clear?
• Aesthetics and Creativity - What is the degree of creativity in the exhibition?
• Favorite Exhibits - Which exhibits among each of the exhibitions stand out as favorites?
• Group and Family Interactions - What is the degree of group and family interaction promoted by the exhibitions?
• Navigation and Holding Power - What is the holding power of the exhibitions and the individual exhibits?
• Conceptual Understandings - Do the exhibitions hold inherently interesting ideas? Do visitors connect and engage with the key conceptual content?
• Inquiry - Do the exhibitions address visitors' questions and encourage inquiry?
• Travelability and Maintenance - Are the exhibitions durable? Will they travel well? How much maintenance will they require while on the floor at the host museum?

General Factors Influencing the Overall Quality of the Exhibitions

One factor influencing the overall quality is that the exhibitions very much reflect the institutional cultures of the five museums: the values of those institutions' staff members, the way in which they work, staff turnover, and other institutional influences. For example, some institutions rely heavily on advisory boards for generating exhibit ideas and building prototypes while others do not. For some of the museums, developing these exhibitions and programs was their main project; for others, it was one of many.

It is also important to note that for most of these museums, this was their first foray into building traveling exhibitions. In fact, in several of the museums, the exhibit developers were newcomers to the museum field who hadn't built any exhibits before, let alone traveling exhibits. In addition, staff at all of the museums were trying to build exhibits that would enhance family interactions and family learning; thus, they were experimenting with new design ideas.

Another factor influencing the final products is the challenge museum staff face in designing traveling exhibitions. Museum staff are trying to design exhibitions that meet very specific space requirements, are creative, inquiry-based, give visitors positive experiences, communicate specific science concepts, and are easy to maintain, pack up and travel. They are also trying to design components that work together as an exhibition and have a similar graphic look. These challenges are magnified when staff working on the exhibits are those who are responsible for many other duties in addition to building exhibits and/or programs.

All of these factors influenced the final exhibitions that were created. For the most part, at the time of our final visits, each exhibition included at least two components that were not quite up to the level of the others in the exhibition. These were pieces that simply had not had the same level of prototyping that the other components had; staff had not had enough time to take them to the level of the other exhibits, or the exhibit ideas were more difficult and complex. Museum staff at all institutions were open to feedback from visitors and us about how to improve these components prior to their traveling.

Context of Invitation

For most of the exhibitions, the context of invitation to the overall exhibition was clear to visitors and engaging to them. By this we mean it was clear to the visitors what the exhibits were inviting them to do (e.g., solve a problem, explore, see a model, etc.). Usually, there was one exhibit that seemed to draw visitors in; often that exhibit was different for adults than for children. For example, in Ithaca, the Double Your Allowance exhibit tended to help invite visitors into the exhibition and funnel them toward the Superbowl exhibit. In Montshire, the adults tended to be attracted first to the Aeolian Landscape exhibit, while the kids tended to move more towards the Tube Tunnels exhibit. In Catawba, the Worm Farm and Mole Tunnel drew people into the exhibition. Thus, in general, people were invited in and were clear about what the exhibits were all about.

Aesthetics and Creativity

In developing their exhibitions, museum staff paid a great deal of attention to the graphics and overall look of the exhibitions, and their efforts were worthwhile. In addition, each of the museums engaged in a great deal of experimentation and creativity in producing their exhibits. Also, some of the museums worked with topics that have not been dealt with much in traveling exhibitions before, such as clothing, mathematics and dirt. They chose difficult subjects and handled them creatively.

For example, Sciencenter staff paid extra attention to improving their signage, hiring a graphics design firm to help them with the layout and production. They simplified the wording on the signs. The net result is a set of colorful labels that are easy to read and follow, and that visitors appreciate. As several visitors noted:

[The labels] are nice, an improvement over what they have had in the past.

The signs are nice. This really grabbed me that it was a cohesive exhibit and the labels helped with that.

Catawba experimented with copper laminate for some of the exhibits in their exhibition; rather than trying to design a surface for dirt exhibits that would resist scratching, museum staff incorporated materials where the scratching would ultimately enhance the look of the exhibits.

Ann Arbor experimented with a kiosk design that would allow them to place related exhibits in a conceptual "hub." This also allowed them to run the power for three or more exhibits through a central location.

Discovery Center staff experimented with designing around key big ideas. They created cornerstone, more open-ended exhibits around Newton's Laws, then built smaller exhibits that focused in on one or more key concepts, such as friction.

Montshire focused on trying to design an exhibition in which each of the components was as open-ended and inquiry-based as possible.

These differences in style reflected, to some extent, the differences in institutional culture and design philosophy. Nonetheless, with some exceptions, the overall aesthetic qualities of the exhibitions are good.

Favorite Exhibits

For each exhibition, there were clearly "favorite" exhibits. For adults, these tended to be exhibits that clearly demonstrated a concept, or were aesthetically pleasing, or provided a lot of "hands-on" activities for their children. Children seemed to like things that were surprising or that they could do again and again.

For example, in Ann Arbor, the exhibits about thermal insulation of gloves, weaving, and the Velcro under the microscope were mentioned by visitors as most popular.

The contrast is really striking at the cold plate.

My son liked the ones with the gloves [Gotta Hand it To You] because there was something more interactive to do there.

In Catawba, the exhibits with the pumps - Percolation Station, Erosion and Filtration Station - were frequently mentioned by visitors as their favorites. Children also tended to mention the Mole Tunnel and the Compost Zoo as their favorites.

I liked the erosion one because I liked using the pump and watching the water.

I liked the mole hole with the light because it shows a real stuffed mole - that was neat!

At the Discovery Center in Rockford, tracking data and observation showed visitors gravitating towards the Momentum Machine and the K'Nex display. They also liked the Bumper Blaster and found the Bump-o-Rama exhibit fun for a short time.

I liked K'nex, and the circular cars... [Bumper Blaster]. I like to see how fast, and how much resistance [it takes]. It's a challenge.

Centrifugal [centripetal] force - always amazes me - like ice skaters, you go faster towards the center.

At the Sciencenter, the Measurement Factory, Superbowl, and How Many is a Million exhibits were mentioned by visitors as ones they liked the best. Some adults noticed the posters scattered throughout the exhibition on math in different countries and thought they added to the exhibition as well.

My son liked the Superbowl one... He likes to learn things that have to do with himself, so he liked the height (histogram) and the Measurement Factory too.

I like Allowance and Bowling. Allowance illustrates an interesting concept; Bowling is just fun.

At Montshire, favorite exhibits were the Ball Floaters, Aeolian Landscape, and Air Maze exhibits. The Air Cannon also got mentioned by several visitors as their favorite.

I liked the air cannon because it feels like a real ball of air coming at you - such a surprise.

The ball floaters - I like the way the balls go flying in the air.

Group/Family Interactions

In addition, staff made a good effort to design for family interactions. Most components were designed with an eye toward being able to be used by more than one person. In fact, trackings showed high group and family interaction at certain exhibits, especially those that offered young children and families a task they could complete together; examples include Measurement Factory at Sciencenter, and Gotta Hand it To You at Ann Arbor.

For example, the Air Maze exhibit in Montshire's AirPlay exhibition can be successfully used by one visitor, but is ideal for encouraging family interactions. Air outlets are placed around the entire exhibit, and while some visitors drop balls into the start position, other visitors open and close different outlets to direct the ball through the maze. Visitors end up working in groups to problem solve which outlets to open at which time, to direct the ball through certain paths.

Exhibits located near benches also promoted family interaction as parents and grandparents rested and "played" with children around the exhibit. At Ithaca, Measurement Factory and Double the Doggie worked well together for these reasons, while in Rockford, adults rested near the K'nex center while children played.

Some museum staff designed exhibits to be especially appealing to very young children. At Rockford, K'nex and Build a Coaster were both popular with pre-Kindergarten through grade three while at Montshire, Tube Tunnels and Sailboats proved inviting to young children. At Catawba, the Mole Tunnel was a magnet for young children, while Ann Arbor staff designed a special pre-school area, with dolls and fish to velcro to specific backgrounds, and dolls with all kinds of fasteners to open and close.

However, not all of the experiments in the area of family interactions were successful. A few exhibits suffered because too much emphasis was placed on designing for group or family interactions. Some ideas and concepts seemed to work better in single interaction types of exhibits; some of the meaningfulness of the experience and ideas would get lost in the attempt to make the components more family friendly.

Navigation and Holding Power

In our trackings, we looked for patterns of movement that might indicate that some exhibits were getting skipped, and for general usage times for each component and the exhibition as a whole. We did not notice significant differences in the attractiveness or holding power of any of the exhibitions across gender, age or ethnicity.

Patterns Across All Five Exhibitions

In terms of holding power, across all five exhibitions, visitors spent an average of just under ten minutes in the exhibitions. We noted a great deal of back and forth movement on the part of visitors between the TEAMS exhibitions and other exhibits in the museums. In many cases, visitors we tracked that spent very little time in the exhibition invariably returned for longer times later in the day. It is also important to note that the relatively small number of trackings at each museum (13 to 36) precludes any sort of statistical validity; however, combined with observation and exit interviews, some trends do emerge.

For example, location of exhibits appeared to play a role in their attraction. In the case of Ann Arbor, visitors regularly migrated out of the Clothing exhibition to use adjacent exhibits (where they spent an average of five minutes) and then sometimes returned to the clothing exhibit. At Catawba, visitors often moved back and forth between the dirt exhibition and a Raceways exhibition sharing the space. In Ithaca, visitors often missed exhibits placed in a large alcove type area near the stairs and ignored two sets of Math Posters but used a third.

Also, in Ithaca, three of the most popular exhibits were located fairly close to one another. This caused visitors to spend most of their time on one side of the exhibition, skipping components on the other side. We recommend that staff at other museums spread those components out so that traffic flow for the exhibition as a whole could be more evenly distributed.

Having exhibits that were similar also influenced visitor usage. In Catawba's Dirt exhibition, there are several microscope components that involve the same activity but looking at different samples. Visitors tended to use one microscope component but not the other, or if they used both, would spend a significantly shorter amount of time at the second microscope.

Finally, ease of movement through crowded areas also affected use. During two very busy days at Montshire, the ability of parents to move strollers through the exhibition and find "parking spaces" affected which exhibits they gravitated to.

These phenomena suggest individual museums should conduct careful observation of traffic flow and use patterns after exhibits are placed in a non-originating museum, and adjust the foot print to the unique characteristic of each venue.

Tracking Data for Individual Exhibitions
(Raw data)

In looking at the data for individual exhibitions, we did notice some differences in holding power among the individual exhibits that make up those exhibitions. Each exhibition had a small number of exhibits with exceptional holding power, a group with average but not exceptional holding power and others that were either overlooked or apparently not interesting to visitors.

Ann Arbor's Clothing: Science from Head to Toe: The average visitor spent nine minutes in the exhibition (the range was from just under four minutes to almost 27 minutes; 13 visitor groups tracked). However, visitors skipped many exhibits. Six of 19 exhibits - Gotta Hand it To You, Windbreakers, Fencer, Be-weave it or Not, All Charged Up and Keep Out the Cold - had holding power of around one minute among visitors who used them. While some exhibits had good holding power when used, very few people used them.

Magni-fiber, All Charged Up, Hook and Loop, Reflection Glasses and the Reflection Room held visitors for an average of 30 seconds. The Preschool area showed mixed use of almost one minute in one case, 40 seconds in a second and less than ten seconds in a third. The remaining seven exhibits, over one third of the exhibition, had usage times ranging from near zero to 10 or 15 seconds. During our trackings only one visitor used From Ewe to You.

Catawba's Dirt: the Story of Soil The average visitor spent nine minutes in the exhibition (the range was from one minute to more than 23 minutes; 15 visitor groups tracked). Compu-Dirt and Compost Zoo held visitors for two minutes. Nine of Catawba's nineteen exhibits almost or did exceed one minute hold time. These exhibits included Tunnel of Dirt, Erosion, Filtration Station, Strata Boxes, Separation Station, Worms, Dirt USA, Touch Station, and Percolation Station. Sedimentation Station, Microscope Station One and Hydrangea held visitors for at least 30 seconds. The entry kiosks and second microscope station were used by few visitors for short periods of time.

Discovery Center's Amusement Park Science: The average Rockford visitor spent eight and a half minutes using the 10 exhibits in the exhibition (the range was from two minutes to 30 minutes; 22 visitor groups tracked). It was not unusual to see visitors, especially young children and their parents, spend 15 minutes in the K'nex Work Area and three to five minutes in Create a Coaster. Older children and their families spent a minute and a half or more at Bumper Blaster, Rotor and Momentum Machine. Magnetic Circus held the few visitors that used it, although in exit interviews, visitors reported frustration; thus, longer usage times here might indicate a negative experience. Visitors used K'nex Display, Bump-O-Rama, Video Kiosk, and Wild Wheels Roadblock for thirty to sixty seconds.

Montshire's AirPlay: Visitors averaged just under ten minutes using Montshire's 11 exhibits (the range was from two minutes to 22 minutes; 30 visitor groups tracked), and there were several exhibits with dwell times over two minutes. Five of the exhibits - Tube Tunnels, Air Maze, Air Race, Ball Floaters and Sailboats - averaged two minutes or more per user with many individual users spending more time at the exhibits or returning to exhibits multiple times. Aeolian Landscape, Fluid Flow, Air Cannon, and Air In/Air Out averaged between one and two minute's use. Hand Crank and Fabric averaged 30 seconds or more per visitor.

Sciencenter's Fun 2, 3, 4 - All About a Number of Things: Sciencenter visitors spent an average of over 12 minutes in the exhibition (the range was from two minutes to 39 minutes; 20 visitor groups tracked). Of Sciencenter's 19 exhibits, visitors used six for over two minutes, including Double the Doggie, How Many is a Million, Aztec Rubbings and Superbowl. Measurement Factory had an average use of almost seven minutes with some family groups spending twenty minutes. Only a few visitors used Math Bloopers, although the adults who did use it stayed over two minutes.

Two sets of Math of the World Posters were glanced at but skipped by most visitors. One set in a more central location averaged over one minute use by visitors who used it. Double your Allowance, Cool Curves, Age vs. Height Histogram, How Many Hands is a Horse, Time Challenge, Thick or Thin, and Weigh Out showed average use of one to two minutes. Cool Curves and Dino were used on average for less than one minute.

Conceptual Understandings

In conducting exit interviews with visitors as they left the exhibition, and mediated interviews with visitors around key components, we were able to get a sense of the meanings visitors were making from their experiences with the exhibitions. In general, visitors were able to articulate some general understandings related to key concepts in the exhibitions. For the most part, these key concepts were ones that exhibit designers had intended for visitors to "come away with" from the exhibition. In only a few cases did significant misconceptions about the nature of the exhibits and exhibition emerge. In those cases, we talked with exhibit developers about how best to modify the exhibits in order to help visitors come in contact with the main ideas more easily.

Below is a sample of visitor comments regarding main ideas and experiences with each of the exhibitions:

Ann Arbor's Clothing - Science from Head to Toe:

Cloth is made of weaves. What clothing is made of and the differences between synthetics and wool.

Clothing and what it is made of.

It is about how they [soils] work; how they help stuff; which works the best.

How water goes through some kinds of dirt better than others.

There's different soils under the ground. Different parts of the country have different soils. There are different worms and how they help us.

It's about motion.

It's about moving, rolling, hitting and blasting around.

It's about basic physical properties, motion.

It's a hands-on understanding of physics as it relates to amusement parks.

Montshire's AirPlay:

[The exhibits are] all about air flow... how air operates, how air moves things.

Even though you can't see air, it still has energy, it still has force.

[The exhibits are about] air, movement and energy.

Sciencenter's Fun 2, 3, 4 - All About a Number of Things:

Visual patterning, weighing, scale, making sense - bring parts together.

Different uses of numbers; different things people do with numbers.

Measuring, ones/tens/hundreds - counting and place values; estimating.

Inquiry

Because of the nature of the topics covered in the exhibitions, some of them were more inquiry-based than others. However, most of the exhibitions had at least two to three components that were very open-ended. For example, the Dirt exhibition from Catawba includes a stream table, where visitors can experiment with different water and soil patterns in an open-ended fashion. The math exhibition from Sciencenter includes a Superbowl exhibit where visitors can make a variety of different types of patterns by bowling tennis balls down and lighting up lights.

Montshire's AirPlay exhibition is perhaps the most inquiry-based of all of the exhibitions. Most of its components are very open-ended, allowing visitors to experiment and use them in a variety of ways.

In fact, visitor trackings suggest that certain open-ended, inquiry-based exhibits tended to be used for longer periods of time. Montshire's Tube Tunnels, Ball Floaters and Sailboats as well as Sciencenter's How Many is a Million were all popular exhibits that had fairly high usage times.

Travelability and Maintenance

Since we conducted our summative site visits at the originating museum, we can only hypothesize about how well the exhibitions will travel, and how easy the exhibitions will be to maintain at the other venues. There are a couple of components in two of the exhibitions that are fairly large, but the collaborative members have discussed these components in great detail and worked out compromises where necessary.

In terms of the maintenance, most of the exhibitions look as though they will be fairly easy to maintain. The Dirt exhibition obviously is going to require more day-to-day maintenance, because of several components that have water, and the sieve station that is a bit messy. Again, collaborative members have talked through these issues and should not be surprised when the exhibitions arrive at their door.