Project Presentations: Preparation, Evaluation, Ethical Concerns

Henry Jenkins III

The Presentations
The teams have 30 minutes to present their material, 10 minutes to dress the stage, and 10 minutes to respond to the judges' questions. We keep a tight time clock to insure fairness to all involved. The teams have to be well organized in order to get themselves into position by the start of the time and to stay on schedule so that all aspects of their presentation can be heard within the allotted time. We have certainly had teams who fell behind and the team leader had to make decisions on the fly about what should/could be cut in order to get the key ideas into the time. Ideas that don't get presented aren't considered by the judges even if the judges have heard them at earlier points in the process, since the judges have no way of knowing what changes might have been made in the last minute preparations for the presentations. No matter how many times you review the projects, there are always surprising elements in the final presentations as the projects only take their final shape the night before.

The order of the presentations is chosen by drawing slips of paper the day before. Everyone comes into the day knowing when their team will present and what the ground rules are for the presentation.

All participants in the competition have to be in place at the start of the first presentation and no further work can be done once the first presentation begins. All participants hear all presentations. The other teams are pushed out of the room momentarily when the judges ask their questions. Clearly, even with this precaution, later groups have some advantage since they can absorb information into their oral presentation, even if they don't adopt it into their visual materials. But, there have really never been any complaints about unfairness here.

As the competition has continued, the quality and elaborateness of presentations has grown each year. We find consistently that showmanship counts for a great deal in terms of the relative merits of presentations. Groups have become more theatrical. For example, it is very common for participants to create and wear costumes that help to establish the characters contained within their game. These costumes range from glitter rock garb for a Making the Band project to Purple Hats and capes for Willy Wonka. Very often, they will dress the stage with elements that evoke the atmosphere of the story world. For Spirited Away, they created the look of a Japanese bathhouse. Several times, we have had teams use puppets to communicate their ideas. One memorable presentation on Princess Smartypants included both a puppet imitating the regal voice of a princess and a man dressed up like a fairy. Given the technical skills of MIT students, it will not be surprising to learn that they build power point presentations, which can contain very elaborate mock-ups done in Photoshop or flash of what the actual game might look like. Sometimes, there are very elaborate computer animations that help communicate the core premise. A group focused on Survivor had animations of cracking fires on laptops arranged around the room. The 1984 team, which won last year, turned a video camera on the audience and projected live feed on the wall so the judges and audience had the uncomfortable experience of watching themselves watching the presentation. More and more students provide give-aways to the judges: one year, for example, a Matrix team provided a gold box with a red and blue sugar pill, asking us to decide which one we would take. Another group, working on Homer's The Odyssey, offered us bottles of root beer with a label designed to imitate the culture of ancient Greece. Some groups mimic the advertising materials or box designs for their products, even going so far as to give us blank CDs in jewel cases mocked up to look like the finished product. For a Fugitive project, each judge was given a 'wanted' poster with his or her own image on it. The groups frequently use music to communicate the audio design of their projects – in most cases, songs sampled from existing CDs, but in a few cases, original compositions the students created for the occasion.

The judges strive to separate out flash which is simply flash from elements carefully chosen to communicate the core elements of the concept and make it come alive for the audience. If there is no serious thinking behind the performance, it can upstage rather than compliment the other elements of the presentation. Throughout the contest, the students are ask to distill down the core features which most be present in order to allow their new product to fit coherently within the existing media franchise. Here is their chance to show you how fully they have achieved that goal. Yet, students need to be aware of the importance of underlying what aspects of the performance are in the game and why. Often, students will play interesting music throughout the presentation but not talk explicitly about sound design elements in their pitch.

We now ask the groups to identify a specific audience to which they are pitching the game. This element requires the groups to do their homework on the different companies which produce games, their connections with other media companies, their existing product lines and preferences, the core market to which they are addressing the bulk of their products, and any notable successes or failures which might impact how they think about your particular product. The group working on Spirited Away, for example, chose to address the Japanese branch of Sony, opening with an extended greeting in Japanese, and drawing on what they knew about protocol and business practices in that culture. Most of the information students need to know about the business can be found on the web. Students search under games that may be similar to what they are proposing and need to know how well they did and what markets they reached. They can find this by reviewing the large number of games-related websites produced by fans, industry insiders, or journalists. They may look up the property that they are drawing upon to conceive the game to see who produced it, how well it did at the box office, and what other connections may exist between the production company and a games-related company. They may look at other games linked to that studio's properties to see which games companies developed and released them. Those with more business interests can look for trends through industry-oriented publications, which might make the company more or less receptive to the kind of game proposed. When we first started the competition, these business aspects took a back seat to the more aesthetic dimensions of the presentations, but in recent years, we have attracted more and more management students interested in getting into creative industries. The teacher may need to assess her or his own students to know where their strengths and weaknesses lie and decide how much emphasis to place on this aspect of the presentations. Some focus on the industry, however, seems desirable as a way to help students understand the economic contexts in which creative decisions get made (including the impact of concentration of ownership on the current media environment); this focus on factual information also gives incentive for students to work on their research skills and especially their web searching skills, since this up to the minute information is not going to be found in books.

The goal of the pitch is to put together the key elements of the game in a way that is compelling for the audience. Everything students know about presentation skills comes into play during this part of the process. Students need to think about the ordering of elements – so that the information makes sense to the audience as they are hearing it, so that the most compelling elements get the proper degree of attention, and so that the closing drives home the core elements with an extra persuasiveness. The groups need to make sure that they address each of the elements they will be judged upon while not getting bogged down in local details to such a degree that the judges do not get a clear sense of the game as a whole. Frequently, students divide up the roles in their presentation based on their expertise and enthusiasm for different aspects of the product. Often, they also identify someone who is their best "finisher," that is, some one who has strong sales skills who can drive home the final pitch at the end of the presentation. We do not require every participant to speak, since we see the value of students who are uncomfortable in front of groups making contributions on other levels. This is, however, a philosophical question, which teachers need to consider in the context of their own programs. If the goal is to get students to really hone their presentation skills and get experience speaking in front of groups, the desire to have every student participate in the presentation is greater than if your goal is to focus on brainstorming and conceptual skills.

The presentations are a time for all team members to drop disagreements and work towards the common good. Often, team leaders instruct participants not to contradict each other during the presentation. If a team member gets hit with a question for which they are unprepared, he or she should make up an answer consistent with the overall approach they are taking and then subsequent team members will need to factor that answer into what they say about the game. One year, we had a group which totally failed to reach a consensus to the point that one group member disagreed about the order in which the PowerPoint slides should be presented and kept flicking back and forth throughout the presentation in a distracting fashion, upstaging and confusing his team mates. Needless to say, this group did badly in the competition and much anger was directed against him after they got off stage.

Judging The Presentations
We recommend having 5 or 6 judges, since each judge will see the presentations in slightly different ways and will focus on somewhat different elements. Having a broad based group of judges helps to take the pressure off any one participant and helps to even out any potential biases the judge may feel towards one group or participant. The judges are given a score sheet. We have established ten criteria of evaluation, which are known by the participants in advance. For each category, the judge gives a score from 1 to 10 with 10 being the highest score. Each judge keeps a running tally throughout the day and reserves the right to adjust scores upwards or downwards as they hear subsequent presentations. Judges often have to rethink their responses to the first few presentations, in particular, since they do not have a very clear baseline of expectations at the start of the day. Each group must be judged by every criterion, which means that they may lower their score significantly if they do not directly address one or another aspect. For some reason, students typically offer much more on game play, narrative, and visual design than they do on sound design.

The judges should set their own standards for each category. As long as they are consistent in applying those standards to each group, then their numbers will be correctly weighted in the final score. It doesn't matter if one judge is strict and another lenient since in the end, it is the total score across all of the judges which we use to weigh the teams. But those judges who are too generous in their grading need to realize that they are flattening out the distinctions between the teams.

In our process, we tally the scores to get a straw poll vote of where the judges have aligned themselves. The judges are often surprised by their own overall scores on the different projects. Subjectively, we will weigh some elements more than others in forming our overall impressions of the team. But if we weigh each element equally, we may find a team did better overall without being the best in any particular category. Once we have a set of scores, we then discuss the presentations to make sure we are fully comfortable with where the collective tally breaks. In some cases, we end up with two teams, which are only a few points apart in scoring, well within the margin of error, and we have occasionally had absolute ties on the first stage of the scoring. Talking through the presentations may help us appreciate things we missed the first time through or think more deeply about potential problems in the team's approach. We often will then recalibrate our scores and retally the results.

Another reason why you always discuss the projects among the judges is because ultimately, we see winning and losing the competition as less important than the critique we are able to provide to each team on its work. Talking it through helps us to identify high and low points on each presentation, which will be discussed as we critique their work. Often, we will give secondary awards which foreground what we see as the biggest strength of each presentation, so that each team has some kind of moral victory in the final presentation. So, for example, we may take each of the criteria and decide which team performed best on each, giving out recognition for best visual design, most faithful to the original content, or most innovative approach to narrative.

When the students come back into the room, we go through each group in order offering our critiques of their strengths and weaknesses. Obviously, one tries to be tactful in describing flaws in the presentations but we also feel it is important for all participants to have a sense of how the judges responded to each presentation. Doing the critiques first keeps students focused on what we are saying since they are looking for clues into which group may have pleased the judges the most. Usually, we have one judge designated to lead the critique of each group with the others jumping in as needed to elaborate on key points. It is important for the students to hear how a range of different judges with different types of expertise and backgrounds responded to their projects. Typically, we spend 5-10 minutes on each group; trying to be even in the amount of time we spend on each. At the end of the process, we then announce the superlatives assigned to each group and finally, the ultimate winner of the competition. In the MIT context, because of our association with Sony, each member of the winning team receives some moderately priced electronic gadget. All participants in the process receive a certificate and many years, Sony brings a book or video tied to one of their current releases for each participant. Teachers will need to think about what kind of prize is appropriate for their students – perhaps a gift certificate from a local merchant.

Ethical Concerns
There are two ethical questions you need to work through in thinking about the judging of the competition:
1. Should you post the breakdown of the final scores or for that matter, rank the teams in order? In the first few years of the competition, we posted every score on the blackboard. This had several negative consequences. First, the absolute loser of the competition felt enormously depressed and discouraged. Second, almost every team disputed some aspect of their score, leaving the competition in controversy. We have found that giving each team some recognition and then only announcing the winning team allows each team to feel better about their performance over all. It does mean that teams, which finished close behind the winner, do not get as much recognition as they deserve. We do feel that giving frank and constructive feedback through other means helps participants know what they need to do to improve on their performance in the future without having to reduce that feedback to some kind of numerical ranking.

2. Should you have a competition that has a winner (and thus, implicitly, losers)? This is a huge issue in American education right now. We have found that the competitive context intensifies participation and performance. Students make more of an effort in the project because they know that doing well comes with bragging rights. For our students, those bragging rights offset any other prize we can offer. It becomes known among the students who won the previous competition and some students come back several years running trying to improve their performance. Yet, there is also a strand in American education which sees competition as harmful, as disrupting the collaboration which occurs elsewhere in the learning process, and as potentially damaging some student's self esteem. These are decisions which teachers need to make for themselves in the context of their own learning environment. We have found that the benefits of competition have outweighed the negative aspects, especially since we stopped posting the actual scores.