by Jerome Bump

            “The argument for a pedagogy which embraces visual and multimodal representation is well established in academic circles (Kress and Van Leeuwen, 1996; New London Group, 1996; Cope and Kalantzis, 2000) and a plethora of literacies congregate around the ever-expanding subject English as the prime site for innovation and development” (Matthewman 2004, 153).

            One of the primary advantages of this multimodal pedagogy is that it “values the lives and experiences that learners bring to English” classes (Albers 2006, p. 76), the first requirement of an inquiry-based curriculum. Consider the experience of one of my grandsons. He is now fourteen-years old, six-foot-two, two-hundred and ninety pounds, and a defensive lineman on his football team. Yet, if given a writing assignment, he cries. On the other hand, he loves Myspace and video games, especially one that enables him to build different versions of automobiles. For his sake, and for millions of others, perhaps we can use multimedia and video games to salvage aspects of print literacy.

            Hence I am grateful for the efforts of all the researchers exploring the educational value of multimedia (e. g. Kress 1996, 2001, 2003) and video games (e. g. Gee 2003, 2005) for grades K-12. But what happens when students like my grandson go on to college and face writing assignments there? In many universities “what happens” occurs in a freshman composition course taught by a graduate student, and thus is out of sight, and out of mind, of the regular faculty. However, “freshman seminars” taught by tenure-track faculty now often introduce students to the university. At our university such seminars are “Substantial Writing Component” courses, which satisfy a basic composition requirement. In one such course I witnessed “what happens.” I took the freshmen to a creek on campus and asked them to write on site and turn in their writing at the end of class to me. As I checked out the various positions along the creek students had taken, I noticed that two students had disappeared, one of whom bore a striking resemblance to my grandson. I recalled them heading down the creek at the beginning of class: “what happened” was that they just kept going rather than do in-class writing.

            Fortunately, I discovered that, if given more time, and presumably help outside of class, they could produce a writing assignment. They also took advantage of the option of doing their other writing assignments in html format on the web, with images. Both of the students who hated in-class writing did so; one in fact did his entire portfolio in html format. This should come as no surprise “for most people and especially for younger people, the exclusion of readily available images from a text seems unnatural” (Faigley 2003).[2] I suspect most of the students in that class would also have benefited from adaptation of the K-12 research on educational games to required composition classes in college.

            We tested that hypothesis on a two-semester, required freshman English course[3] for honors students at a large state university. We chose this class partly because we knew these students would not head down the creek, but would do whatever we asked them to do for two semesters. Admittedly, they were more verbal than the freshman seminar students, but in a survey that featured the question, “Do you feel more skilled about technology skills than other students your age,” 93% answered “yes." The survey also reported that 70% liked playing video games; 86% felt confident playing virtual world games such as Second Life;[4] and 76% liked the use of games in class. (These surveys, along with interviews and other evaluations were administered by a team which had been assigned to the project by the university’s Division of Instructional Innovation and Assessment team.[5])         

            These responses enabled us to consider the relevance for college English of the following facts: (1) “videogames are a push technology, providing people entrée into other important technologies, such as computers”; (2) “the online affinity groups that emerge around games function as a kind of push community, engaging members in identities, values, and practices, markedly similar to the intellectual and social practices that characterize high level, conceptual communities of innovation in fields such as science, technology, and engineering”; and (3) “massively multiplayer online games (MMOGs) are the quintessential example of such communities” (Steinkuehler p. 4).     

            Many MMOGs are narratives, the most famous being World of Warcraft, and these are the focus for much of the K-12 research. However, Edward Miller, senior researcher at the Alliance for Childhood, has criticized K-12 educational games because they fail "to teach higher-order thinking and encourage creativity and imagination in the classroom" (“Gaming” 2007). These skills and qualities are especially important at the college level, of course. So we turned to what are called “sandbox” MMOGs. Instead of providing a narrative like most of the other video games, a sandbox game invites members to participate in the actual construction of the virtual world, creating their own objects as well as their own avatars. The “objects” can be “bots” and scripted actions, as well as buildings, rooms, landscapes, sculptures, paintings, etc.

The sandbox game which has received the most publicity is Second Life (hereafter represented as SL). It has even been identified as the fulfillment of Stephenson’s prophecies in Snow Crash.[6] How

            So the next step seemed to be exploring the more innovative possibilities of writing in SL itself.      Creating a place for our class would also help us test the idea of building a virtual campus of our university in SL. Such a campus would help the students "create a sense of place" for their university experience, as recommended by the Carnegie report,[7] and, ultimately, a complete virtual campus could be used for student recruitment, orientation, and retention as well as tourist and alumni interaction with the university. Hence, during the summer, Alex Games, of Ed. Psych., built in SL the main building of our university. One’s avatar could not actually enter the building, but the exterior was constructed in painstaking detail:

the Main Building in SL

He also created a version of the campus creek, with trees, and added a Greek ampitheater for debates.

   In the fall, the students were given their first assignment: to “write” their “road maps” in this world. The “road map” is a visual-verbal-musical presentation of the most important places in the student’s life. In freshman courses it is also a relatively painless transition from one kind of high school “writing” to college “writing.” Because it can be as visual and musical as the student desires it is considered “fun,” rather than work. Even the weakest students are able to make impressive Power Point presentations. Indeed, it is so much fun that students often spend inordinate amounts of time on it and produce extraordinarily creative and effective autobiographical works. All the students then make brief in-class presentations of their road maps and thus get to know each other.[8]

            The challenge in this class was to put the road map in SL somehow. One of the most obvious ways was to convert one’s Power Point presentation into a website and then somehow link that website to a location in the virtual world. It is not difficult to make a web version of a Power Point presentation, though some options are obviated by simply using that command in the Power Point program. How to put the link in SL was the real problem. Alex Games went to the main island of SL and found a “webloader” script that provided one solution. The students discovered that the easiest way to put multimodal text into SL was to embed this “webloader” script in an object. When a user discovered and then selected such a sign in the virtual world, an internet browser screen opened with the usual 2-D mix of multimedia and text enlarging the message of the sign. For most students the SL object was thus essentially a billboard that presented the first picture in a series.

            This option was particularly interesting to us because it invited comparison with a previous research program involving a MOO.[9] Use of MUVEs in English courses was fairly common when they were primarily text-based MOOs.[10] Eventually, the EnCore MOO, out of the University of Dallas, allowed access to the web and thus to internet multimedia. Alex Games and I had tested this aspect of the Encore MOO, especially the claim that it is “architextural” (Haynes, 1998, p. 4).[11] The goal was to give ordinary students a sense, however primitive, of what it was like to experience study abroad. Because the English department had a summer program at Oxford, students were asked to use the “room” and “bot” options in the MOO to recreate Oxford and its people. (Eventually, the focus shifted to adding a similar version of our own campus.) The project was fairly successful in terms of creating architectural relationships and the personalities of the campus genii locii but hampered by limited access to and representation of 2D internet multimedia in the MOO. Eventually, fear that the open-source MOO software could be penetrated by hackers put an end to further research.

            SL provided many opportunities to go beyond the multimedia potential of the Encore MOO. For example, one student, Brad Barry, actually made a path in SL from one picture to the next so that one’s avatar had to walk through his life. Interactivity was especially obvious when one’s avatar had to walk across Waller Creek on a narrow board. Selecting a picture then loaded its website with text and more pictures.[12]

Another student, Mauro Caffarelli, cleverly used an SL note-writing script to embed texts about his life in four 3-D objects. First there was a billboard on Waller Creek with the text: “As you walk down the river towards the horizon, you shall find four distinct objects that symbolize four great aspects of my life: religion, academics, athleticism , and music. Within each of the objects are two or three notes with concise narratives explaining how I have evolved in the four subjects over the course of my life. Some of them start from when I was of a young age, other begin at a time only three years ago. In order to access the notes, right-click on the object….”

This was an extraordinary accomplishment. First of all, Mauro sculpted four very difficult objects, a very complex, time-consuming process in SL. Secondly, he had actually embedded text in the objects themselves. No other student had been able to do either task, because of inadequate directions in SL and the absence of an expert to show them the way. Like Brad, Mauro also incorporated interactive motions of the “reader” who had to learn to walk or fly along the creek (or walk in its water) to find the objects, as well as “select” them to read the text. In a sense, the reader had to “walk the talk” to briefly experience Brad’s and Mauro’s lives. Sculpting the objects required not only creativity and imagination, but also higher-order thinking, including knowledge of geometry. Mauro’s road map obviously required much more time than was allotted for the assignment, but it was “fun,” both to “write” and to “read.”

            Once the students got to know each other through their road maps, we could turn to more traditional academic concerns. A primary goal of the course in the second semester was to facilitate reading and writing about leadership, compassion, and the sympathetic imagination (the ability to put oneself in someone else's shoes). So we took advantage of the most successful feature of SL, the social interaction. Following a writing project on role models,[13] the students redesigned their personal avatars in SL to look like their role models.[14] The range of avatars. seen below, was a testament to the diversity goals of the university and connected the concept of the sympathetic imagination to the goal of multicultural understanding.

Students made avatars of their role models

They were then assigned four different locations on our island to discuss the relationship between leadership and three other topics: compassion, diversity, and history.[15] This aspect of was very popular because the most “fun” activity in SL is flying through the air over the island, something most of us have always wanted to do.

avatar interaction at the “office”

two of the other locations: the campfire and the rug

They flew or teleported from one location to the other and saved the text[16] of their discussions which they were then encouraged to analyze for the best statements on those topics.

            This exercise accomplished many of our goals. One of the primary goals of educational games such as this is to get students "in the flow" and in this case 84% did find it "engaging"; 72% felt that time seemed to fly by during the activity; and 57% found that their attention was "entirely focused on the tasks to be completed." (This recalls Matthewman’s sense of the “pupils working with high levels of engagement,” p. 166). More importantly, 67% agreed that "it was a good learning experience for this class" and 50% thought that their "understanding of the sympathetic imagination increased because of the activity." Of course, some of these goals could have been met to some extent by face-to-face and internet chat discussion. Hence it is significant that 61% thought that "chatting with others while seeing my and others' avatars facilitated discussions better than text-based chats (without visuals)," while only 33% argued that "the activity would have been better if we had done it in classroom in person."

            Clearly this use of a multiplayer online game was successful at the college level. This result is congruent with the research of Schroeder (1997) and others who discovered that “the combined effect of using text, navigating a virtual environment and engaging with other users via avatars is greater than the sum of its parts,” resulting in a striking “feeling of immersion in a virtual environment” (Bromage (2004), p. 134). Schroeder (2002) later used the term “telepresence” for this feeling and “copresence” for that essential “sense of being together” in that world (Peterson 2006, p. 81).[17]

            On the other hand, only 45% agreed that "chatting in Second Life seemed natural during the activity."  The basic problem, as described by one student, is the one that text-based chats have faced since they were invented: "I felt the chat was much too cluttered to really get any cohesive discussion going. I couldn't tell when someone else was attempting to reply to a statement and would often accidentally make comments much after the discussion had moved on because I would be typing while others were already discussing something else." Peterson, experimenting in Tokyo with avatars in ActiveWorlds, also noticed that “the messages in the text box are intermixed and strict turn adjacency is frequently interrupted” (2006, p. 82). This problem was exacerbated in SL by the extremely small space devoted to the text interchange itself (symbolic of the role of words in the new literacies?). That can be increased, as in iChat, but would be at the expense of the multimedia. Perhaps the speech bubbles above the avatars’ heads in ActiveWorlds suggest a useful compromise. However, the basic problem remains, one that was encountered in the 1980's at our university when Paul Taylor, a graduate student in English, invented the Daedalus program, Interchange, to conduct class discussion on a local area network .[18] The result then as now was a truly egalitarian, student-centered forum which supported relatively democratic discussion by all concerned of the goals and methods as well as the subject of the course. Unfortunately, what was discovered then about the disadvantages of reliance on keyboarding, the loss of voice communication, the slow speed compared to speaking, the loss of coherence, the absence of a controlling instructor, and technostress is still relevant today (see Bump, 1990).      

            Nevertheless, that assignment was “fun.” The students’ first major “writing” project in SL had not been. During the first semester they were to write their own versions of the campus architecture master plan and construct in SL models of the kinds of buildings they wanted to see on campus. Advance planning is obviously crucial. In this case, the preparation seemed to be fairly thorough, but a month before the course was to begin the SL building expert left the university and our most challenging assignment (increasing multimodal literacy by simultaneously composing essays and buildings in SL) was in jeopardy. At first, in the absence of our building expert, constructing true 3-D buildings that one could walk into and “inhabit” in SL seemed impossible. However, one student, Elizabeth Wong, stayed up all night and was able to construct such a building.

the breakthrough

The other students gained confidence and went on to produce some extraordinary buildings of their own:

the result: a virtual campus

Then their task was to integrate their verbal arguments for their preferred architectural styles with the virtual buildings which exemplified the styles. In other words, they were to somehow fix their arguments onto or into the buildings themselves. As Mauro had shown, in the SL virtual campus the most visual but in some ways most difficult way to integrate text and architecture was to embed the text itself in part of a building. Some students were able to do this, but the results were usually not visible to others because of the complex SL layers of permissions.

            Alex Games had provided a third option, bringing in from the main SL site “Thincbooks,” simulations of books, with covers, pages that turned, etc. They thus almost literally resurrected print literacy in the new virtual world. Though the atavistic Thincbooks were difficult to master, some students were able to put their entire multimedia projects in the books. In other words, the 2-D mix of words and images they would have put on the internet was now inserted into the pages of these virtual books, which were then put on or near their buildings. The “reader” then was able to “turn” the pages of the books and “read” the project and get an overview of it before exploring further. Often the “writer” then provided more embedded texts in the buildings. This process of constructing buildings and embedding texts all took an extraordinary amount of time and I doubt these students would have succeeded if they had not been a special class of very advanced, extremely competitive students.

            The project was clearly successful in three respects: 250% of the students agreed that their “awareness of campus architecture has increased because of SL.” They had created “a sense of place" for their university experience, as recommended by the Carnegie report.[19] Secondly, 77% found that their “sense of U.T. as my alma mater increased because of SL.” Thirdly, over 80% felt that a virtual campus of their university in SL would be “a good recruiting tool”; “good for freshman orientation”; and “good for retaining almuni interest.”

            The relation of SL to multimodal writing was more complicated. The interviews conducted during the first semester by Michael Mayrath revealed that the goal of multimodal literacy had been achieved by some of the students. For example:

M: When I was writing the paper about the architecture he had us get snapshots from SL and talk about what we had built. It let me talk from the inside about architecture. Because I could talk about the things I built rather than the things other people have done.e description of the class he said we will be looking at world lit. as the world around us rather than actual books.[20] So learning about iconography has made me aware of stuff; then building in SL where you’re actually creating a world with it’s own symbols and stuff.

            Obviously, students' perceptions and understandings of their own visual and verbal literacies do not always match the teacher’s, again recalling the research of Matthewman et al in England. When they asked their high school students to convert a Power Point story into a written text the teachers all agreed “that the translation from multimodal story into written language had proved very problematic” (2004, p. 157), and they agreed with Andrews’ conclusion, in his overview of prior research, that the result was “two quite separate sets of creative activity rather than … a liberating interaction” (2001, pp. 125-6).

            On the other hand, we need to consider the fact that the survey was administered toward the end of the semester when the students had become burned out on SL. At the end of the first semester, 53% revealed that they did not “really” enjoy the experience of building in SL and 65% were “glad that we are using SL less next semester.”.When the evaluators came at the end of the semester to do focus groups on SL in class the students responded on note cards which were then assembled and discussed. The students emphasized how much their creativity was stimulated:

the creativity

But they also vented their frustration:

the frustration

           There were various causes of this aggravation, many related to maintenance and security problems in SL. SL was a rapidly growing site with hundreds of thousands of users, and permissions and other matters were difficult to resolve at a distance. It was often down for maintenance and even  when it wasn't, almost every time the students logged onto the program, they had to download and install a new desktop client. One time it was shut down completely for days to deal with a security breach and a change of passwords. This agrees with Matthewman’s research: “I had already collected and analysed data which showed frequent instances of English and technology clashing uncomfortably . . . . The main themes were: technological hitches, lack of technical support,…. tension between the need for coverage of curriculum content against the time taken up by technology, as well as the time take by pupils in their exploratory and often time-consuming uses of technology” (p. 158).

            However, the basic problem may be a gulf between two cultures. While the students in the class liked video games and were good at them, and some were adept at the visual arts, all were highly verbal. When they found themselves alone in SL, they found a world of very few words. Indeed, at times it seemed to me that SL was not only basically nonverbal, but even antiverbal. After all, the basic method of learning in the computer world seems to be trial and error, on your own. If you get effective help, it is usually someone showing you, by example, how to do a task, not putting the process into words. Of course it may well be that computers and especially video games attract more nonverbal customers than other activities. And this may be only an extrapolation of a strong tendency to do rather than to read. For example, how many of us, when given the task of assembling something we have purchased, really take the time first to read the directions carefully? How many of us just skim them or skip them and plunge instead into the assembly process as fast we can, reading directions carefully only as a last resort, when all else fails? Without good clear directions, without someone to show them how to build in SL, that approach did not work well for the students in this English class.

           “’One might say the following with some confidence. Language-as-speech will remain the major mode of communication; language-as-writing will be increasingly displaced by image in many domains of public communication, though writing will remain the preferred mode of the political and cultural elites.’ (Kress, 2003, p. 1)” (Matthewman p. 172). While these very verbal, elite students were no doubt aware that the visual was replacing the verbal in public communication, they were also probably aware that writing was still the key to success in the elite class to which they aspired.

            Nevertheless, out of that class come the leaders of this nation who have primary responsibility for our democracy. This expectation is very strong at a state school. The first President of the Republic of Texas, Sam Houston, clearly stated: “"The benefits of education and of useful knowledge, generally diffused through a community, are essential to the preservation of a free government."[21] Mirabeau B. Lamar, second President of the Republic of Texas, simply said, "The cultivated mind is the guardian genius of democracy….” The seal of the university features a Latin version of this statement:

When these leaders take an English composition class, should it not prepare them for the multimedia electronic world of the twenty-first century? Should they not be able to communicate in the increasing visual, digital language of the majority of the people? “We have no justification aside from disciplinary baggage to restrict our conception of rhetoric to words alone. More important, this expansion is necessary if we are to make good on our claims of preparing students to engage in public discourse” (Faigley 2003, p. 187).

            No doubt most of these particular students will be able to master both print and digital literacy. But what about the usual English courses? In my experience, most English lit. majors do not want to do their assigned writing in html pages, much less more advanced multimedia. In fact, many of these students strenuously resist the transition from paper to electronic page. Yet, some time ago web-page writing became one of the skills that young professionals need in their work (Lankshear & Knobel 2006). Clearly, in college, at the upper-division level especially, in my experience, English lit. is not an “ever-expanding subject,” nor is it “the prime site for innovation and development” of “a plethora of literacies.” We must face the question, should English remain focused almost exclusively on the elite? We can’t keep pretending that most English majors are going on to graduate school in English. Are they to be sent into the world as functional illiterates in the digital world? Should they not be prepared to get jobs in the twenty-first century?

            What about lower-division composition courses? There is a “growing percentage of students who believe that their ability to communicate using new media will be critical to their futures” (Faigley 2003, 179). Do English departments have no role to play in teaching digital communication skills to the rest of the population? If multimodal literacies are not incorporated into the required English composition courses what happens to the average college student? What happens to students like the two who headed down the creek? Are they to be sent into the world as functional print illiterates?       Hopefully, some day such students will benefit from what we learned from the honors students’ experiments in salvaging print literacy in SL, and designers and researchers will be able adapt features of educational games that not only encourage creativity, imagination and higher-order thinking in the college classroom, but do so in multimodal formats that integrate the verbal and the visual. As Richard Lanham said in 1993, "we can neither preserve the educational system unchanged nor throw out the 'literate' ways of thinking. We have, in some way, to move the humanities from the old to the new operating system"( 264).          

REFERENCES

Albers, Peggy (2006), “Imagining the Possibilities in Multimodal Curriculum Design.” English Education 38(1): 75-101.

Andrews, Richard (2001), Teaching and Learning English: a Guide to Recent Research and its Applications. London: Continuum.

Boyer Commission on Educating Undergraduates in the Research University, 1998, “Reinventing Undergraduate Education: A Blueprint for America's Research Universities.” http://naples.cc.sunysb.edu/Pres/boyer.nsf/

Bromage, Adrian (2004), “Atavistic Avatars: Ontology, Education and ‘Virtual Worlds’” in Preston, David Seth (ed), Virtual Learning and Higher Education. Amsterdam, Netherlands: Rodopi: 133-49.

Bump, Jerome (1999), "Left vs. Right Side of the Brain: Hypermedia and the new Puritanism." Currents in Electronic Literacy 1.2. http://www.cwrl.utexas.edu/currents/fall99/bump.html

Bump, Jerome (1990), "Radical Changes in Class Discussion Using Networked Computers." Computers and the Humanities 24: 49-65. http://www.cwrl.utexas.edu/~bump/research/Radical%20Changes%20in%20Class%20Discus.htm

Conklin, Megan (2007), “101 Uses for Second Life in the College Classroom.” http://facstaff.elon.edu/mconklin/pubs/glshandout.pdf

Cope, B. and Kalantzis, C. (editors for the New London Group) (2000), Multiliteracies: Literacy Learning and the Design of Social Futures. London: Routledge.

Faigley, Lester  (2006) Little Penguin Handbook. Boston: Longman

Faigley, Lester  (2004), Picturing Texts. Co-authors: Diana George, Anna Palchik, Cynthia Selfe. New York: Norton.

Faigley, Lester  (2003), “The Challenge of the Multimedia Essay,” in Composition Studies in the New Millennium: Rereading the Past, Rewriting the Future. Carbondale: Southern Illinois U P. 174-187.

Faigley, Lester  (2001), “They Are Already In It,” in John F. Baber & Dene Grigar (eds) New Worlds, New Words: Exploring Pathways for Writing About and In Electronic Environments. Cresskill, N.J.: Hampton P. 417-420.

"Gaming the System," Newsweek April 16, 2007, p. 17.

Gee, James Paul (2005), Why Video Games are Good for Your Soul. Australia: Common Ground P.

Gee, James Paul (2003), What Video Games Have to Teach Us about Learning and Literacy. New York : MacMillan, Palgrave.

George, Diana (2002), “From Analysis to Design: Visual Communication in the Teaching of Writing.” College Composition and Communication. 54: 11-39

Haynes, Cynthia and Jan Rune Holmevik (1998), High Wired, On the Design, Use, and Theory of Educational MOOs. Ann Arbor: U of Michigan P.

Horizons Report (2007), EDUCAUSE and the New Media Consortium.

Kress, Gunther R. (2003), Literacy in the new media age. London: Routledge.

Kress, G. and Van .Leeuwen, T. (2001), Multimodal Discourse: the Modes and Media of Contemporary Communication. New York: Oxford U. P.

Kress, G. and Van .Leeuwen, T. (1996), 2^nd ed. Reading Images: the Grammar of Visual Design. (London: Arnold).

Lanham, Richard (1993), The Electronic Word: Democracy, Technology, and the Arts. Chicago: Univ. of Chicago Press.

Lankshear, C., & Knobel, M. (2006) New Literacies: Everyday Practices and Classroom Learning. 2^nd ed. Maidenhead: Open U. P.

Matthewman, Sasha, with Adrian Blight and Chris Davies (2004), “What does Multimodality Mean for English? Creative Tensions in Teaching New Texts and New Literacies.” Education, Communication, and Information 4.1:153-174.

New London Group (1996), “A Pedagogy of Multiliteracies: Designing Social Futures.” Harvard Educational Review 66: 60-92.

Peterson, Mark (2006), “Learner Interaction in a Chat-based Virtual World.” Computer
Assisted Language Learning 19.1: 79-103.

Schroeder, R. (1997), “Networked worlds: Social aspects of multi-user virtual reality technology.” Sociological Research Online. 2(4). http://www.socresonline.org.uk/socresonline/2/4/5.html

Schroeder, R. (2002), “Social interaction in virtual environments: key issues, common themes, and a framework for research,” in R. Schroeder (Ed.), The Social Life of Avatars: Presence and Interaction in Shared Virtual Environments. London: Springer: 1-6.

Steinkuehler, Constance A. (2005) “Cognition and Literacy in Massively Multiplayer Online Games.” http://website.education.wisc.edu/steinkuehler/papers/SteinkuehlerNEWLIT2005.pdf

Stephenson, Neal. (1992) Snow Crash New York: Bantam.