Introduction to Multimodality and Digital Humanities

TL;DR

Multimodality recognizes that meaning is made through various modes like text, image, and sound, not just written words. Digital Humanities (DH) uses computational tools and methods to study and create humanities scholarship. Together, they offer new ways to analyze and present cultural artifacts beyond traditional print-based approaches.

1. The Mental Model

Think of multimodality as understanding that communication isn't just words on a page, but a blend of different "languages" like visuals, sounds, and gestures. Digital Humanities is like having a powerful digital toolkit to explore these different languages and their meanings in cultural works.

2. The Core Material

Multimodality, in simple terms, is the idea that meaning is made through a combination of different communication modes. These modes aren't just text; they include images, sound, gestures, spatial arrangements, and more. When we analyze a multimodal artifact, we look at how these different modes work together to create meaning that might not be possible with just one mode. For instance, a film combines visual images, spoken dialogue, music, and sound effects to tell a story.

Digital Humanities (DH) is an academic field that uses computational methods and digital resources to research, teach, and create in the humanities. It's about applying new technologies to traditional humanities questions. This could involve digitizing old texts for analysis, using data visualization to explore historical trends, or creating interactive digital editions of literary works. DH often focuses on accessibility, collaboration, and new forms of scholarly publication.

Understanding Modes

Photo by Engin Akyurt on Pexels

Modes are socially and culturally shaped resources for meaning-making. Think of them as channels or systems through which we communicate.

• Linguistic mode: Written or spoken language (words, grammar, vocabulary).
• Visual mode: Images, colors, layouts, shapes, perspectives.
• Audio mode: Music, sound effects, spoken dialogue, tone of voice.
• Gestural mode: Body language, facial expressions, movement.
• Spatial mode: Arrangement of elements in space, proximity, distance.

The Intersection: Multimodality and DH

Photo by Google DeepMind on Pexels

When multimodality meets DH, it opens up exciting possibilities. DH tools can help us analyze multimodal texts more systematically, reveal patterns that are hard to see manually, or create new multimodal scholarship. For example, you could use DH tools to:

• Analyze the visual composition of thousands of historical advertisements.
• Map the movement of characters in a film over time.
• Create interactive digital editions of poems that include audio readings and visual annotations.

Here's a simple workflow for how multimodality and DH often connect:

graph TD
    A["Identify Multimodal Artifact"] --> B["Select DH Tool(s)"];
    B --> C["Digitize/Prepare Data"];
    C --> D["Analyze Modes (with DH tools)"];
    D --> E["Interpret Findings"];
    E --> F["Present New Scholarship (often multimodally)"];

3. Worked Example

Let's consider a historical political cartoon. Traditionally, you might analyze its visual elements (caricatures, symbols) and any accompanying text.

Using a DH approach, you could:
1. Digitize: Scan hundreds of such cartoons into high-resolution images.
2. Annotate: Use image annotation software (like tools within a DH platform) to tag specific visual elements in each cartoon: "Uncle Sam," "bear (representing Russia)," "factory smoke," "speech bubble text." You could also tag their spatial relationships (e.g., "Uncle Sam larger than bear").
3. Analyze: Run scripts or use DH tools to count frequencies of certain symbols, track their evolution over decades, or identify common visual layouts. You could visualize this data as a timeline or network graph.
4. Interpret: Based on the quantifiable data, you might discover trends in how certain nations were visually represented during specific historical periods, leading to new insights into propaganda or political rhetoric that weren't obvious from just looking at a few cartoons.

4. Key Takeaways

• Meaning is often created through the interaction of multiple communication modes (text, image, sound, etc.).
• Multimodality helps you analyze how these different modes work together to convey messages.
• Digital Humanities uses computational methods and tools to study and create humanities scholarship.
• DH allows for systematic analysis of large datasets and the creation of new, interactive forms of scholarship.
• Combining multimodality and DH offers powerful ways to understand and present complex cultural artifacts.

Common mistakes to avoid:
- Treating all modes as equally important in every artifact; their importance varies.
- Ignoring the context in which a multimodal artifact was created or received.
- Thinking DH is just about digitizing things; it's about using digital methods for new insights.
- Believing DH replaces traditional scholarship; it enhances and expands it.

5. Now Try It

Choose any non-textbook item you own – a magazine advertisement, a movie poster, a song's album art, or even a meme. Spend 15 minutes identifying at least three different communication modes at play (e.g., linguistic, visual, spatial, audio if applicable) and describe how each mode contributes to the overall message. Then, brainstorm one way a simple digital tool (like image analysis software or a spreadsheet) could help you analyze similar items on a larger scale.

Success looks like: You've clearly listed the modes, explained their individual contributions, and suggested a plausible digital approach to extend your analysis.