Mediate is a collaborative time-based media annotation tool developed by River Campus Libraries and Joel Burges, Associate Professor, Department of English, Film and Media Studies, Digital Media Studies, and Director of the Graduate Program in Visual and Cultural Studies at University of Rochester.

Re-imagining Annotation

Media literacy is one of the most pressing concerns for research and teaching due to the centrality of multi-modal content—images, sounds, and text—in our culture. From film and television to video games, music videos, social media, music, and podcasts, multimodal content is ubiquitous in our everyday lives. Yet education still focuses primarily on text-based literacies. Mediate, a web-based platform that allows users to annotate multimedia content, tackles this problem by providing a means for individual or collective inquiry into time-based media. Users can to upload video or audio, generate automated markers, annotate their content on the basis of customizable schema, produce real-time notes, and export their data to generate visualizations.

User dashboard at login.
The annotation interface is made up of an annotation panel, timeline, and video player. The panel includes: thumbnails, marker type, and time code. The annotation panel syncs with the timeline and allows users to scrub through the video.

Once the annotation process is finished, users have a wealth of preserved data and observations about the media. Past students have used that data to create visualizations based on their observations, using the visualizations to test hypotheses and produce interpretations of multimodal content. The utility of Mediate ranges from students learning to close-read time-based media narratively, visually, and sonically to scholars working individually or collectively to reevaluate central questions in the history of moving images or sound recording at both microscopic and macroscopic scales.

Sample Use Cases

Introduction to Media Studies
Joel Burges

Linguistics and Advertising
Solveiga Armoskaite

Poetics of Television
Joel Burges

Music and Digital Culture
Darren Mueller

Annotation interface for a linguistics analysis.
Annotation interface for a music analysis.

Technical Specs

Mediate is a web-enabled software platform built in Python and JavaScript. It makes use of several popular and well-maintained open source libraries, including Django for its server-side application, OpenCV for computer vision and image processing, FFMPEG for multimedia transcoding, and ReactJS for its client-side application. Mediate is designed to handle large sets of annotations (which we also call “markers”), and supports concurrent updates through websockets, enabling a truly collaborative real-time system composed of several modules all tied together by a RESTful API. Through the API, users are able to create custom tags, upload media, create groups, add annotations, and query data in CSV or JSON formats. All uploaded video files and resulting derivatives are currently stored in a secure, aliased directory, and are only available to those users who have sufficient permissions to view them. In addition to the images and video files produced through the video processing pipeline, all annotations and relevant metadata are stored in a secure PostgreSQL database and backed up regularly. Since Mediate is not designed as a digital asset management or preservation system, original uploads are removed from the server once the derivatives have been successfully created.

What’s Next?

Mediate has entered the second phase of development. Based on student feedback, Josh Romphf has made substantial changes to the user interface. In spring 2019, we ran student focus groups on Mediate in order to gain further insight to help prioritize future features. In the coming months, we look forward to working with Sean Morris to develop a more user friendly admin interface. In fall of 2019, undergraduate researcher assistant Tiamat Fox will work with the project team to develop more robust documentation and synthesize feedback from the expanded use cases.

Receive Updates

Sign-up to receive updates about Mediate. In the coming year, we will reach out to colleagues at other institutions to test the new administrative interface and possibly see if there is interest from the community in using Mediate in a research project or course.

Solveiga Armoskaite (Linguistics) discussing Mediate as part of the Digital Pedagogy Workshop, Summer 2018

Project Team

Platform Development

  • Joel Burges, Principal Investigator, Associate Professor, Department of English, Film and Media Studies, Digital Media Studies, and Director of the Graduate Program in Visual and Cultural Studies, College of Arts, Sciences, and Engineering
  • Josh Romphf, Lead Designer and Developer, Digital Scholarship Programmer, Digital Scholarship Lab, River Campus Libraries
  • Emily Sherwood, Project Manager, Director, Digital Scholarship Lab, River Campus Libraries
  • Sean Morris, Interface Designer, Information Discovery Team, River Campus Libraries
  • Nathan Sarr, Web Applications Programmer, Information Discovery Team, River Campus Libraries
  • Ralph Arbelo, Library Network Systems Administrator, Library IT Services, River Campus Libraries
  • Stephen Alexander, Library Network Systems Administrator, Library IT Services, River Campus Libraries
  • Maurini Strub, Director of Assessment, River Campus Libraries

Project Collaborators

  • Darren Mueller, Assistant Professor, Musicology, Eastman School of Music
  • Solveiga Armoskaite, Assistant Professor, Linguistics, College of Arts, Sciences, and Engineering
  • Patrick Sullivan, Doctoral Candidate, Visual and Cultural Studies, College of Arts, Sciences, and Engineering
  • Tiamat Fox, Undergraduate Research Assistant, College of Arts, Sciences, and Engineering

Previous Collaborators

The project team owes a debt to those who worked on the early stages of development, including:

  • Nora Dimmock, former Project Manager and Assistant Dean, River Campus Libraries; currently, Deputy University Librarian, Brown University
  • Tom Downey, former Library Network Systems Administrator, Library IT Services, River Campus Libraries; currently, Infrastructure Engineer, Springshare