The gaining influence of digital research has not gone unnoticed among media historians. Many audiovisual archives have opened their digital collections via large scale infrastructures, including new digital tools that enable scholars to explore, compare or analyse these collections with new research questions – or revisit old ones in new ways. As expected, this digital transformation has raised a debate as to what it might mean for the research field of media history. Eef Masson recently argued that the rise of the digital humanities is characterised by a collision of epistemic traditions: hermeneutics and positivism.1 Huub Wijfjes has also reflected on the implications of the ‘digital turn’ for media historical research.2 As Masson, he has argued that this ‘turn’ has reignited a longstanding debate on whether “history should hermeneutically focus on understanding and contextualising unique events or on analysing structure and patterns based on quantifiable units and data.”3
However, the discussion not only revolves around the question whether we should take quantitative or qualitative approaches, but also on the possible transformative aspect of established research practices as a result of changing research environments. What happens if we base our research on digital tools developed within a specific digital infrastructure: are we still able to “achieve our analytical goals”?4 Indeed, do digital tools invite new methodological approaches, for instance related to automated retrieval of metadata out of a larger dataset for further statistical analysis? Or can the tools also be deployed traditionally, in more hermeneutic media historical research, by supporting scholars in reconstructing particular historical trends in a digitised corpus of audiovisual archival materials?
We arrive at these preliminary questions because of our involvement as researchpilot scholars in the Common Lab Research Infrastructures for the Arts and Humanities (CLARIAH CORE, 2015-2018) project, funded by the Netherlands Organisation for Scientific Research (NWO). Our task was to conduct a research project, entitled Me & Myself: Tracing First Person in Documentary History in AV-Collections (M&M), to explore the possibilities and constraints of doing media historical research through our use of the Media Suite (MS), a research environment of the CLARIAH infrastructure.5 The Media Suite is currently in development and ties into the CLARIAH focus area of media studies.6 It provides an integrated research environment that makes possible to search, annotate, analyse and enrich large digitised audiovisual and contextual collections from archives and other cultural heritage institutions across the Netherlands.7
Our pilot project aims to explore the added value of video annotation by testing the integrated manual video annotation tool in connection to its use in a research project that addresses a more or less traditional media historical research question. In addition, the M&M research project aims to trace when and where the emergence and rise of an autobiographical and confessional mode of documentary filmmaking can be located on Dutch public service television in the last five decades. The main collection used by the project is the digitised audiovisual collection of the Netherlands Institute for Sound and Vision (NISV), which, due to the CLARIAH project, has been made available for online access and viewing to researchers in the Netherlands for the first time.8
Video annotation tools have gained a growing influence in the field of film and media studies since the second half of the 2000s.9 This is predominantly due to the increasing availability of major video collections, whether they are accessible through media archives or are obtainable on social media platforms. The size of those collections, and the fact that they are available as digital data, invite new research opportunities and challenges for which annotation offers a solution. The act of annotation, as a common and longstanding analytical aid to close reading practices of media scholars, is often employed as a means to an end. But annotation can entail so much more, especially when done with digital tools. Video annotations tools can be re-usable for the sake of retrieval, enrichment and contextualisation by creating metadata which allows connecting one’s sources or data to other archival material or scholarly annotations; while other tools are particularly useful in preparing the research material for quantitative analysis by using a stricter formalist or rigorous qualitative coding approach.
In short, there are several ways of doing digital video annotations and each type of annotation align to different research traditions. The tool at hand – the manual annotation tool in the Media Suite – does hence not meet every specific research need. This challenge touches on the central issue of this article: we want to explore and explain how the MS video annotation tool influences our research practice while still ‘in development’. However, we are also interested in how we as researchers execute agency as co-developers of the tool. Hence, this article addresses how doing media history is methodologically influenced by a digital tool that aims to assists us in the analysis process. We reflect on the affordances of the tool and on how they impact our scholarly work. By asking these and other questions about the effects of digital tools on our scholarly work, we follow the suggestions made by Marijn Koolen, Jasmijn Van Gorp and Jacco van Ossenbruggen to integrate ‘tool criticism’ and ‘reflection’ as essential elements in the research practice of a digital scholar.10 These include reflecting on our experiences with the infrastructure at hand, and accessibility to data since our usage of data are always mediated via a particular set of tools.
Scholarly Challenges in Building a Digital Research Infrastructure
Over the past few years, several large-scale digital infrastructural projects have emerged in Europe. Well-known examples include DARIAH – a pan-European infrastructure for arts and humanities scholars working with computational methods – and CLARIN – European Research Infrastructure for Language Resources and Technology. In the Netherlands, the CLARIAH project aims to build upon these infrastructures, designed to further explore the wishes of researchers in three focus areas within the humanities: linguistics, socio-economic history and media studies.
Together these focus areas often use several important types of data in the humanities: text, images, audiovisual material and structured data. Different challenges and opportunities emerge for scholars as users, or as part of the creation of these infrastructures. We identify, for instance: the changing role of the scholar as co-developer, the tension between generalisation and specificity in developing information and research services for scholars, the limitations in the breadth of digitised collections, and – most importantly for this article – the need for scholarly reflection and tool criticism.
As stated, the arrival of new research infrastructures has affected the role played by the ‘digital scholar.’11 Not only do researchers assess the usefulness of infrastructures and their tools, they are often also involved as codevelopers. The result is that scholars can, to a certain extent, directly influence how the infrastructure and tools are built according to their specific needs.12 The latter has been the case regarding the development of the CLARIAH infrastructure, and the Media Suite, because one of the services of this infrastructure is suitable for media historical research. This new dynamic between the researcher and the infrastructure ‘in development’ seems beneficial, but can also present challenges: one of them is how to align particular research activities with more general institutional implementation roadmaps. The latter often has a significant impact on research practices. Even if pilot scholars were being asked to list their requirements, and thus were able to make their needs explicit, they also had to adapt to the inherently slow pace of developing new systems. Indeed, system development in the context of the research infrastructure is complex due to the accommodation of diverse requirements, not in the least because alternative solutions before actually implementing the services often needs to be investigated.
All of these specific circumstances are relatively novel in the development of Dutch digital humanities infrastructures. As Wolfgang Kaltenbrunner noted, the emergence of digital research infrastructures within the humanities took place relatively late compared to other research domains. Concerning the Netherlands, he observed that the first contours of a more comprehensive digital research infrastructure in the humanities emerged to reduce “the organizational fragmentation of the humanities.”13 This fragmentation relates to a larger international debate on whether the digital humanities should strive to become a ‘big tent,’ by unifying the variety of research disciplines and traditions in the humanities into one standardised ontological and epistemic domain and research practice.14
This approach enticed Joris van Zundert to warn about the dangers of what he termed the ‘generalization paradox.’15 According to him, the contradiction usually pertains to a desire within infrastructures to cater to various research communities, which pushes designers toward generalisation, while individual researchers hope to find specific methods, tools and data models according to their own requirements. In van Zundert’s view, if this path is followed, many burgeoning large-scale infrastructures in the humanities will result in “highways that connect nothing to nowhere.”16 Indeed, one of the effects he fears is a lack of interest by those infrastructural projects to pursue an agenda in which “the existing heuristics and hermeneutics are appropriately translated into their equivalent digital counterparts.”17
Another perspective is to think about infrastructures as a ‘house with many rooms,’ enabling a wide variety of disciplines and traditions to explore novel digital tools and research practices from the context of their respective academic backgrounds.18 So far, CLARIAH represents the second approach more than the first, which seems to be a positive development. The infrastructure offers a diversity of tools and data, designed for specific research areas, while it also allows researchers to be active contributors to its further development. Nevertheless, institutional policies and technological investments greatly determine decision-making, including priorities set for further developing functionalities or eventual copyright limitations.
The advantages of CLARIAH as an infrastructure – and the Media Suite as a workspace – are that they provide authenticated access, via login with university credentials to audiovisual data collections and related mixed-media, as for example contextual sources provided by Dutch cultural heritage and knowledge institutions. For our M&M project, we were mainly interested in Dutch autobiographical documentaries which are part of the digitised historical television collection of the NISV.19 For this project, we were able to make a selection of 80 relevant items based on an extensive exploratory search process.20 The growing accessibility of various data collections is greatly welcomed, and many advances are made right now in the Media Suite to ensure that more and more scholars can make use of audiovisual collections thanks to the existence of the research infrastructure and large-scale availability of digitised and digital public service broadcasting heritage.
It is nevertheless relevant to keep in mind that there are potential drawbacks of doing media history in the Media Suite on the basis of available digitised, audiovisual collections. Audiovisual heritage was preserved unevenly, and as media historian Helle Standgaard Jensen has noted, “many programs were never kept even in their analogue format (…) and therefore we do not know what was left to digitize in the first place. The problem here is that with no knowledge of the pre-selection, we are left in the dark when it comes to doing critical source analysis in its most basic form.”21 So, it is important to acknowledge that digitised sources are by no means considered to be taken as representative solely on the basis of them being digitally available.
Therefore, we stress the inherent hybridity of historical research practice, especially when digitised collections are at play. As Gerben Zaagsma argued, the discipline of history is faced with the “real challenge (…) to be consciously hybrid and to integrate ‘traditional’ and ‘digital’ approaches in a new practice of doing history.”22 We are especially interested in whether the CLARIAH Media Suite is (or is not) facilitating the ‘heuristics’ and ‘hermeneutics’ of traditional media historical research. In other words, we seek to assess whether the Media Suite – including the integrated tools – might suffer from the ‘generalization paradox’ discussed earlier, and we will return to the issue in the conclusion of this article.
Digital Tool Criticism
The project Me & Myself: Tracing First Person in Documentary History in AV-Collections explores the video annotation tool as a mode of scholarly description, including a specific approach to analyse items in audiovisual collections whilst reconstructing a historical trend of a sub-genre. In this article we focus on the impact this specific tool has on the analytical research phase when identifying interrelated formal, stylistic and narrative elements within our corpus of documentary TV-programmes. Part of our explorations of the potential of the video annotation tool is to explicitly include an active process of tool criticism in the project design.
Media historians who use digital humanities research methods should be aware that tools – like texts – make an argument.23 Tools can be seen as “theoretical objects and as such can be argued about, analysed and criticised.”24 In addition, Koolen, Van Gorp and van Ossenbruggen strongly recommended to reflect explicitly on the influence of the tool on every phase of the research process.25 On the basis of our experiences as pilot scholars, we therefore aim to conduct a form of tool criticism: what does digital video annotation offer, and which of its affordances and functionalities will help us to accomplish our research goals – and which ones are less suitable? How did the tool influence our research methodology, and did it stimulate us to change our research question or strategies?
As Fred Gibbs and Trevor Owens have argued, it is essential to engage in methodological transparency in our historical writing when using novel means to explore and interpret historical data.26 This methodological transparency entails a clear understanding of what the tool actually does and what it does not do, and therefore should be discussed, explained and reflected upon together with the results. We argue that such consideration is a distinctly transformative aspect of media historical research in a digital setting. Previously, (media) historians did not always include explicit methodological reflections, but digital humanities research requires explicit tool criticism and reflection to fully understand the scope, possibilities and constraints of a particular research question or project.
From the start of our project, we became aware of the importance to understand the functionalities of the tool and how it influenced our research practice. Once we entered the Media Suite, we were immediately confronted with the need of ‘tool-thinking’, because we had to familiarise ourselves with the use of a tool we had not used before. At the same time, our usage also became part of the development of the tool as well. From the outset, this dual position strongly affected our way of thinking and working, stimulating a reflective scholarly awareness. This particular awareness also helped us to perceive our new encounters with a tool as a necessary experimental moment and, consequently, became an important part of our digital scholarship. “The research process should include experimentation to find out how digital tools work in terms of modeling and transforming data, and to bring out and refine a scholars’ own assumptions about tools”, as Koolen, Van Gorp and Van Ossenbruggen have reminded us.27 In what follows, we will explain more in-depth what happened when we began to experiment with the basic and generic manual annotation tool, how our hands-on experiences challenged us to come to terms with its possibilities and limitations, but also how the tool provided us with alternative strategies helping us to deal with the tool in novel ways.
Video Annotation Tools
John Unsworth has termed annotation a ‘scholarly primitive’ and regards it as one of the basic research ‘functions’ necessary to any scholar regardless of the respective discipline.28 Annotation includes activities that occur during searching, reading or interpreting, such as selecting or choosing relevant media objects, bookmarking passages or segments, adding notes, writing comments or memos, adding metadata, coding according to personal or common ‘code books’, or creating links or connections between items and segments. These activities can be done manually (as was common practice until recently), but can also be mediated or assisted computationally and, only to a certain extent in the humanities, be fully automated. Since digital libraries became more and more accessible to scholars and other users, information systems aimed to support annotation during reading, basically through highlighting passages or adding notes ‘in the margins.’29
In addition, since the widespread use of the VCR in the 1980s and 1990s, and the arrival of DVD in the late 1990s, annotating videos also became a common research practice. Moving images thus became more ‘attainable’ for analysis and interpretation by media scholars.30 Today, video annotation tools are abundantly available. In general, these tools offer the option to upload or stream online, one or more audio or video files. These can then be segmented according to one (or more) criteria, by adding labels (also called ‘tags’ or ‘codes’), and enhanced further by the addition of other types of annotations (e.g. comments). While most of these tools share similar core features, they also differ in how they model data and functionalities, and in the support they provide to generate different data visualisations. Given the available tools, these have resulted in various affordances that can (or cannot) support scholars in performing their analyses – while at the same time influence research methods and outcomes.31
The most well-known audiovisual annotation tools originate from at least four different research and professional traditions. The first relates to the field of linguistics and communication studies that often use the ELAN and ANVIL software packages. Both packages are free of charge and offer a variety of functionalities to annotate on multiple tiers, work from controlled vocabularies, and even work with 3D motion capture files.32 Second are the so-called QDAS (Qualitative Data Analysis Software) packages used in ethnography and qualitative analysis, such as Atlas.ti and Nvivo.33 These tools enable the user to organise and structure multiple data sources, such as audio, visual or text files of interviews or other documents. By using these tools, detailed analyses can be made through coding, segmenting and linking of data, aided by visualisation functionalities.34 The third tool kit pertains to professional applications for video editing, such as Final Cut Pro.35 The fourth, lastly, consists of web-based tools offered by archival aggregators, such as EUScreen or Europeana. These aggregator systems, which were originally made for searching across collections, are now progressively integrating video annotation functionalities to support users in common research tasks that occur while working with online collections (for example, with the creation of personal clips, posters, or collections of bookmarks).
The first group of tools facilitate audio-visual segmentation, as for example storing timestamps with a certain ‘code,’ or ‘tag,’ attached to it. This kind of annotation has particularly appealed to media scholars, who also went on to contribute to the further development of video annotation with tailored-made tools for film analysis, such as Les lignes du Temps.36 However, possibilities of annotation were prior to these digital tools also of importance to a branch of film studies devoted to counting and labelling shots, most famously done so by film scholar Barry Salt. In 1974, he proposed a statistical method of stylistic film analysis as an alternative to the more traditional, interpretative practices prevalent in film studies at the time.37 Salt manually counted shot lengths and labelled them according to a coding scheme based on such variables as camera movement, angle and framing. More recently, Yuri Tsivian and Gunars Civjans similarly aspired to a more positivist mode of historical film scholarship.38 They developed the Cinemetrics tool to enable researchers to computationally segment and label films, and storing this information in a web-based database.
In the Digital Formalism project (2007-2010), headed by film historian Adelheid Heftberger, the video annotation tool ANVIL was repurposed for the formal study of Russian filmmaker Dziga Vertov’s films, using a coding protocol based on the same one used in Cinemetrics.39 The Digital Formalism project created histograms, key frame visualisations and an annotated DVD edition of its corpus. Similarly, film historian Barbara Flueckiger’s film colour research project (2015-2020) reused annotation functionalities of the video annotation tool ELAN to fit their specific project needs. The project resulted in the ‘Timeline of Historical Film Colors,’ an online presentation of annotation-based analyses of the aesthetics and technological history of colour in film. Flueckiger’s team also used manual and semi-automatic annotations in order to relate specific metadata to the films, ranging from highly fine-grained classifications to linking annotated segmentations to technical film journals or film-theoretical essays.40
Multiple-tier annotation support is another important innovation in the development of video annotation tools and is available in, for example, ELAN and ANVIL. By using multiple-tier annotation it is possible to assign multiple layers of annotations – such as aesthetic aspects in one layer, and objects or characters in another layer – to audiovisual documents. The Digital Formalism project exploited this functionality to provide a multi-layered description of Vertov’s films on the basis of multiple variables (shot-scale, camera movement, etcetera) and a strict code book or ‘protocol.’41 Tiers, or annotation layers, have proven to be a powerful and necessary feature in digital annotation to enable more complex and interrelated analyses that go beyond solely identifying and annotating a single defined unit of analysis (such as neatly defined categories as shot length, or camera movement). Tiers enable analyses in which the grouping of tags, according to specific ‘facets’, becomes possible. This important feature of video annotation, however, is currently unavailable in the CLARIAH Media Suite. We therefore had to consider and make alternative methodological decisions for our research project in order to deal with this limitation in our tool-use.
M&M and the CLARIAH Media Suite Annotation Tool
The Media Suite annotation tool offers a combination of functionalities, and was originally developed for other projects at the Netherlands Institute for Sound and Vision, such as LinkedTV, Axes, ArtTube, and The Mind of the Universe. In the first project, LinkedTV, emphasis was put on the automatic extraction of entities (names of persons, locations, buildings) from spoken, written and visual information in media documents. Because automatic retrieval proved to be problematic for various reasons, it was decided to focus instead on manual annotation, using automatically extracted information to aid the annotator.42
The tool’s evolution hence gradually moved towards manual annotation and was eventually adapted for integration within the CLARIAH Media Suite. On the basis of functionalities, affordances and research applications of video annotation discussed in the previous section, the tool developers decided to first of all focus on supporting the most essential tasks such as segmenting, tagging, commenting, linking, and adding personal metadata via customizable templates. The tool, however, continues to be updated and refined on the basis of new recommendations by scholars as they engage in their research projects.
Advice and suggestions were also given by the M&M research project. At the start of our project, we recommended to develop the tool with multiple-tier annotation support. Analysing autobiographical documentaries needs an analytical approach that acknowledges the inherent complexities of this genre. We believe that the possibility to annotate segments on the basis of its multimodal complexity, by identifying a meaningful interrelationship between sound, speech, editing and framing, would benefit our media historical approach. Unfortunately, the CLARIAH infrastructural and tool development process prioritised essential functionalities over multiple-tier annotation support. This meant that we had to abandon our intention to use multiple-tier annotation in our project.
Because the tool currently only allows for single-tier annotation, we thus had to explore other ways to achieve our analytical goals. We redesigned our methodology by focusing on alternative strategies that would fit our type of research. In order to be able to annotate interrelated features within our dataset of autobiographical documentaries, we devised a methodological workaround that proved to be suitable for our project: the choice to make segmentations on the basis of units of interest rather than the strict, codified units of analysis traditionally deployed in the annotation process. The decision was intricate, and needs some further explanation. Formal units of analysis, as was the case in the Cinemetrics projects, are well-suited for statistical analysis of singular elements in Hollywood film style and aesthetics that tend to be commonly agreed upon by scholars, such as shot boundaries, framing, cinematography and so on. This approach can be computationally designed due to the highly formalised characteristics and structure of narrative Hollywood cinema. Throughout film history, Hollywood narrative fiction film has aligned to a mode of storytelling that more or less follows the rules of the continuity system.43 This means that these films are highly suitable for annotation-based analyses that apply a strict coding protocol with well-defined variables.
Annotating documentaries, however, is another matter, because “[t]hey evolve, change, consolidate, and scatter in unpredictable ways,” following media theorist Bill Nichols.44 Documentaries are often a compound of different strategies; fictional, non-fictional, cinematic, and non-cinematic.45 This means that while one documentary may use the diary form (adopted from literature) in conveying its narrative, others may draw from journalistic (adopted from newspaper and reportage conventions) or other strategies. In other words, the artistic and formal diversity of documentaries is the most relevant common denominator. As Nichols has stated, documentaries often tend to have a particular ‘voice’ that determines “the entirety of each film’s audio-visual presence: the selection of shots, the framing of subjects, the juxtaposition of scenes, the mixing of sounds, the use of titles and inter-titles.”46
Due to this idiosyncratic status, documentaries are not necessarily obvious candidates for a Cinemetrics-inspired and more formalist analysis, but rather need an approach that does justice to their complexities. As a consequence, we choose to introduce the unit of interest as a methodological way to determine the focus of our video annotation protocol. We therefore define our understanding of units of interest as a way of segmenting particular moments in a documentary (or of any other moving image document) on the basis of multiple, interrelated formal and aesthetic elements that cannot be subsumed under one strictly defined variable in a code book. In the case of the M&M project, for example, units of interests could refer to segments – similar to the idea of ‘clips’ – in which we found evidence of, or attribute ‘traits’ to, multiple interrelated formal and stylistic manifestations of the autobiographical and confessional.
By introducing the unit of interest as our methodological focus, we acquired some flexibility in the segmentation process because we were not bound to strictly formal divisions of singular stylistic or formal categories in our corpus such as shots, sequences or scenes. Moreover, this approach enabled us to integrate our interpretative efforts with the video annotation tool and thus explore video annotation in terms of its possible hermeneutic affordances. Furthermore, we believe that by annotating on the basis of units of interest we were able to create evidence to historically reconstruct the development of the confessional mode in Dutch documentary history. In contrast to a more formal analysis, the main advantage of this form of reconstruction was that it offered means to map a genre by the identification of a cluster of distinct features or traits. In short, the form of ‘reconstruction’ gave us a flexible methodology that encourages the analysis of the genre via the identification of the interplay of its most distinct features, without favouring or taking the formal dimension as the main basis.
Testing the Tool
With the help of a test case of the Dutch documentary Namens onze Ouders (In the Name of Our Parents; directed by Monique Wolf and Hans Fels, IKON, 1978) and Pappa is weg en ik wilde nog wat vragen (Dad’s Gone and I Still Wanted to Ask Him Something; directed by
Marijn Frank, VPRO, 2007), we will illustrate how we used the tool. Our aim is to reflect on how the functionalities enabled us to deal with the formal complexity of documentaries, and the interpretative dynamics involved. We will also show how the segmenting and tagging functionality in particular enhanced traditional media historical research practices, while also bringing up new challenges.
On a methodological level, challenges oscillated between the affordances and functionalities given to us by the tool, and our interpretative strategies to comprehend an audiovisual document in its historical and stylistic context. The first phase of our research process consisted of building a corpus of documentaries which we suspected to be autobiographical or confessional.47 In this phase the MS enabled us to build our corpus through the bookmarking functionality. This presented the possibility to save our selection of documentaries (or other archival documents) to a user project, and consequently made available in the personal workspace. This functionality created an overview of the documents which were created and attached to our user project. It also allowed for searching our corpus on the basis of keywords, periodisation, and a wide range of metadata-based filters. In general, this search functionality became more informative as research progressed with added annotations to the documents in the corpus.
The affordance of the workspace, the user projects and the bookmarking functionality all form part in a more general affordance of the ‘digital archive’ and its web-based, networked topology.48 Because of this specific topology the researcher is able to outsource the activity of data storage, for which the responsibility lies at CLARIAH and NISV, who ensures the longevity of the corpus that is built within its infrastructure. On the one hand, this frees media historians of the task of keeping records of their own corpus, via copies, photos, registers, analogue annotations, or other means. On the other, this situation creates a dependency on external providers for sustaining the interface, infrastructure and data.
Our use of the annotation tool yielded some interesting results regarding the challenges that came up when opting for a unit of interest approach. To illustrate these challenges, we will briefly discuss one particular unit of interest from the documentary Namens onze Ouders. We have chosen this scene because it contains a very noticeable idiosyncratic mode of expression and a complex formal interplay of visual and auditory elements. All of these elements, moreover, can be found throughout the documentary. The scene under scrutiny also features some of the more general characteristics attributed to the genre of autobiographical and confessional documentaries. In this particular scene, documentary maker Hans Fels stands on a field at the outskirts of the Auschwitz-Birkenau concentration and extermination camp, trying to imagine what his parents had experienced there during the Holocaust. It is a notable moment in the documentary because we see the filmmaker on-screen, directly addressing the viewer, while explicitly reflecting on his painful family history. This is a moment of soliloquy, a recurring artistic device in the genre of autobiographical documentary that often takes the shape of a film or video diary. This device can be seen as an ‘embryonic instance’ of ‘techno-analysis,’ referring to a particular moment in documentary history “when the camera as confessional instrument is taken up by the confessant herself,” following documentary theorist Michael Renov.49 We tagged this segment with ‘filmmaker’s history,’ ‘direct sound,’ ‘filmmaker on-screen’ and ‘family history’ to represent what happened in the sequence. We could, naturally, have continued our tagging, adding for instance ‘Auschwitz,’ ‘field,’ ‘medium close-up,’ ‘hand-held,’ all related to the contents of Fels’s narration and performance in the scene.
Pappa is weg en ik wilde nog wat vragen is another example from our corpus. It represents a more recent example of the evolution of the confessional and autobiographical genre in documentary history. In the documentary, Marijn Frank chronicles her relationship with her father, who is terminally ill and eventually passes away during production. Against the backdrop of her father’s failing health and subsequent death, Franken thematises the sudden realisation that she lacks any intimate knowledge of her father and his family history, in part, due to his introvert nature. To get closer to her father, she sets out to reconstruct his biography by interviewing relatives, and more importantly, by interrogating herself on their strained relationship.
The documentary is rife with self-reflexive, autobiographical moments, captured by Frank during intimate moments on her sofa, in bed, or on the toilet. These segments can be tagged with labels as ‘video diary,’ ‘self-reflexivity,’ and ‘family history.’ There are also other stylistic features that could be tagged, such as the overt presence of the camcorder, and the noise this non-professional equipment produces, but also the moments during which her cat walks in front of the camera, or when Frank sneezes on-camera. These confessional moments heighten the sense of authenticity. However, instead of annotating all of the segmentations in which we observed these stylistic elements, our unit of interest approach allows us to single out the most significant ones, and attribute to them a cluster of interrelated features or ‘traits.’
Assessing the Tool
We can make several observations in relation to how the video annotation tool helped us to reconstruct the autobiographical in the history of Dutch documentaries aired on Dutch public service broadcasting. Having annotations – saved to a personal workspace in the Media Suite – meant that we could search and browse the tags we had added to the various segmentations. This enabled us, for instance, to easily retrieve segmentations tagged with ‘filmmaker’s history,’ and then do a comparative analysis of different instances where a filmmaker’s own biography, or personal history, is part of documentaries. Moreover, we found that our annotations added more in-depth traits and characteristics to our sources. We thus were able to create our own structure, with collections and sub-collections. Furthermore, we could enhance and work with our own qualitative metadata parallel to the existing archival metadata. Finally, the tool forced us to reflect explicitly on our interpretations, and the underlying methodological considerations, and compare them to more traditional, analogue practices.
As an example of analogue research habits, one can imagine a media historian physically visiting an archive, requesting the aid of a technician to watch a film on an editing table or a TV-programme on a specialised video-playback device, taking notes on paper. Even if the digitisation of these activities into (potentially biased) features such as bookmarking, tagging, and segmenting has led to affordances as keyword search, they have also created the possibility for us to use these functionalities to implement a coding regime that can be used for more consistent qualitative data analysis. The fact that our tool was based on single-tier annotation, eventually led to our proposal to deploy units of interest. This allowed us to recognise and emphasise the complexity of our research material. In the two cases we presented, we illustrated how we chose to adopt the notion of units of interest in order to sidestep formal reductionism. This decision allowed us to bring together different layers present within a segment, and to capture the complexity of our research material with a cluster of tags. These tags can be seen as a representation of the interpretative dynamics between us, as researchers, and the documentaries under scrutiny.
Of course, these choices and this strand of research have their shortcomings. We did, for example, not abide to strict formal codes for our annotation strategies. Nor did we follow a consistent coding manual coming from broadly agreed upon formal hierarchies and taxonomies. The disadvantage of this open-ended mode of tagging is that further retrieval (or grouping) of segments will be difficult. This challenge will become even more pressing when scaling up annotation projects, having multiple annotators (for multiple years) working on a corpus – which in turn will result in difficult questions like: Will it ever be possible to agree on guidelines as to what categories might be used to tag and segment documentaries? How can the basic affordances of annotation tools accommodate for disagreement or complementary views? Moreover, is it necessary to agree?
We found that these questions touch upon the core of our methodological challenge, because to examine the complexity and the historicity of a scene, by segmenting it according to a cluster of traits, still requires a more elaborate motivation and validation of our interpretative strategies. Overall, however, the video annotation tool enabled us to go beyond the available archival metadata and explore our qualitative, hermeneutic pursuit to understand the stylistic history of a genre. The combination of contextual information from, for example, broadcast programme guides, and automatically generated metadata about the cinematic, audio and textual elements is something that needs to be explored further when these options are ready in the CLARIAH Media Suite.
In 2015, Jasmijn Van Gorp et. al. published an article in this journal in which they argued that new digital tools will prove essential in unearthing ‘alternative histories’ but also in arousing new methodological questions for media historians in the digital age.50 Our research has not been able to deliver an alternative history, but it certainly inspired us exploring alternative methodological questions and reflect on the impact of digital tools on our research practice. The fact that we entered a specific large-scale infrastructure, including available tools, meant that some of our research routines had to be redefined, or rather, made the need for methodological transparency and reflection more explicit. As Koolen, Van Gorp and van Ossenbruggen pointed out, “the explicitness of digital tools prompts scholars to ask questions.”51
The circumstances in which we found ourselves by using the CLARIAH infrastructure, including the Media Suite and its tools, implied a different research dynamic towards and relationship with our research material. The M&M project turned out to be a valuable experiment for prompting reflection on the use of a tool and infrastructure that are still in development. This meant that we had to adapt to the situation by redesigning our methodology. In retrospect, we have come to appreciate these obstacles as an opportunity to develop a more reflexive attitude towards the affordances of a tool. It has also confirmed the need to develop an integrated research design, offering an opportunity to be “trained in the critical analysis of the creation, enrichment, editing and retrieval of digital data as much as in the classical internal and external source critique.”52
One general conclusion from our work is an apparent interest to see how the evolution of (future) research infrastructures – which combine tools, such as video annotation, and access to data sets (collections of films, oral history records or digitised newspapers) – can improve media historical research. The development of the video annotation tool, at first glance, has seemed to move towards the ‘generalisation paradox’ discusses by Joris van Zundert. Our role as co-developers might mitigate this ‘paradox,’ because we recommended CLARIAH to enhance the tool with multiple-tier annotation support. A layered approach to moving image analysis, we believe, provides flexibility to various modes and strategies of annotation (automatic, manual, curatorial), and allows for different analytical perspectives. We are also delighted that steps are now being taken to implement our recommendations for further development of the Media Suite.
As a research infrastructure, CLARIAH seems to embrace a heterogenous, ‘house with many rooms’ approach, by enabling different types of scholarly work and developing tools that are open to adjustments. At the same time, our exploration also showed that this is not an easy process. There is an inherent tendency within the digital humanities to replace hermeneutic complexity with a desire to strive for reductionism by an increased focus on automated annotation processes and distant viewing. It is nevertheless important to stress that our struggle has resulted in an important lesson, a ‘tale of a tool encounter.’ We were able to keep our ideas intact, to find workarounds. We could also reflect on our own position in relation to the relevance of specific digital tools and the role of (inter)national research infrastructures. By going through this process, we found room to negotiate our way of doing media history, with its distinct potential benefits and pitfalls. One thing is clear, however: the ongoing digitisation of media archives, as well as our contemporary media culture, necessitates and requires us to keep testing our skills to make sure that we do not only follow but also are followed by the infrastructures and the tools.