Sophie Bunz, Brian Castriota, Flaminia Fortunato
The Electronic Media Review, Volume Four: 2015-2016
ABSTRACT
The acquisition of file-based video artworks into collecting institutions, charged with ensuring their long-term viability and accessibility, presents conservators and collection caretakers with many new challenges. This paper explores issues observed in daily practice at the Time-Based Media Conservation Laboratory of the Solomon R. Guggenheim Museum and is the product of a research consortium that was formed as part of a collaboration between the Guggenheim Conservation Department and the Master’s degree program in Conservation at the Bern University of the Arts, Switzerland. The authors of the paper employed a research methodology that included literature review, practical tests, and interviews with internationally-recognized experts engaged with codec development, software engineering, archiving, and digital video preservation. This study highlights specific areas of consensus around the many factors affecting a video file’s sustainability and playback consistency; considers some of the preservation options currently available, including normalization; and offers suggestions around the development of a basis for best practice in the acquisition and conservation of born-digital, file-based video artworks.
INTRODUCTION
This research focuses on the sustainability of video file formats frequently employed by artists and acquired by museums, as well as some of the current options available for ensuring the long-term accessibility and playback consistency of born-digital video artworks. Carried out between 2015 and 2016, it investigates the root causes of some of the playback inconsistencies observed in daily practice at the Solomon R. Guggenheim’s Conservation Department through a combination of practical tests and interviews with experts in the field of codec development, archiving, and preservation. The primary goal of this project was to inform the Guggenheim’s guidelines for artists’ deliverables for the acquisition of born-digital file-based video artworks. Conducted as a collaboration between the Solomon R. Guggenheim’s Conservation Department and the Bern University of the Arts Master’s program in Conservation and Restoration of Modern Materials and Media, this research expands on the findings presented by Joanna Phillips and Agathe Jarczyk to AIC’s Electronic Media Group in 2014 in their talk “Life After Tape: Collecting Digital Video Art.”
THE SITUATION IN MUSEUM PRACTICE
Historically, video artists frequently collaborated with post-production houses to produce tape masters that conformed to industry-standardized specifications. Masters and exhibition copies were therefore delivered to museums on a finite range of standardized tape formats and optical disks for exhibition purposes. Today—since the shift to file-based production and delivery—video artworks are now commonly produced by artists and their studios and acquired by museums as digital files, produced in a variety of codecs, containers, resolutions, frame rates, and compression factors, selected by artists for either deliberate or incidental reasons.
THE ARTISTIC DIMENSION
While museums and archives have overlapping interests—particularly in maintaining both the quality and integrity of their assets, as well as their long-term access—the museum is a unique stakeholder due to the high value and artistic nature of these assets. These files are conferred with special status within the museum, valued as documents of the artist’s creative process and a reference for how the artwork was experienced at the point of its creation. The museum is therefore particularly concerned with sustaining access to all of the intended and incidental audiovisual properties of the artist’s masters and exhibition copies, properties that may be determined by file format, encoding settings, as well as the original component libraries, playback software and display environment.
THE SITUATION AT THE SOLOMON R. GUGGENHEIM MUSEUM
Born-digital video artworks created in the last ten to fifteen years have been produced in a variety of file formats, which differ not only in codec and container, but also in bit sample depth, chroma subsampling patterns, resolution, and pixel aspect ratios. Because many artists have employed various versions of Apple’s editing software Final Cut Pro to create their works, a large number of master files that the Guggenheim has received and now cares for are encoded in a version or flavor of Apple ProRes, Apple’s family of intermediary editing codecs. These include ProRes 422 HQ, ProRes 422 LT, and ProRes 4444. These are generally wrapped in the MOV Apple QuickTime container. In the case of works recorded and edited in DV, HDV, or DVCPRO HD, the Guggenheim cares for master files in these codecs, either in raw form (with a .dv extension) or wrapped into a MOV (QuickTime Movie) container. The Guggenheim also receives compressed exhibition copies from artists, frequently as H.264 in a MOV or MP4 (MPEG-4) wrapper. Some artists also provide normalized preservation masters—these are in many cases video files that have been transcoded to 10-bit uncompressed 422, also known by its FourCC identifier as v210. These files are wrapped with either a MOV or AVI (Audio Video Interleave) wrapper depending on whether they were produced in or for a Mac or Windows environment.
SUSTAINABILITY
In the last decade, several institutions have published guidelines and frameworks outlining the factors that may characterize a file format’s sustainability. These include the documents published by the U.S. Library of Congress (2005), the National Library of the Netherlands (2008), the National Archives in the UK (2008), and the Federal Agencies Digitization Guidelines Initiative (2014). These institutions and initiatives highlighted several overarching and interrelated sustainability factors, briefly summarized:
The adoption or ubiquity of a format—its well-established and widespread use by a critical mass of users—is believed to enhance a format’s sustainability because it is more likely to be guaranteed broad and long-lasting support from software developers.
Formats with external dependencies on specific kinds of hardware are generally thought to be less sustainable in the long term. For file-based video, playback depends on display hardware, playback or viewing software, as well as the presence of specific decoders. These may all be characterized as external dependencies.
A format’s disclosure or openness refers to “the degree to which complete specifications and tools for validating technical integrity exist and are freely accessible” (NDIIPP 2005). Some specifications are completely open and free, some may be obtained with a fee, and some are proprietary and not well-documented in the public domain.
Interoperability or functionality are frequently discussed in the context of sustainability. Some formats may have greater interoperability if they are more widely adopted and their decoders are broadly distributed and implemented, and if they contain certain self-descriptive metadata.
In terms of self-documentation, it is thought that files that contain self-descriptive metadata will be easier to render correctly in the long-term, and will be easier to manage and monitor for integrity. Additionally, streams in containers with chunk- or frame-level fixity, such as MKV (Matroska) or MXF (Material Exchange Format) may be more sustainable by allowing for more granular pinpointing of digital change (Rice 2012).
Robustness refers to a format’s resilience against corruption. Uncompressed codecs are more resilient against perceptible loss due to bit flips than compressed codecs like H.264.
Transparency refers to the degree to which a format is open to direct analysis with basic tools, including human readability using a text-only editor.
NORMALIZATION
Because none of the aforementioned production formats in the Guggenheim’s collection satisfy all of these criteria for sustainability, there is an argument that normalization may enhance the sustainability of these works. Within the digital archiving world, normalization is understood as the standardization of an existing collection of diverse file types to a single format, thought to provide the best compromise of sustainability factors. In museum practice, the term normalization is used to describe the action of transcoding the artist’s production format—the “native master”—in order to create a second, supplementary backup file in a format that may be more sustainable in the long-term. This may be carried out either on the artist’s side or within the museum. A normalized file, in theory, may provide a) greater resilience against corruption if, for example, a more robust uncompressed codec is employed, b) greater operability and playback consistency if a more widely-implemented and/or self-documenting format is selected, and c) an additional means of access to the audiovisual content should the production format of the original file become obsolete and inaccessible, if a non-proprietary format with good disclosure or openness is chosen.
PROBLEMS IN PRACTICE
A number of key research questions were raised by the problems and inconsistencies encountered in daily practice at the Guggenheim. On many occasions we observed a discernable audiovisual discrepancy between an artist’s native master file and their provided preservation file. The research carried out by Phillips and Jarczyk revealed the shifts in color, luminance, and display size that can occur when a file is transcoded or normalized to particular formats. This led us to ask the following: what workflows and formats yield acceptable results in terms of maintaining the audiovisual properties of a source file, as well as the playback consistency of its transcodes? In what circumstances does normalization produce a more sustainable version of the work, and what criteria are met? And in what circumstances does normalization not lead to better sustainability, and why not?
We also encountered a number of files with insufficient self-documentation due to inconsistent or incomplete metadata, specifically lacking metadata related to the files’ color matrix information. Was this due to the limitations of the metadata extraction tools being employed or was this a symptom of the files themselves? In the case of the later, was this a consequence of the software used to create the file or the limits of the codec or container?
A third problem area we identified pertained to the inconsistent file playback on software players. We observed the same video file render differently in different technical environments, that is, on different machines, in different platforms, and with different software players (see fig. 1). What were the root causes of these shifts and discrepancies, and how do they relate to a) the viewing software, b) the component library or decoders present, c) the file’s self-descriptive metadata (or the lack thereof), d) the codec, and e) the container?
Lastly, we identified the lack of commonly implemented practices for monitoring and quality control for file-based video artworks as a major risk to their sustainability. There are currently no consistent practices within museum conservation for viewing and assessing files; each museum uses different workflows and tools. What metadata extraction tools and viewing software are most appropriate for objectively evaluating and characterizing files? Are there other analytical tools that should become part of the time-based media conservator’s tool kit? What do they each uniquely provide, and what are the limitations that collection caretakers should be aware of?
INTERVIEWS WITH SPECIALISTS
These issues and questions prompted us to carry out interviews with experts and specialists in the field of codec development, software engineering, and digital archiving. Our interviewees included:
-Eric Hoffert, Technologist and Co-founder of QuickTime
-Reto Kromer, AV Conservator and Restoration Scientist, owner of AV Preservation by reto.ch
-Jérôme Martinez, Digital Media Analysis Specialist, founder and president of MediaArea
-Erik Piil, Digital Archivist, Associate Conservator at the Kramlich Collection and New Art Trust
-Julien Reichel, Technology architect, part of the JVT for the development of H.264
-Dave Rice, AV archivist and technologist, moving image archivist at CUNY television
These interviews, conducted over the course of twelve months, allowed us to draw several important conclusions that guided our practical tests.
There was a general consensus that care must be taken in normalization to maintain the source file’s sample bit depth and chroma subsampling patterns or shifts in chroma or luminance may arise in the transcoded file (Rice 2015). Specifically, for formats like DV and HDV—with an 8-bit sample depth and that employ 4:2:0 or 4:1:1 subsampling patterns (depending on whether they are PAL or NTSC respectively)—the target format must be capable of maintaining these features or color shifts will occur. Likewise, because Apple ProRes 4444 does not employ any chroma subsampling, loss will invariably occur if this format is transcoded to a 4:2:2 format like v210.
Some formats like ProRes, DV, and HDV contain self-descriptive metadata that may be lost in a transcoding process. DV and HDV, for example, are compound encodings and thus contain video, audio, as well as data related to timecode, captioning, and camera or tape read metadata (Rice 2015 b). ProRes MOV files contain color matrix metadata which may be lost when transcoding to certain formats, and these may be integral to audiovisual and/or playback consistency (Rice 2015b).
All interviewees also shared the view that no one codec-container combination is currently suitable for the diversity of all file-based video that exists, nor satisfies every criterion of sustainability defined above. While some of the uncompressed codecs meet certain criteria such as robustness and openness, more obscure uncompressed formats that support 4:1:1 and 4:4:4 subsampling patterns are poorly implemented, and also key, self-descriptive metadata may be lost in transcoding.
Moreover, the dependency of a file on specific decoders in the system’s component libraries emerged as a major limiting factor affecting a file’s playback consistency and sustainability. In order for files to be read and displayed correctly, the computer on which the file is being viewed must have the necessary decoders installed. If these change or are missing, a file may not display or may be interpreted incorrectly.
TRANSCODING & PLAYBACK TESTS
Based on these insights, we sought to investigate what essential features and metadata are maintained or lost when transcoding between different codecs, wrappers, and with different tools, as well as the effects these variables have on color shifts and playback consistency. For our tests we chose ProRes 422 LT transcoded to 10-bit uncompressed 4:2:2 as a case study.
TEST 1: MAINTAINING AUDIOVISUAL CONSISTENCY WITH DIFFERENT TRANSCODING SOFTWARES
In our first test, our primary objective was to determine which transcoding tools maintain audiovisual consistency between the original file and its transcode, that is, produce a transcode without any color shifts. An Apple ProRes LT 422 MOV file consisting of ten seconds of SMPTE color bars was transcoded to v210 in a MOV wrapper using Final Cut Pro 6, MPEG Streamclip, Adobe Premiere Pro, QuickTime Pro 7 and FFmpeg (v3.0.3).
Command used in FFmpeg for transcoding to uncompressed 4:2:2:
$ffmpeg -i input -c:v v210 -c:a copy output.[mov]
To quantitatively evaluate any color shifts that occur through transcoding, we compared the RGB values of the left-most gray color bar in each file, measured using Apple’s Digital Color Meter Utility (v4.4) in Display native values mode (fig. 2).
When viewed in QuickTime Player 7 all transcodes exhibited RGB value shifts, except for the transcode produced in Final Cut Pro 6; in the source file, the RGB values of the left-most color bar measured 135, 135, 135 and this was only maintained in the Final Cut Pro 6 transcode while the others read 123, 123, 123. Moreover, in addition to RGB value shifts, the transcode created with QuickTime Pro 7 displayed at a slightly scaled size, although the file itself remained at 1920 by 1080 pixels. The same results that occurred in QuickTime Player 7 were observed in QuickTime Player 10. When viewed in VLC Media Player (v2.0.9), the left-most gray color bar measured 122, 123, 122 in the original, as well as each of the transcodes (see table 1, Environment 1).
TEST 2: MAINTAINING AUDIOVISUAL CONSISTENCY IN DIFFERENT CONTAINERS
In our second test we investigated how different containers maintain audiovisual consistency between the original file and the v210 transcodes. Files were again viewed in QuickTime Player 7, QuickTime Player 10, and VLC Media Player (v2.0.9). We compared the original source file with a v210 MOV transcode produced in Final Cut Pro 7, as well as three transcodes produced with FFmpeg (v3.0.3) wrapped in MOV, AVI, and MKV (fig. 3).
FFmpeg command used in test 2 for transcoding to uncompressed 4:2:2:
$ffmpeg -i input -c:v v210 -c:a copy output.[mov/avi/mkv]
When viewed in QuickTime Player 7 and 10, neither the FFmpeg transcodes wrapped in AVI and MKV, nor the MOV produced with Final Cut Pro 7 exhibited RGB value shifts. A shift was measured in the FFmpeg-produced v210 MOV (fig. 3). In VLC, we observed no RGB value shifts between the original file and any of the transcodes, but the RGB values were different from those of the original file when viewed in QuickTime Player 7 and 10 (see table 1, environment 1).
In an effort to establish whether a specific workflow and container/codec combination yields a more sustainable normalized file with adequate and correct self-documentation, we also examined and compared the metadata of each of these files. Metadata were inspected with MediaInfo (command line tool and GUI v0.7.74), and Invisor (v3.7). The v210 MOV produced with Final Cut Pro 7 was the only file that retained metadata related to color matrix coefficients, transfer characteristics, as well as the date encoded and date tagged as seen in figure 4.
Surprisingly, this information was not present for the MOV, AVI and MKV v210 transcodes created with FFmpeg, although no shifts in color were observed in the AVI and MKV file. Further analysis of the v210 MOV transcodes with Atom Inspector revealed that the ‘colr’ atom was written only in the Final Cut Pro 7 transcode and not in the FFmpeg transcode (see fig. 6).
TEST 3: IMPACT OF THE ‘COLR’ ATOM ON MAINTAINING AUDIOVISUAL CONSISTENCY
In the previous test we measured RGB value shifts for the v210 MOV transcode created with FFmpeg and we also observed that it was missing its ‘colr’ atom. The ‘colr’ atom is a chunk of data specific to QuickTime formats that, in the case of video, includes the color parameter type (‘nclc’) and three indexes consisting of “a table of primaries, a table of transfer function coefficients, and a table of matrixes” (Apple Inc. 2015). This atom allows for the video file to be rendered in the appropriate color space, which, for HD video, is defined by the ITU-R Recommendation BT.709. As Dave Rice (2015) has noted, “with the absence of ‘nclc’ data, QuickTime applications defer to use SMPTE-C/BT.601 to convert YUV to RGB, which would give incorrect colors if the intention were to use EBU PAL or Rec. 709.”
In this test we employed FFmpeg’s experimental -movflags +write_colr option—a recently implemented flag that copies the ‘colr’ atom of the source file—to determine whether the RGB value shifts were a result of the QuickTime Player’s incorrect interpretation of the color space due to this missing ‘nclc’ tag. As Dave Rice explained in our interview: “the ‘nclc’ atom is only occasionally required within the QuickTime container. Sometimes the corresponding information is not provided or known. Also the QuickTime muxer of FFmpeg only recently added support for writing the ‘nclc’ atom (see the –write_colr option).” (Rice, written interview with authors, November, 2015).
The subsequent transcode produced with the addition of the -movflags +write_colr option into the command yielded a file that maintained the information contained in the ‘colr’ atom, visible both in Invisor (fig. 5) and Atom Inspector (v.1.0.3) (fig. 6). Moreover, no RGB value shifts were detected in QuickTime Player 7 and 10, confirming that the RGB value shifts measured previously were a result of QuickTime’s incorrect interpretation of the file in the absence of this information. Additionally, we found the -map_metadata g option necessary to preserve the metadata related to the ‘encoded date’ and ‘tagged date’ fields. Using following FFmpeg command:
$ffmpeg -i input -c:v v210 -c:a copy -movflags +write_colr -map_metadata g output.mov
TEST 4: PLAYBACK CONSISTENCY ON COMPUTERS WITH DIFFERENT OPERATING SYSTEMS, SOFTWARE VERSIONS, AND COMPONENT LIBRARIES
In our fourth test we introduced another variable: the impact of differing technical environments on audiovisual and playback consistency (see table 1). Test 1 and Test 2, carried out on a Mac running OS X 10.7.5 (Lion), were repeated on three other computers: one running OS X 10.7.5 (Lion), and two others running Mac OS X 10.10.5 (Yosemite). QuickTime 7 version 7.6.6 was installed on all four computers; Environments 1 and 2 were running QuickTime 10 version 10.1 while Environments 3 and 4 were running version 10.4. Three different versions of VLC were tested, up to and including the latest build, version 3.0.0.
In order to investigate how the presence of certain decoders (or the lack thereof) affected playback consistency, we examined the component library on each computer (in the folder ~/Library/QuickTime) and employed the command qt_thing, developed by David Van Brink as part of the qt_tools suite, to extract a list of the QuickTime components installed on each machine (Van Brink 2018).
The four technical environments tested were different from each other in the number of library components present in each OS (see table 1). However, only Environment 1 was uniquely different from the other three in terms of the number of 10-bit uncompressed components installed; the qt_thing command showed that there were eight v210 components present on Environment 1—two encoders and six decoders—compared to the four uncompressed components present on the other technical environments (see fig. 7).
When the seven video files were played back, the following observations were made: in Environment 2, RGB value shifts were measured for the FFmpeg v210 AVI file when viewed in QuickTime Player 7, which were not observed when viewed in the other three environments. In Environment 2, no RGB value shifts were measured for the transcodes produced with Final Cut Pro 6 or Final Cut Pro 7 when the files were played back in QuickTime Player 7 and 10. Finally, interoperability issues arose when the transcode wrapped in MKV was played back in QuickTime Player 7 and 10.
When played back in QuickTime Player 10, in both Environments 3 and 4, we measured RGB value shifts in the test files transcoded with FFmpeg (both in MOV and AVI containers) and with Final Cut Pro 6. These shifts did not appear when the files were played back in QuickTime Player 7. No RGB value shifts occurred for the transcode produced in Final Cut Pro 7 when viewed both in QuickTime Player 7 and 10. Across all environments, in the original file and all transcodes, the RGB values of the left-most gray color bar measured consistently 122, 123, 122 when viewed in VLC.
INTERPRETATION OF FINDINGS
Metadata information related to the BT.709 specification for color primaries, matrix coefficients, and transfer characteristics together with encoded and tagged date were preserved in the transcode created with Final Cut Pro 7 and no RGB value shifts were measured in QuickTime Player 7. The same consistency was maintained in QuickTime Player 7 for the v210 MOV file produced with FFmpeg using the -movflags +write_colr -map_metadata g options.
In general, audiovisual and playback inconsistencies were observed more so in QuickTime Player 10 than in QuickTime Player 7. The consistent RGB value measurements in VLC for the ProRes file and all v210 transcodes in all tested environments and software versions differed from those measured in QuickTime Player 7 and 10. The values observed in VLC are similar to those observed in the QuickTime Players when the ‘colr’ atom was missing, suggesting that the shifts may be due to the incorrect interpretation of these versions of VLC, which seem to default to the BT.601 color matrix specification.
Our tests confirmed that the amount of self-descriptive metadata maintained in a transcoding is very much dependent on the codec and container of both the source and target files. Moreover, both the software and the settings or options employed play a key role in retaining metadata, when exporting from an editing timeline, transcoding, or rewrapping. In the case of ProRes LT 422, our tests revealed a correlation between audiovisual consistency and the retention of metadata related to color primaries, matrix coefficients, and transfer characteristics contained in the ‘colr’ atom; the v210 MOV transcodes produced with Final Cut Pro 7 and FFmpeg (using the -movflags +write_colr -map_metadata g option) retained this metadata and no RGB value shifts were measured in QuickTime Player 7.
When the transcodes were viewed in QuickTime Player 7 and 10, we observed no discrepancy in RGB values across identical technical environments (with the same OS) when the same decoders (v210) and software versions were present, excluding the v210 AVI transcode. We were unable to determine the root causes of the RGB value shifts observed for the v210 AVI transcode played back on Environment 2 in QuickTime 7 and more broadly in QuickTime 10 across the environments. This observation could be linked to Dave Rice’s findings (2015) that even if two computers have the same OS and QuickTime version, playback inconsistencies may arise when the component libraries differ.
Our tests also confirmed that chroma subsampling pattern and bit depth are specific to codecs and must be maintained in transcoding or color shifts may arise; in separate tests, not presented here, we found that color shifts occur when H.264, DV, HDV, and ProRes 4444 files are transcoded to 4:2:2 codecs like v210. We found that v210 in a MOV wrapper may however be a suitable preservation format for ProRes 422 codecs if a particular workflow is followed. Having both good disclosure and resilience, v210 satisfies several sustainability criteria that ProRes does not. We found that v210 in a MOV container can also maintain the same level of self-documentation, i.e. the self-descriptive metadata that existed in the original file. However, it should be stressed that 10-bit uncompressed 4:2:2 is not suitable for transcoding all kind of video files, particularly those formats that do not employ chroma subsampling like ProRes 4444, or have embedded timecode data like DV, DVCPRO HD, and HDV.
Overall, our practical tests demonstrated how viewing software, software versions, operating systems, component libraries, and a file’s metadata can each play a role in the correct and consistent rendering of a video file.
NORMALIZATION AND BEYOND
Due in large part to the complexity and diversity of file-based video that has been created in the last twenty years, there is still no single preservation format capable of satisfying every criterion of sustainability. In many cases normalization entails a compromise. That said, this may change in the coming years if resources are focused on developing and implementing formats that satisfy more of these criteria. Bearing in mind that certain formats cannot be normalized to a well-implemented codec without manipulating chroma subsampling and bit depth, and thus introducing color shifts, further efforts should be undertaken to explore viable solutions for their preservation. In particular, further research should investigate the sustainability of DVCPRO HD (8-bit 4:2:0) and HDV (8-bit 4:2:0/4:1:1), two relatively-short-lived HD formats that employ anamorphic pixels, comparing both the changes and advantages that normalization to square-pixel formats entails.
We believe close attention should be paid to the Advanced Media Workflow Association’s efforts to develop the AS-07 specification for MXF (AMWA 2016), which will allow for native encoded DV bitstreams or MPEG transport streams and their metadata to be preserved and mapped to the MXF wrapper. Additionally, the open source container Matroska, developed by Steve Lhomme, seems to be promising in its capacity for storing self-descriptive metadata and frame-level fixity data. The lossless, open source codec FFV1 should also be investigated in future tests.
DOCUMENTING AND COLLECTING COMPONENT LIBRARIES
As long as the consistent viewing experience of file-based video artworks relies on external dependencies such as particular display hardwares, viewing softwares, and decoders, these works remain vulnerable to changes that may be perceived as losses. Like with many time-based media artworks, the preservation of one element in the system such as the file does not necessarily result in the preservation of the work as a whole (note 1).
Given the importance of decoders in maintaining access and playback consistency of video files, further research should explore the viability of documenting and collecting component libraries from artists as a possible preservation strategy for museums. As demonstrated in our tests, the qt_thing command in qt_tools is useful for documenting QuickTime components present on a Macintosh system, however, this tool has not been updated since 2008, and this may cause problems in the future for its use and adoption.
The Pericles Extraction Tool—an open-source tool based on Java software under development within the PERICLES Project—will hopefully offer a way to capture and document the components in use when playing back a video file in a specific playback environment in the future (Corubolo et al. 2014). Such a tool could potentially allow for the comprehensive documentation of the native production and viewing environment of a file-based video artwork at the point of acquisition. Further development should allow users to determine a) where individual components are located within the system, b) the software versions they correspond to, c) the date they were installed, and d) the decoder in use during playback of a video file. Running such a tool at the point of acquisition would of course require greater collaboration between collection caretakers and artists, but this remains an admirable goal.
TOOLS FOR VIEWING AND ASSESSING FILES
The DigitalColor Meter Utility proved useful in identifying audiovisual inconsistencies between source files and rewrapped or transcoded files. In the course of our research, we were also able to identify several other software tools useful in the examination and technical analysis of file-based video. The open source metadata extraction tools developed by the MediaArea team – including MediaInfo and MediaConch – were enormously helpful in our testing, and they have emerged as essential analytical tools for caretakers of file-based video. Invisor—a GUI tool built on MediaInfo—allows for side-by-side metadata comparison of up to twenty files. In our research it proved to be an extremely useful tool for comparing metadata between files and their derivatives. The software has been updated recently (May 19, 2016) to Version 3.7 with the latest MediaInfo library.
To date, no standard viewing software has been adopted for evaluating file-based video and this remains a problem for the field. Within the scope of our tests we were unable to determine whether VLC is currently an appropriate tool to objectively assess and characterize the audiovisual properties of video files. But according to Felix Paul Kühne at VideoLAN (E-mail correspondence, August 7, 2016), there are efforts underway to implement ‘colr’ support into the latest version of VLC (v.3.0.0).
Given the proprietary nature of the QuickTime Players and Apple’s announcement in 2016 that it will drop support for them in Windows due to security issues (US-CERT 2016), there is an urgent need for a non-proprietary, widely-adopted viewing software that correctly renders a video file across a range of technical environments. Assuming it is provided with regular and long-term maintenance, such a tool would allow caretakers to reliably characterize digital video files entering collections and would also allow access to the variety of native video file formats in museum collections around the world for many years to come.
CONCLUSION
During the course of this research it became clear to us that there were no simple or easy answers to our questions pertaining to the long-term sustainability of file-based video artworks. This is due in large part to the complex and ever-evolving digital landscape into which these works were born. The threat to their long-term sustainability posed by the unannounced redesign and drop of support for obsolete formats is real. These risks are compounded by a lack of implemented practices and digital tools for characterizing and evaluating digital video artworks and their native viewing environment. That said, many efforts are underway to address these issues.
Those of us in the museum world caring for file-based video assets must therefore make our interests and concerns known by working more closely with the archival community in developing their open source projects like FFmpeg, VideoLAN’s VLC player, and MediaArea’s metadata extraction and quality control tools; participating in meetings of standards bodies such as the Society for Motion Picture and Television Engineers (SMPTE) and the Motion Picture Experts Group (MPEG); and by reaching out to industries and software developers like Apple and others. Rather than relying solely on the efforts of those in adjacent fields and industries to develop tools and protocols for us to borrow, we should be actively engaging and collaborating with these experts and specialists. Only then will we be able to establish and maintain our own best practices of care around file-based video art.
ACKNOWLEDGEMENTS
We are very grateful to our supervisors Agathe Jarczyk and Joanna Phillips for providing guidance and feedback at each stage of our research, and for facilitating this collaboration. We also thank our interviewees—Eric Hoffert, Reto Kromer, Jerome Martinez, Erik Piil, Julien Reichel, and Dave Rice—for their critical insights and participation in our research. Finally, we are grateful for the generous support of the Bern University of the Arts, the Solomon R. Guggenheim Museum, the Samuel H. Kress Foundation, and the European Union’s Horizon 2020 Actions under the Marie Skłodowska-Curie Grant no. 642892.
NOTES
- As Dave Rice (2015) indicates, “Sustaining video presentations through emulation requires maintaining a player and all of its dependencies. For instance, if a creator determines that a video is intended for presentation through QuickTime Pro 7 this may mean preserving QuickTime Pro 7 as an application along with its underlying set of components as well as an operating system that can support QuickTime Pro 7’s underlying 32 bit QtKit framework. Often video players rely on system codec libraries or audio-visual frameworks working behind the scenes, so it can be challenging to determine what exactly is necessary to document emulation sufficiently.”
REFERENCES
The Advanced Media Workflow Association (AMWA). 2016. MXF for Archive & Preservation: AS-07. http://amwa.tv/projects/2016-07-06(AMWA%20AS-07%20handout,%20A4).pdf (accessed 08/18/16).
Apple Inc. 2002. QuickTime File Format. Cupertino: Apple Computer Inc. https://developer.apple.com/standards/qtff-2001.pdf (accessed 08/18/16).
Apple Inc. 2009. Final Cut Pro 7 Professional Formats and Workflows. Cupertino: Apple Computer Inc. http://operationalportal.com/wp-content/uploads/Final-Cut-Pro-7-Professional-Formats-and-Workflows-en.pdf (accessed 08/18/16).
Apple Inc. 2012. QuickTime File Format Specification. Cupertino: Apple Computer Inc. https://wikileaks.org/sony/docs/05/docs/Apple/qtff.pdf (accessed 08/18/16).
Apple Inc. 2014. Apple ProRes: White Paper. Cupertino: Apple Computer Inc. www.apple.com/final-cutpro/docs/Apple_ProRes_White_Paper.pdf (accessed 08/18/16).
Apple Inc. 2015. Media Data Atom Types. https://developer.apple.com/library/mac/documentation/QuickTime/QTFF/QTFFChap3/qtff3.html (accessed 08/18/16).
Blood, G. 2011. Refining Conversion Contract Specifications: Determining Suitable Digital Video Formats for Medium-term Storage. www.digitizationguidelines.gov/audio-visual/documents/IntrmMastVidFormatRecs_20111001.pdf (accessed 08/18/16).
Cherna, T., P. Hoddie, and C. Pirazzi. 1999. Technical Note TN2162: Uncompressed Y′CbCr Video in QuickTime Files. https://developer.apple.com/library/mac/technotes/tn2162/_index.html (accessed 08/18/16).
Corubolo, F., A. Eggers, A. Hasan, M. Hedges, S. Waddington, and J. Ludwig. 2014.A pragmatic approach to significant environment information collection to support object reuse. Proceedings of the 11th International Conference on Digital Preservation (iPres14). State Library of Victoria.
Dance Heritage Coalition, Inc. and Media Matters, LLC. 2004. Digital Video Preservation Reformatting Project: A Report. www.danceheritage.org/digitalvideopreservation.pdf (accessed 08/18/16).
Escobedo, K. 2011. Digital Video Preservation: Identifying Containers and Codecs. https://siarchives.si.edu/blog/digital-video-preservation-identifying-containers-and-codecs (accessed 08/18/16).
Escobedo, K. 2013. Digital Video Preservation: Continuing the Conversation. https://siarchives.si.edu/blog/digital-video-preservation-continuing-conversation (accessed 08/18/16).
Federal Agencies Digitization Guidelines Initiative (FADGI). 2014.Digital File Formats for Video Tape Reformatting. www.digitizationguidelines.gov/guidelines/video_reformatting_compare.html (accessed 08/18/16).
Fleischhauer, C. and FADGI. 2011. Preservation Video File Format Issues and Considerations: What Have We Encountered as We Drafted MXF AS-AP? www.digitizationguidelines.gov/guidelines/FADGI_MXF_ASAP_Issues_20110815.pdf (accessed 08/18/16).
Fleischhauer, C. 2011. Format Considerations in Audio-Visual Preservation Reformatting: Snapshots from the Federal Agencies Digitization Guidelines Initiative. Information Quarterly 22 (2, 2010): 35-40. www.digitizationguidelines.gov/audio-visual/documents/IP_Fleischhauer_AudioVisual_Reformatting_isqv22no2.pdf (accessed 08/18/16).
Goldsmith, B. 2013. Digitizing Video for Long-Term Preservation: An RFP Guide and Template. Preservation and Conservation Department, New York University Libraries. www.scart.be/?q=en/content/digitizing-video-long-term-preservation-rfp-guide-and-template (accessed 08/18/16).
Jarczyk, A. and J. Phillips. 2014. Life after Tape: Collecting Digital Video Art. Talk delivered at the Electronic Media Group Session of AIC’s 42nd Annual Meeting “Conscientious Conservation: Sustainable Choices in Collection Care”, San Francisco.
Lacinak, C. 2010. A Primer on Codecs for Moving Image and Sound Archives. www.avpreserve.com/wp-content/uploads/2010/04/AVPS_Codec_Primer.pdf (accessed 08/18/16).
Library of Congress. 2013. Sustainability Factors. www.digitalpreservation.gov/formats/sustain/sustain.shtml (accessed 08/18/16).
Lorrain, E. 2014. A short guide to choosing a digital format for video archiving masters. SCART by PACKED vzw. www.scart.be/?q=en/content/short-guide-choosing-digital-format-video-archiving-masters (accessed 08/18/16).
Matroska (2018). Matroska Media Container. https://www.matroska.org/ (accessed 04/07/19).
MediaArea (2019). MediaInfo. https://mediaarea.net/de/MediaInfo (accessed 04/07/19).
The National Archives. 2008. Selecting File Formats for Long-Term Preservation.www.nationalarchives.gov.uk/documents/selecting-file-formats.pdf (accessed 08/18/16).
National Library of the Netherlands. 2008.Evaluating File Formats for Long-Term Preservation. www.kb.nl/sites/default/files/docs/KB_file_format_evaluation_method_27022008.pdf (accessed 08/18/16).
Sacredote, A. and L. Sorensen. 2009.Codec Comparison for the digital Preservation for Analog Video. Proceedings of the Electronic Media Group Session, AIC 35th Annual Meeting May 15-20, 2009, Los Angeles. The Electronic Media Review (1, 012). Washington D. C.: AIC. 59-70. www.bavc.org/sites/default/files/resource/codecs_sacerdotesorensen.pdf (accessed 08/18/16).
Poynton, C. 2012. Digital Video and HD: Algorithms and Interfaces. Waltham: Elsevier.
Pozdeev, M. 2019. Invisor. https://www.invisorapp.com/ (accessed 07/07/19).
Rice, D. 2012. Reconsidering the Checksum for Audiovisual Preservation: Detecting digital change in audiovisual data with decoders and checksums. IASA Journal 39.
Rice, D. 2015. Sustaining Consistent Video Presentation. www.tate.org.uk/research/publications/sustaining-consistent-video-presentation (accessed 08/18/16).
Rice, D, 2015b. Interview by Bunz, Sophie, Castriota, Brian & Fortunato, Flaminia. Research Towards the Preservation of Digital Video Artworks: File Sustainability and Risk Factors. Written Interview, November 2015.
Richardson, I. 2010. The H.264 Advanced Video Compression Standard. Chichester: Wiley.
United States Library of Congress and National Digital Information Infrastructure and Preservation Program (NDIIPP). 2005. Sustainability of Digital Formats: Planning for Library of Congress Collections. www.digitalpreservation.gov/formats/index.shtml (accessed 08/18/16).
US-CERT. 2016. Alert (TA16-105A): Apple Ends Support for QuickTime for Windows; New Vulnerabilities Announced. https://www.us-cert.gov/ncas/alerts/TA16-105A (accessed 08/18/16).
Van Brink, D. 2018. QT_TOOLS. http://dvb.omino.com/sw/qt_tools/ (accessed 01/25/19).
Sophie Bunz, MA Student
Conservation-Restoration Modern Materials and Media
Bern University of the Arts
bunzsophie@gmail.com
Brian Castriota
Samuel H. Kress Fellow in Time-Based Media Conservation
Solomon R. Guggenheim Museum
bcastriota@gmail.com
Flaminia Fortunato, MA Student
Conservation-Restoration Modern Materials and Media
Bern University of the Arts
flaminia.fortunato7@gmail.com