Julia Betancor, Daniel Finn, Alvar García, and Ana L. Mata
Electronic Media Review, Volume Six: 2019-2020
ABSTRACT
This article introduces an advanced preservation method for artworks created using artificial intelligence, a medium that poses new challenges for art conservators. With technology-based artworks, it is necessary to find a balance between respecting original artwork components, mitigating obsolescence caused by the dizzying speed of technological development. An interdisciplinary art conservation team convened to discuss how to preserve Mario Klingemann’s Memories of Passersby I (2018). An artist interview identified the significant behaviors and characteristics of the artwork that are required for a successful iteration of the piece. Using this information, the team developed an outline of conservation strategies for an array of possible scenarios. Looking forward to the potential need for migration, the team successfully created a virtualized container of the artwork using the program Docker. There is also a discussion of the legal concerns surrounding the artwork.
KEYWORDS: Clustering Art Conservation Laboratory, Artificial intelligence, Time-based media art, Machine learning, Neural networks, Pattern, Technology conservation, Protective relaying, Testing, Training, Interdisciplinary, Digital, Virtual, Containerization.
Introduction
This article examines a case study of an artwork by Mario Klingemann, Memories of Passersby I, created with artificial intelligence (AI), a powerful and emerging tool for new media artists (fig. 1 ).
Talking about deep machine learning, AI, algorithms, neural networks, and generative adversarial networks (GANs) is not easy in the world of art conservation; for many of us, these terms sound like something from a science-fiction movie. So, let us try to demystify this vocabulary. Are computers a mere tool to be used by the artist? How can we preserve the algorithms at work as the machine ages? To someone who is not a computer engineer, this artwork may seem alien, but to AI art pioneer Mario Klingemann, algorithms are artistic tools, just like pen and paper.
Conserving an AI artwork over the long term is a challenge. Multiple interviews with the artist identified the loss of the algorithm as the most significant risk facing Memories of Passersby I. Concerns regarding hardware obsolescence were of secondary importance to the artist. AI is permeating many fields, and it is time to include it in our vocabulary as art conservators. In this case study, developing a nuanced understanding of the artwork and the behavior of the algorithm was crucial to developing appropriate conservation strategies.
Description of the Artwork
Memories of Passersby I by Mario Klingemann
Portraiture Meets Convulsive Beauty
Memories of Passersby I—Companion Version (2018) uses a complex system of neural networks to generate a never-ending stream of portraits displayed on two 4K monitors. These are disquieting visions of non-existant people, male and female faces created by a machine.
Unlike earlier generative art installations, Memories of Passersby I does not contain a database. It uses an AI algorithm, developed, and trained by Mario, that creates brand new portraits, pixel by pixel, in real time. The outputs displayed on screen are not random or programmed combinations of existing images but unique, AI-generated artworks.
The flow of images presented does not follow a predefined choreography but is the result of the AI interpreting its own output. The complex nature of this feedback loop means that no image will ever be repeated. Memories of Passersby I contains all of the algorithms and GANs necessary to produce an endless succession of new images as long as it is running. In this sense, Memories of Passersby I marks a significant step forward in the rapidly emerging field of AI art. Up until now, collectors have been able to acquire human-curated outputs of neural networks; Memories of Passersby I is a self-contained creative agent.
To develop Memories of Passersby I, Mario trained his AI model using thousands of portraits from the 17th to 19th centuries. He created a Tinder-like application to accelerate the learning process and teach the machine his aesthetic preferences, influenced by surrealist figures such as Max Ernst.
For the viewer, Memories of Passersby I is a hypnotic experience, the opportunity to watch an AI brain “think” in real time and view unique portraits that are neither recorded nor saved. The artwork is presented as an installation piece: the AI machine is housed in a custom-made chestnut wood cabinet for this purpose, connected to two framed screens (fig. 2).
ONKAOS/Colección SOLO
The artwork was commissioned by ONKAOS, a project set up to support artists working with new technologies and one branch of Colección SOLO, a private art collection based in Madrid. ONKAOS assists represented artists in promoting their creations across the globe, bringing them to the attention of collectors and institutions.
Colección SOLO is an international collection of contemporary art housed at a private museum (Espacio SOLO) located in Puerta de Alcalá, Madrid. Created by Spanish entrepreneurs Ana Gervás and David Cantolla, Colección SOLO comprises a growing art collection, museum, and numerous art patronage programmers.
Conservation
The Intersection of Art, Technology, and Conservation
The Colección SOLO Media Art Conservation Lab was composed by a digital lawyer, a technical engineer, an ATMB conservator and Julia Betancor, the author, functioning as manager and coordinator. We have mentoring from the director of ONKAOS. As the head of art conservation at Colección SOLO, it initially seemed difficult to face this challenge without investing in expensive and sophisticated resources; a simple solution for AI art conservation did not appear to be an easy task. However, through a series of informal interviews, the collaboration of Mario Klingemann became key to our decision making process, helping us identify his requirements and understand the way he wants his artwork to age (fig. 3).
For example, Mario, in an exercise of honesty and simplicity, confirmed to us that the box, the pieces, and the screens are only external resources that the algorithm uses to function (fig. 4). They are extrinsic, in this case, to the work.
AI and machine learning are similar but not equivalent concepts. What could be more complex than this? GANs like those used in Mario’s work are based on two algorithms that compete in each iteration to be the best. The pieces are the generative algorithm, the training set, and the validation criteria.
References and Advisor
Our team sought the opinion of Smithsonian American Art Museum (SAAM) collection’s media conservator, Dan Finn, based on his experience with similarly complex software-based installations such as Nam June Paik’s Megatron/Matrix (1995), Jenny Holzer’s For SAAM (2007), and Leo Villareal’s Volume (Renwick) (2015) (fig. 4).
His primary contribution to the project was to offer guidance on documentation strategies. SAAM’s documentation procedures track change in complex artworks through the use of identity reports and iteration reports. Change in complex media works can result from relocation, component wear, technology migration, and many other factors. So instead of focusing on the components of a work as the objects of conservation, it is sometimes better to focus on the work’s overall behavior. The concept of “identity” is employed in this way and has been quite influential in our field. With each object, conservation can create an identity report specially adapted to the work of art in question. The identity report of a work comprises all significant characteristics that must be maintained in a successful presentation, while an iteration report exhaustively documents a specific instance of an artwork, such as each time the work is exhibited. The overarching identity report and the periodic iteration reports work in tandem to track the change in an artwork and ensure that new iterations can respect the artwork’s identity.
These documentation methods helped organize the information the team gathered and served to generate several lines of questioning in the artist interviews. Templates of SAAM’s conservation reports, as well as templates of reports from other Smithsonian Institution museums, are available at https://www.si.edu/tbma (fig. 5).
A section of the identity report that proved especially relevant to this project is the discussion of risk and conservation recommendations. Discussions between the artist and the team helped develop a number of potential conservation strategies based on various possible hardware failures.
Containerization Procedure for Memories of Passersby I
One area that seemed especially promising was the possibility of containerizing the work. A container is a standalone, executable package of software and its dependencies (e.g., code, runtime, system tools, libraries and settings). Alvar García successfully developed a procedure to containerize the work using Docker, a development tool that runs on Linux (fig. 6.).
For a generative AI work, the system can be represented by what is called the disk image, a bit-for-bit copy of the content on the source drive. If the disk image safely stored away from technological and geological disasters or even domestic accidents like the typical coffee spilled on the proverbial keyboard, then the system can theoretically live forever.
Disk images can be run using emulators, virtual machines, and containers, which isolate algorithms and make them work outside of their original element, on other machines, in other programming languages, in other (future?) times, through the study or reverse engineering of systems that were never designed to be studied.
One essential quality about Memories of Passersby I is its tempo. Therefore, it was important to ensure that the rendering and transition pace is the same as the original artwork. This is significant since the graphics card used for the tests was almost 45% faster than the one used in the piece. Advancements in technology typically aim to make computers work as fast as possible, but emulators know how to respect the original machines. By containerizing Mario’s work, it can be displayed using new technology—if the new environment does not respect the work and its cadence, measures can be taken to make the hardware slow down.
Legal System
Alvar García conducted the containerization procedure, considering the license agreement that mentions the integrity of the artwork, in particular, a clause stating: “the Owner acknowledges that the Artist’s moral right to protect the Artwork’s integrity prevails.” The containerization procedure is respectful of this agreement, as it does not require looking at the internal workings of the artwork. Only a superficial trial and error approach is used to gather the dependencies of the code and build a container image that is isolated from the host computer.
Within the framework of such an innovative project, AI does not only represent a challenge from the point of view of artistic conservation as we have seen so far. The legal system is also without clear answers when faced with such new technologies. These circumstances need swift, creative responses, as it is clear that AI has come to stay. The lack of laws regarding AI together with the international success of Memories of Passersby I result in additional difficulties when assessing the legal implications connected to its conservation.
AI-driven solutions, including predictive tools, raise legal questions linked to liability, ethics, transparency, and, above all, to the ownership of the data that has been used to train the algorithm powering the solution. Algorithms are examined to determine their accuracy and fairness. The aim is to identify possible biases, as well as potential discriminatory results, and, finally, to measure the protection of privacy and security measures contained in their design.
However, the artistic context forces us to make a qualitative leap and significantly limits the scope of the common legal approach to AI. Here it is not a question of ethics and protection against potential discriminatory effects when using AI but a matter of maintenance and ownership of the data in a multi-jurisdictional context.
Mario managed to avoid the questions that could arise in relation to intellectual property rights and ownership of the data used to train the algorithm using old master paintings. The license that accompanies the acquisition of the art piece can easily warranty that the images used to train the algorithm and the development of the algorithm do not infringe the intellectual property rights of third parties and comply with the much-feared data protection regulations. It would be quite different if, for example, photographs of faces had been used to feed the algorithm.
Furthermore, the legal responsibility to provide maintenance and technical assistance will be essential for the owner of the work to keep Memories of Passersby I in optimum condition.
Technological Sheet: Memories of Passersby I
The technological data of Memories of Passersby I is presented below, indicating potential challenges associated with each technology due to its obsolescence level or the linkage to a specific hardware or software brand.
The reason for putting the software of the artwork in a container is to reduce the effects of obsolescence and the dependency on specific technologies, although, by its very nature, some dependencies remain, as indicated below.
OPERATING SYSTEM: Ubuntu 16.04 LTS (Long-Time Support)
This operating system has been obsolete for 2 years since the release of Ubuntu 18.04 LTS and also 20.04 LTS. As its name (Long-Time Support) suggests, it will continue to receive support until April 2021, however, its use of Python 2 makes it particularly vulnerable to modifications.
Auxiliary packages of the operating system with specific versions:
- TCL/TK 8.6
Auxiliary packages of the operating system without specific versions:
- libbz2-dev
- xorg-dev
Tools from this package are used to connect the container’s graphical output with the host system’s graphical output, making it a technological dependency of the container image. Not only must these tools continue to exist, but the host operating system must also continue to allow this connection. Part of this functionality is also carried out by the NVIDIA Container Runtime extension in the host system.
- unzip
- cmake
- mesa-utils
- libglvnd0
- libglfw3
- libglfw3-dev
- libssl-dev
Python 2.7.15
The artwork uses an obsolete version of Python, released on May 1, 2018, and has already received subsequent updates. Python 2 is no longer supported on all operating systems and is only maintained because it is critical for the operation of many Linux systems (e.g., Ubuntu).
Python libraries without specific versions:
- torch
- torchvision
- future
- All of the above libraries are technological dependencies of the procedure.
Compute Unified Device Architecture (CUDA): 9.2
The artwork uses the GPU programming system CUDA, which ties the artwork irrevocably to NVIDIA graphics cards, as it is a private, closed-source system incompatible with other GPUs.
Technological dependency of the container image: NVIDIA GPU
Different versions of CUDA are associated with specific models of graphics cards. For example, the previous generation of RTX 20XX graphics cards (Turing architecture) and the current generation RTX 30XX (Ampere architecture) are not compatible with CUDA 9.2. Although all versions of CUDA have been backward compatible with software developed for previous versions, this may not be the case in the future. The artwork has been placed in the container using CUDA 10.2 without issues.
Technological dependency of the container image: CUDA
As a base for the container image, one provided by NVIDIA with CUDA 10.2, and all necessary libraries have been used. However, since that base image, with the indicated characteristics, may disappear at any time, it becomes another technological dependency of the procedure.
- Ubuntu 18.04 LTS
- OpenGL (CudaGL)
- CUDA 10.2
CUDA requires specific versions of NVIDIA drivers on the host system where the container is executed, as indicated in the image nvidia/cudagl:10.2-base-ubuntu18.04.
Although the graphics card model Tesla is specified, it is not practically necessary:
“cuda>=10.2 brand=tesla,driver>=396,driver<397 brand=tesla,driver>=410,driver<411 brand=tesla,driver>=418,driver<419 brand=tesla,driver>=440,driver<441”
The version of the graphics card drivers is a technological dependency of the container image.
Container System
The base image nvidia/cudagl:10.2-base-ubuntu18.04 relies on an extension to the container system called NVIDIA Container Runtime. This extension operates on any container system that complies with the Open Container Initiative (OCI) specification.
- Docker
- Runc
- Others
- The NVIDIA Container Runtime extension is a technological dependency of the container image.
- NOTE: It is not considered a technological dependency since it works independently on any system that adheres to the OCI specification.
Notes and Glossary
- Host system: The physical computer where the container is executed.
- Technological dependency of the procedure: Technological component required when putting the artwork back into a container.
- Technological dependency of the container image: Technological component required when running the container image generated during the procedure (file solitairecuda102.tar.gz).
Conclusions
In the field of technology, the first instinct is to make everything complicated. Gone are those times when Andy Warhol would color illustrations on a Commodore Amiga as a publicity stunt.
Despite the temptations, the best thing is always employing simplicity and transparency, because they allow understanding and rebuilding an algorithm or machine for a third party, usually without the help of the original creator.
It is not necessary to trust that any format or system will last more than between 10 and 20 years, however it is necessary to live in permanent evolution and improvement of the preservation of the master copy and the running copy.
We believe that we have created a replicable model that allows technological collections to breathe, and hope to appeal to all communities, museums, electronic teams and time-based media conservation labs to continue to create new additional resources and tools that help us all (fig. 7).
Let’s not forget to save. (As grandma says) those who save, can find.
ACKNOWLEDGEMENTS
We are grateful to A. Moro, M. Castillo Lorente, O. Hormigos, M. Chang Park, R. Jane Rhodes, A. Triano, Sanz, M. Klingemann, A. Gervás, D. Cantolla, P. Martínez, E. de Andrés, S. Passuello, Dr. J. L. de la Serna and Onkaos.
AUTHORS
JULIA BETANCOR is the head of art conservation for Colección SOLO and ONKAOS, and has 28 years of fine arts conservation and restoration experience, mainly in private practice, in Spain, London, and the United States. She holds a fine art degree and a master’s degree from Madrid Complutense University (UCM). Her expertise is in easel painting, and she has completed training in Florence, Instituto per l’arte e il Restauro. She has worked for Hamish Dewar’s LTD Fine Art Conservation London as a painting conservator and for the Spanish Royal Palace Cultural Heritage, Jesuit’s Company Cultural Heritage, Barrie de la Maza Arty Collection Foundation, Cultural Department Madrid City Hall, and Spanish Ministry of Defense, among others. She completed an internship at the Reina Sofía Art Conservation Department of the National Museum of Modern Art. She is a member of the AIC, ICOM-CC, IIC, and GE-IIC Board Committee (Spanish Group IIC). She hosts the podcast “Por Amor al Arte” (about art and conservation in Spanish) and is the co-founder of For the Love of Art.
DAN FINN is a media conservator at the Smithsonian American Art Museum, where he built the museum’s Media Conservation Lab and has refined conservation practices for time-based media art collections. He also serves as chair of the Smithsonian Institution’s Time-based Media and Digital Art Working Group. Previously, he assisted in establishing the Media Archive of the Smithsonian National Museum of African American History and Culture, and worked in media preservation at Democracy Now! the City University of New York Television Station, and the Academy of Motion Picture Arts and Sciences. He holds an MA from New York University’s Moving Image Archiving and Preservation program.
ALVAR GARCIA is an ICT developer with more than 15 years of experience in many positions in the information technology industry: marketing for large companies, consulting, operations, support, and people and project management. He is working with video games and making them work after their expiration date. This last ability is extensible to technology in general.
ANA LIZETH MATA DELGADO earned her degree in restoration from ENCRyM, INAH and a master’s degree in art history with a specialization in contemporary art from UNAM. She is a professor-researcher of the Workshop Seminar on Restoration of Modern and Contemporary Art and a member of the International Urban Art Group and the XX Century Art Group, and is currently linked to the GE-IIC. She is a member of the Steering Committee of the International Network for the Conservation of Contemporary Art.
ISABELLA GALEANO is the head of legal operations at Abertis Mobility Services, where she has the opportunity to advise on cutting-edge tech projects around the world. She is also a legal tech and innovation consultant and has worked on projects regarding artificial intelligence in the contemporary art industry, blockchain, and regulation of algorithms. She graduated from Esade Law School in 2010. Her professional practice has always been related to corporate law, M&A, and the business world. She has practiced in the corporate and M&A teams at Freshfields Bruckchaus Deringer, Gómez-Acebo y Pombo, and Uría Menéndez. She has also worked in the legal departments at Nike and Medtronics. In 2018, she graduated from an LLM at Georgetown University Law Center and passed the New York Bar in July 2018. Isabella is a frequent speaker in international legal innovation conferences and was the director of the legal Tech Programme at Esade Law School between 2019 and 2020.