Shu-Wen Lin
Electronic Media Review, Volume Six: 2019-2020
Introduction
Contemporary artist Jim Campbell is renowned for his innovative approach of integrating custom electronics with multimedia works. Opposing the high-resolution world we are currently situated in, Campbell’s blurry, pixelated moving images in his ongoing Low Resolution series (1999- ) forces viewers instead to focus on movement and shapes. Collected by the Smithsonian American Art Museum, Grand Central Station #2 (2009) illuminates moving images of passengers in the main terminal at Grand Central Station in New York City. It consists of a grid of nine LED panels as a display screen, nine FPGA (Field Programmable Gate Array) controlling electronics adhered to the back of each LED panel, a printed Plexiglas panel, and a HWS 300-15V power supply. In the artist’s own words, the controlling electronics are the brains, and the FPGA integrated circuits are the hearts. To communicate between non-digital electronic components employed in the artwork, the artist’s source designs were taken through a complicated optimization process where human-readable code is translated into specific binary forms to be loaded onto the circuitry. The LED matrix display with the adhered controlling electronics were collected as a self-contained unit comprising the information carrier, playback, and display equipment.
With a background in electrical engineering and mathematics, Jim Campbell designs and configures all the custom hardware for the unique purpose of his art works. Over the course of twenty years, the artist has demonstrated an early adoption of FPGA, a technology that requires highly specialized skills, and has worked consistently with the same technology. Unlike other mass-produced consumer products used in time-based media, these modules employed in his works are carefully and thoughtfully designed by the artist and his studio in San Francisco, California. The evolution of his design technique also reflects the technological development and progression in the industry.
Reprogrammable circuits and controllers play an increasingly significant role in interactive installations because they offer great flexibility for artists to execute specific functions. Compared to their software counterparts, not only do they provide different solutions, but they require different collection care knowledge and strategies to address obsolescence challenges. Emulation may allow a caretaker to retain software’s functionalities over virtual machines running on contemporary hardware; nonetheless, it cannot be accounted for maintaining and migrating embedded hardware designs over legacy technology.
To address the unparalleled role of artist-made hardware, I performed a comprehensive study of both the creation process as well as prevalent industrial practices from 1995 to 2019. With the assistance of electrical engineers and computer science experts, I aimed to identify the complex trajectory and technical characteristics of hardware configurations, and to further disseminate the overarching principles for conservators to care for custom electronics in time-based media.
Artist Biography
Jim Campbell was born in 1956 in Chicago, Illinois. He received degrees in both electrical engineering and mathematics from the Massachusetts Institute of Technology (MIT) in Cambridge, Massachusetts, in 1978. He is currently living and working in San Francisco, California, and holds more than a dozen patents in the field of image processing. In 1988, Campbell began creating interactive installations using “custom electronics” that he designs for the unique purpose of each installation. His artworks have been collected and exhibited in museums throughout the United States and Europe, including the San Francisco Museum of Modern Art (California), The Metropolitan Museum of Art (New York), the Museum of Modern Art (New York), the Smithsonian American Art Museum (Washington, D.C.), and the Whitney Museum of American Art (New York) (Plohman 2005).
The Artwork
Grand Central Station #2 was acquired by the Smithsonian American Art Museum (SAAM) in 2010, and after entering the collection, it was exhibited at Watch This!: Revelations in Media Art at SAAM in 2015.
The artwork consists of a grid of nine LED panels as a display screen with adhered custom electronics, a piece of Plexiglas with mounted photo-transparency, and a HWS 300-15V power supply. Campbell provided a manual with detailed mounting instructions (fig. 2).
The Plexiglas panel is installed in front of the tilted LED board. Once the LED board is connected with the power supply provided by the artist, the video is automatically played on a loop on the LED matrix screen. The work appears as a self-contained unit. The installation manual gives detailed instructions to mount the work as the artist envisioned, but it did not include enough information for future maintenance of the artwork.
Creation Process
Dissecting the structural framework of the production workflow will help unfold the complex details that enabled me to respect the nature of the work for the development of conservation strategies.
In the process of investigating any past treatments performed at the Smithsonian American Art Museum (SAAM), I discovered a group of photographs on the internal server, showing different configurations of the back of Grand Central Station #2. Ariel O’Connor, SAAM’s Object Conservator, suggested those might be taken by previous Object Conservator Hugh Shockey. He then confirmed his studio visits back in 2013 and his conversation with the artist about the possibility to update the circuit boards three years after the acquisition. However, he wasn’t able to proceed with the proposal before his departure in 2015.
In Grand Central Station #2, the artist used nine separate LED light panels to play a video as a unified display, and the composition of each LED panel along with the green circuit boards are mostly identical (fig. 3).
These circuit boards were described as “custom electronics” by the artist, and this term has been universally adopted by multiple museums for medium descriptions of Campbell’s works. Compared to other technology-based works in SAAM’s collection, “custom electronics” is an unclear term, and we need more technical descriptions in the TMS record that will enable us to better manage the preservation challenges in the future.
The following sections will carefully analyze the functionality and the diversity of elements that were used at different points in the artist’s creation process. Information was gathered from multiple sources, one of which is the artist interview I participated in at The Metropolitan Museum of Art (The Met) in 2015 (Campbell 2015). On May 31, 2019, Saisha Grayson, SAAM’s Curator of Time-Based Media, Dan Finn, SAAM’s Time-based Media Conservator, and I had a phone interview with the artist to discuss his 2013 update proposal, the creation process, and our ideas to approach the challenges brought on by technological obsolescence. Additionally, I consulted Dr. Peter Jamieson, associate professor in the Department of Electrical & Computer Engineering at Miami University, to ensure accurate presentation of the creation process as well as the terminology commonly used within those industries.
It is impossible to talk about Campbell’s working methods and his innovative use of mechanical engineering technology without discussing the field programmable gate array (FPGA) in more depth. It has been incorporated in many of Campbell’s works, in particular, the Low Resolution series (1999-ongoing). The FPGA is an integrated circuit, containing an array of programmable logic blocks. It allows the blocks to be inter-wired in different configurations so that the circuit can be reprogrammed (Morris 2015). During the 1980s to 2010s, Xilinx [1] and Altera [2] were the two major companies in the field.
The artist used nine custom FPGA circuit boards on nine separate LED light panels to play a set of motions on the LED matrix as if it were a unified display. The “master” FPGA controls eight “slave” FPGAs [3] and synchronizes the playback of different sections of the LED lighting sequence. Each “slave” board carries a part of the moving image for the adhered LED board. Xilinx Spartan 3A FPGA is used in Grand Central Station #2.
The overall workflows to prepare the artwork content are described below and it involves FPGA CAD flow where human-readable codes would be converted to low-level bitstreams. To begin, the artist uses a custom-built programmable hardware device to serve as a low-pass digital filter. It connects between the video camcorder (or a playback device) and the computer. The hardware subsamples video signals from the camcorder, and transforms it into low-resolution imagery to be captured by the computer. This process can be seen as the equivalent of video digitization. Instead of capturing the highest quality possible, the artist employs this custom hardware filter to output low resolution video for his custom LED displays.
After deriving a desired low-resolution video and laying out the bitmap, the artist uses the hardware description language, Verilog, to design lighting sequences for each LED panel based on its associated location in the matrix display. This part of the design is human readable and is often called “high-level design.” Programming an FPGA is the process of loading a bitstream into the FPGAs. The high-level designs are taken through a time-consuming, complicated process called the FPGA CAD flow, and there are significant optimizations and algorithms that are executed to convert high-level designs into low-level bitstreams. In other words, the human-readable designs are translated into binary forms, meaning they are 1’s and 0’s.
Since each FPGA has its own architecture, it adds another layer of complexity. The conversion process compiles the high-level designs and generates a specific bitstream to be embedded as a hardware platform on a particular FPGA. The FPGA itself doesn’t have memory space. Therefore, the bitstream is usually placed in non-volatile memory, and the hardware is configured to program the FPGA when powered on (Xilinx 2019).
In summary, the following tools and systems are used during the artwork production:
1. Programmable hardware built by the artist
2. Operating System: Windows XP or later
3. FPGA design tools: proprietary software such as Vivado to convert high-level designs to low-level bitstreams
4. Custom FPGA-based circuit boards
Two types of designs are created:
1. High-level designs: hardware description language Verilog
2. Low-level designs: bitstreams (1’s and 0’s)
Differences and Definitions
Although both FPGA bitstreams and software-based files are digital signals in binary forms, they are of a different nature that need to be addressed. FPGA bitstream is compiled in a specific way to configure a particular FPGA circuit, based on its unique architecture. FPGAs are extremely incompatible between manufacturers and models. An FPGA bitstream needs the native hardware configurations in order to make sense of the data. It is a hardware-based bitstream.
A software-based file is compiled through software programs, and it needs the associated digital systems for information retrieval. The established practices in the field of digital preservation, including digital forensic workflows or emulation of operating systems, are designed for the purpose of caring for software-based files. It wouldn’t be applicable to compile the native operating environment, such as Windows XP, or to analyze FPGA bitstreams with forensic tools.
The hardware filter part of the creation process was provided by the artist during The Met’s 2015 interview, but the way that the information was stored on the custom circuit boards was later misinterpreted. It shows the knowledge gap in understanding and preserving custom hardware. The field of media conservation has grown significantly, and it has responded to the demands of collections incorporating a wide array of media technologies. Much attention has been devoted to computer-based artworks with an emphasis on the impermanent aspects of software and digital components of artworks. However, there has not been the same level of research into the hardware side of the equation.
Changes and Progression
The artist proposed to update the FPGA circuit boards of Grand Central Station #2 in 2013. The new boards would allow for a single “mother” board to carry the design and send responding parts to multiple “daughter” boards (figs. 3, 5 and 6).
Please note the change of terms: The artist shifted the relations from “master vs. slaves” to “mother vs. daughters.” The physical appearance of the circuit boards used in the original configurations are identical, but they each carry a part of the designs. Hence, they are not interchangeable. By contrast, in the proposed hardware update, the physical appearance of the mother board is different from the daughters but the daughters are interchangeable. The daughter boards can be swapped to control different LED panels, and the displayed light sequence would remain the same. Unlike some treatments for time-based works that sometimes change the “look” of the work, Campbell’s proposed update is not supposed to alter the light sequences on the LED display even though the hardware would change.
To research the conservation actions performed by other institutions, I reached out to The Met and the Denver Art Museum (DAM). I learned that the artist has been the point person for these projects, and the studio oversees and makes any configuration changes of the collected artworks. DAM has a sound installation called I Have Never Read a Bible created in 1995. The Met has Motion and Reset #2 created in 2002 and is the only collection that has acquired a spare LED display from the artist.
What SAAM, The Met, and DAM have in common:
- Custom FPGA boards: three different board designs in 1995, 2002, and 2009 were implemented with different FPGA circuits.
- “White Light Labs” was printed on the original boards from the artist’s studio.
- No high-level/source designs were collected.
- No update has been made either by the artist or by the conservators.
The first FPGA company, Altera, was founded in 1983, and their first reprogrammable logic device was introduced in 1984. The 90’s were a period of rapid growth for FPGAs, and they have been widely used in networking and telecom systems. The FPGA-based custom circuit boards designed by Campbell have showcased the artist’s early adoption of a lesser-known technology that required highly specialized skills. He built custom circuit boards that could not be found in the commercial market in order to achieve his vision. He has been consistently working with the same technology over the course of 20 years, and his design technique has gradually changed as the technology makes progress.
In 1995, a stack of FPGA-based circuit boards would have been needed to perform a rather simple function—whereas in 2013, one single board can carry the entire design of light sequences. A change in Campbell’s artwork circuit board labels may also suggest how his working methods have closely reflected technical progression in the industry. The name of his studio, White Light Labs, is printed on the custom circuit boards from 1995 to 2009 and disappeared from the 2013 daughter boards for the proposed update. I believe this indicates the possibility that by 2013, Campbell didn’t need to custom make his circuit board since a commercially available one in the market could fulfill his needs. In addition to the change of board designs, I believe he credited his studio on the custom-made circuit board showing the collaborative nature in the engineering community.
Risk Assessment
In this section, I will provide detailed risk assessments for 1) digital source designs and 2) physical configurations.
Digital Source Designs
Here, I will discuss digital source designs for Grand Central Station #2 in three parts: 1. High-level designs, 2. Low-level bitstream, and 3. FPGA design tools. The artist might not have the high-level designs for this artwork, while the low-level bitstream is on the physical FPGA circuits.
- High-level Designs
Early in the creation, the artist wrote human-readable designs in the hardware description language, Verilog. No museum had acquired this high-level design as an artwork deliverable, and the artist might not have it any longer. If an update were pursued, the artist clearly said it will be a new design— not only the custom circuit boards, but the high-level designs (Campbell 2019). If SAAM is able to acquire the high-level designs, we will need to work with an experienced engineer to generate a separate annotated version in order to understand the behavior written in the hardware description language. - Low-level Bitstreams
Low-level bitstream is a file that contains a hardware platform for a specific FPGA. SAAM has very little to no knowledge of the bitstream data structure, and there are no established practices and workflows for its safe retrieval. A secure boot might be implemented to store encrypted bitstream; therefore, there is a potential risk that retrieval of bitstream through a logic analyzer or other device could permanently damage the circuit (Pocklassery 2018).
Without the high-level designs or the low-level bitstream, the museum loses the ability to take appropriate measures to provide adequate stewardship. In the event of a hardware failure, SAAM wouldn’t be able to repair, copy, and migrate the designs. While it’s possible to derive original high-level designs through reverse engineering, it requires very specific knowledge and expertise. I will explore more about reverse-engineering in later sections. - FPGA Design Tools
During the development phase, the FPGA design tools are used to convert high-level designs and load the low-level bitstreams to program the FPGAs. Most of the prevalent FPGA design tools are proprietary software tools, and the exact one used in the production of Grand Central #2 remains unknown. It is usually employed by developers to manage and produce bitstreams with compatible data structure for specific FPGAs. It contains important information about the related FPGA architectures, and most of those are not made available by the manufacturers. I have several questions, such as: Can high-level designs be converted without the original tools for the same FPGA model? Can low-level bitstream be reverse-engineered without the original tool?
Physical Configurations
This part of the assessment relates to physical configurations of hardware and will be divided into LED matrix displays as well as custom circuit boards designed by the artist.
- LED Matrix Displays
Nine separate LED panels mounted onto a metal chassis, which acts as a mounting mechanism. The artist mentioned in the 2019 interview with SAAM that most LED models can become obsolete in 2 years, so it would be difficult to find an exact light bulb or LED panel for replacement. Instead of finding the same model of LED, the artist said it will be more important to maintain the original size of the light bulb as well as the same distance between the light bulbs on the panel. This information was not documented in our records at SAAM.
LEDs have a general life expectancy of 50,000 hours (LED Benchmark 2020). If installing the work on display for 10 hours a day, it would most likely last for 13.7 years. It is difficult if not impossible to unify brightness level and color temperature between different LED light bulbs. Continuous update and changes of LED would permanently change the original configurations, and conservation actions would become irreversible. - Custom Circuit Boards
The original custom circuit boards used in Campbell’s artworks are printed with the name of his studio, “White Light Labs.” The studio would have had to submit a schematic circuit diagram to a third party for manufacture. The schematic was not collected by SAAM. The functionality of each component on the custom circuit board remains unclear, so it is difficult for the conservator to troubleshoot, repair, replace, or find appropriate equipment to extract the bitstream. It will become more and more difficult to migrate the design to later models when the components used in the original configuration become obsolete.
The board has a certain life expectancy, so we need to plan for conservation actions for long-term access. Customers require a 15-year operational lifetime for FPGAs, so availability is often 15 years or more. A properly designed and documented Xilinx FPGA device design may be ported to a newer circuit with equal or better speed and capacity. However, it is too early to know if the change of speed would increment over updates and eventually have an impact on the play speed of light sequences on the display.
Future Goals
“One interesting note is that it would have been infinitely easier, and still would be infinitely easier, to do some of these things in software. But because my background is hardware, I choose to do things in hardware, because that’s what I kind of have a good sense of, a more intuitive relationship with.”
Jim Campbell, 2015 artist interview.
The artist described his working methods, technique, and relationships with reconfigurable hardware in the 2015 interview at The Met. This statement gives an insight into how it best suits his temperament and provides the cornerstone of our proposed strategies to tackle challenges from acquisition to preservation. SAAM hopes to collaborate with our research partners to build tools and establish integrated workflows for artists, engineers, and conservators to manage obsolescence challenges related to FPGA integrated circuits and to tackle life cycle challenges from acquisition to conservation.
Our goal will primarily denote works with custom-built computer hardware—components that present specific challenges for conservation, documentation, installation, and acquisition. Reprogrammable circuits and controllers play an increasingly significant role in interactive installations because they offer great flexibility for artists to execute specific functions. Without established practices for secure handling or the fundamental knowledge about the technical characteristics of different hardware components, institutions’ caretakers may consider most of the hardware to be generic, mass-produced, and uniform. This would cause them to overlook the equipment’s significance as well as the change and potential loss introduced by different hardware technology.
Recognizing the knowledge gap, we wish to initiate a knowledge base that will enable conservators to accurately describe and document custom hardware. We aim to develop an integrated practice that can carefully document and preserve artworks using FPGA-based controllers as well as monitor changes in LED installations’ performance over time. We hope that our project will stimulate broader understanding and appreciation over custom electronics as an advocate for the unparalleled role of artist-made hardware in technical art history and time-based media conservation.
ACKNOWLEDGEMENTS
I would like to thank everyone at the Smithsonian American Art Museum, especially Dan Finn, Amber Kerr, Laura Hoffman, Saisha Grayson, Luke Moses, Lynn Putney, and Ariel O’Connor for their generous support and guidance throughout the process. I also want to express my greatest gratitude for those who have generously lent their knowledge and expertise. Thanks go to Dr. Peter Jamieson (Miami University), Dr. William Tu and Dr. Paul Wang (Howard University), Dr. Glenn Wharton (New York University), Kate Moomaw (Denver Art Museum), Alexandra Nichols (the Metropolitan Museum), Hugh Shockey (Saint Louis Art Museum), Dr. Dirk Koch (the University of Manchester), Dr. Ua Chen (United Microelectronics Corporation), Chia-Yu Kuo and Guan-Qing Zhuang (HTC), and Kuan-Chen Wu (Kamia Designs).
NOTES
- Xilinx, Inc., part of Advanced Micro Devices, Inc. (AMD) in Oct. 2020. www.xilinx.com.
- Altera Corporation, part of Intel Corporation in 2015. www.intel.com.
- The terminology used to describe FPGA controller relationships was derived from the artist interview. Although outdated, it was commonly used in the industry in the 90s and early 2000s.
REFERENCES
Campbell, Jim. 2015. Artist Interview at The Metropolitan Museum of Art, October 28, 2015.
Campbell, Jim. 2019. Artist interview with Smithsonian American Art Museum, May 31st 2019.
Cheung, Peter Y.K. 2018. “Mastering Digital Design in Verilog using FPGAs.” Imperial College London. Oct 18, 2018. www.ee.ic.ac.uk/pcheung/teaching/msc_experiment/ (accessed August 6th, 2020).
LED Benchmark. 2020. “FAQ – LED Life Expectancy.” LED Benchmark. www.ledbenchmark.com/faq/life-expectancy.html (accessed August 6th, 2020).
Morris, Kevin. 2015. “The FPGA Tool Problem.” Electronic Engineering Journal. October 4, 2015. www.eejournal.com/article/20161004-opensource/ (accessed August 6th, 2020).
Plohman, Angela. 2005. “Jim Campbell.” Fondation Daniel Langlois. www.fondation-langlois.org/html/e/page.php?NumPage=77 (accessed August 6th, 2020).
Pocklassery, G, Che, W., Saqib, F., Areno, M., and J. Plusquellic. 2018. “Self-Authenticating Secure Boot for FPGAs.” 2018 IEEE International Symposium on Hardware Oriented Security and Trust (HOST). https://ece-research.unm.edu/jimp/pubs/FPGASecureBoot.pdf (accessed August 6th, 2020).
Xilinx. 2019. “FPGA Bitstream.” Xilinx Inc.. www.xilinx.com/html_docs/xilinx2019_1/SDK_Doc/SDK_concepts/concept_fpgabitstream.html (accessed August 6th, 2020).