commentary
Transparency in film: increasing credibility of scientific animation using citation Stuart G Jantzen, Jodie Jenkinson & Gaël McGill
npg
© 2015 Nature America, Inc. All rights reserved.
Scientific animations have tremendous potential as instruments of insight and dissemination. However, audiences are often unable to determine the degree to which visualizations are informed by scientific evidence. By providing a more detailed account of source use, developers can increase the credibility of animations as scientific tools. Scientific animations are designed for many different venues, audiences and purposes: education, research dissemination, entertainment, outreach, inspiration and hypothesis generation. Whereas static depictions of molecular models can fall short of communicating complex spatiotemporal interactions, animations can more fully reflect our understanding of such models1. These dynamic visualizations draw on a variety of sources and use them in different ways. Visual elements within even a single scene of an animation may be based on direct empirical data, interpretations or pure speculation. Currently, there is no mechanism in place for discerning how the design and behavior of features within an animated display are informed. Even in relatively rare cases where animations include a bibliography in the end credits, there is no way to link these references to individual features of the display. It is left to the viewer to examine all references and make assumptions as to where and how they were used to inform the work. As discussed previously in an Editorial in this journal2, this situation can lead to a lack of clarity regarding the credibility of scientific animation. In peer-reviewed scientific publications, there is an established system of citation; most references are used to provide context Stuart G. Jantzen and Jodie Jenkinson are at the Biomedical Communications Unit, Department of Biology, University of Toronto, Mississauga, Ontario, Canada; Gaël McGill is at the Center for Molecular and Cellular Dynamics, Harvard Medical School, Boston, Massachusetts, USA. e-mail: [email protected]
Arrow (accessory)
Dynein (character)
Retrograde transport (text)
Vesicle (character)
Microtubule (environment)
Figure 1 | Elements in a scene from a scientific animation. The scene has been deconstructed into labeled elements with associated element types listed in parentheses. The character and environment elements could be further characterized with the following properties: Dynein by structure, appearance, motion and interaction; Vesicle by structure, appearance, motion and population; Microtubule by structure, appearance and population. The elements and properties are each informed by a distinct set of references. Figure adapted from an animation created by S.G.J. (https://vimeo.com/70039214/).
for research or to support claims. Though the production of an animation involves an even broader range of reference use, no such system exists for describing source usage. Data may be pruned, extracted or interpolated in ways that would be considered inappropriate in peer-reviewed publications but that are valid for supporting different forms of visual explanation. To add further complexity, data may be used to inform certain properties of an element but not others. For instance, Protein Data Bank structural data may be used to create accurate threedimensional (3D) surfaces3,4, but binding
interactions with another molecule may be more speculative, based on estimates or descriptive sources. Although this may seem an unconventional mix of standards, such treatment is often necessary for explaining a cellular process visually, depending on the availability of data. Nevertheless, it would be informative to explain to audiences that a molecule’s structure was based on a specific data set, whereas its interaction behavior was a combination of speculation and description from other sources. Animations also use a greater variety of references than traditional scholarly works.
nature methods | VOL.12 NO.4 | APRIL 2015 | 293
commentary
npg
© 2015 Nature America, Inc. All rights reserved.
Table 1 | Types of elements Type
Description
Character
Central to narrative
Examples Ribosome, lymphocyte, syringe
Environment
Supportive of narrative; provides context
Lipid bilayer, intestinal wall, surgical field
Text
Written on-screen text
Label, caption
Information graphic
Synthesis and visual communication of data
Bar chart, geographical map
Accessory
Assistive communication device
Arrow, glow, circle
Audio
Auditory track
Narration, music, sound effect
Personal communications with domain experts as well as information from database queries, images, simulations, videos, articles and textbooks may all be incorporated into animated media. Nonstandard reference material should not necessarily be dismissed, yet transparency is important: it should be possible to identify and recognize the class and strength of different sources used in producing an animation. Here we discuss the importance, benefits and difficulties of identifying sources that inform the development of scientific and medical animation. We also describe possible information hierarchies that can be used to establish a standardized citation system. Components of animated displays Throughout a project, and in particular at the research and pre-production stage, developers of scientific animation gather a large selection of sources that represent many types of media. In order to categorically organize these materials and link these sources more directly to specific features and behaviors in an animation, we propose an initial approach in which a ‘cast of characters’ and all supporting elements (Fig. 1 and Table 1) in the animation are described. Different element types have distinct visual and explanatory roles, whether they be characters, environments, text, information graphics, accessories or audio. A scene may be divided into these functional elements partly on the basis of where different references were used. A character could be a single molecule, a cell, a medical device or even an anatomical structure or tissue. The element must simply possess some particular significance in a section of the narrative. A single character might actually encompass a population of a given species, such as a cloud of ions. In this scheme, it is particularly important to cite data sources for characters because they convey the bulk of scientific meaning to the audience. Environments, in contrast, have a supporting role; they would include components
that provide spatial context. Text elements are typically labels or captions and would require references only in cases where there is no visual counterpart to the piece of information conveyed through text. Information graphics typically take the form of animated or static graphical representations derived directly from data. Accessories are auxiliary storytelling tools such as arrows, callouts, light rays, visual analogies or any number of other non-data-driven devices that serve simply to facilitate communication. Rarely would these require citations, but on occasion the need to explain or justify the use of a relevant accessory may arise. In the case of audio content, citations may be needed for elements of narration or sound effects based on specific data that are not also represented visually. We note that we do not refer here to providing credit for content creators (i.e., authors of sound effects, music, etc.); these should be included separately. Seldom does a single reference provide information pertinent to the visualization of all aspects of a character or environment. One source might provide data on structural components and another on connections to surrounding elements. In the case of macroscopic subjects, visual properties might have yet another source. Because different references often inform different properties, we propose subcategorizing elements into properties. We have identified five properties of most character and environment elements— structure, appearance, motion, interaction
and population—each encompassing a number of data types (Table 2). Data informing the structure of an element might come from crystallography or medical imaging, for example. Appearance might include texture or color. Motion data include conformational changes, molecular dynamics simulations and organism locomotion. Interaction relates to the nature of an element’s association with other elements, such as permanent connections, biological ‘machinery’ involved in the creation of the element, transient physical contact, or other associations that result in changes being made to the element. Population data include the quantity, concentration and organization of the element. Other types of elements such as information graphics and accessory elements (Table 1) will not necessarily possess any or all of these properties. Reference types and uses Consideration of all elements and their associated properties, perhaps organized according to the scene in which the element first appeared, allows for a breakdown of an animation into a logical hierarchy of components that indicate where different references are employed. From this point, the question of how references inform visualization can be asked. We broadly classify the different types of sources used in the production of an animation as visual, qualitative and quantitative (Table 3). Animators use different processes when working with these different data types. It might be quite apparent that a visual source matches the relevant element in the animation, but the influence is likely to be more cryptic for other reference types. Depending on the characteristics of the source data, abstraction and conversion must be performed to varying degrees. A benefit of categorizing the type of source is that different subsets of information from a reference could be indicated (for example, figures, descriptions, tables): the resulting
Table 2 | Properties of elements Type
Description
Structure
Structural data, spatial data or subunit Thickness, sequence of monomers, tortuosity composition
Appearance
Visual data; interaction with real or imagined light
Texture, color, transparency
Motion
Spatiotemporal data
Conformational change, trajectory, locomotion
Interaction
Association of an element with other elements
Binding, degradation, social behaviors
Population
Properties of multiples or group; interaction within element
Orientation, aggregation, concentration gradient
294 | VOL.12 NO.4 | APRIL 2015 | nature methods
Examples
npg
© 2015 Nature America, Inc. All rights reserved.
commentary depiction of an element might seem more objective if it were supported by quantitative data rather than descriptive text. Regardless of the data types incorporated, the nature of the scientific support consequently becomes more transparent to the audience. Speculation and artistic license designate cases in which no sources are used to inform properties of an element. Speculative content is common even in peer-reviewed literature when unresolved hypotheses are put forward. Speculation in the context of animation is even more necessary because animated displays require visualization of many aspects of each element despite the inevitable lack of data on certain details. The depiction of elements with missing information poses problems for the animator. Creative decisions depend on the ultimate purpose and focus of the narrative, but the fact that data are missing should also be of interest to certain viewers. Interestingly, it is sometimes during the development of visualizations that it becomes apparent to domain experts exactly what remains poorly understood1; in these situations it is necessary to speculate5. In other cases features are intentionally depicted without the support of references and decisions are made arbitrarily, for simplicity and clarity, or for aesthetic reasons. Sometimes visual information must be supplied that is not physically applicable, such as color and other surface properties at molecular scales. These cases can be categorized under the general notion of artistic license5. We believe that explicit citation of speculation and artistic license adds transparency and credibility to animated works. Furthermore, promoting a single point of view or comparing multiple existing hypotheses is a legitimate use for scientific animation and can involve selectively excluding data to emphasize a specific view1,5. We then categorize how these data sources may be manipulated or interpreted (Table 4) as direct import, adaptation, pruning, smoothing, interpolation and sampling. Most computer animation software has the capacity to import data in a variety of formats. Plug-ins created specifically for biomedical animators (for example, ePMV, Molecular Maya and BioBlender) can simplify the process of using empirical data in animation. Data may be imported and used as is (direct import). However, in some cases data may be missing or noisy or include extraneous information6. For example, outliers may need to be removed or the data may require simplification for
Table 3 | Categories of reference Type
Description
Examples
Visual
Translated from visual sources
Photograph, micrograph, illustration, video
Quantitative
Translated from numerical data
Size, speed, frequency
Qualitative
Translated from descriptive data
Color, process, interaction
Speculation or artistic license
No reference; estimated or imagined information
Missing data, simplification for clarity, visual aid
increased clarity (pruning). Occasionally, sampling artifacts must be addressed or visual noise eliminated (smoothing). There are many situations in which an artist must fill in missing pieces from available data (interpolation). For example, 3D objects are often generated from 2D references or virtual transpositions are performed to present a view different from that of the original source. Interpolation differs from speculation in that the depiction is still primarily based on data. Qualitative sources often require interpretation to render them in a visual format, and quantitative sources sometimes require processing and translation (adaptation). Finally, an animator may simply select pieces of information to supplement an artistic interpretation (sampling). Note that this use of “sampling” is different from the statistical concept of collecting a representative set of data from a population. Our definition includes the intentional choosing of information that supports the desired narrative and exclusion of the remainder. It is worth noting that more than one kind of use may be applicable for a single reference. Although these categories and properties may require future revision to successfully define the constituents of every conceivable animation, we propose that a breakdown of animated elements and their corresponding references using the suggested categories could provide a much-needed approach for hierarchically organizing the sources that inform scientific animation.
Implementation Rigorous citation demands careful management throughout a complex pipeline (Supplementary Fig. 1). During pre- production, production and post-production, an animator should keep references organized and add notes and tags to sources as they are collected and used. Many reference-management applications include the ability to annotate sources with labels or populate custom fields. Whether references are managed with custom software or simple spreadsheets, it is beneficial to review all citations at a point during post-production to confirm their accuracy and relevance, as reference usage and information content may evolve through pre-production and production. Next, the animator must decide how to present this collection of information to audiences. Analogously to the prevalence of different citation styles for written documents, animators and scientists together will need to develop different formats for presenting citation information to audiences, with the goal of providing the most information with the greatest clarity. Viewers should be able to access and navigate the information bidirectionally: one viewer might be interested in a specific visual element and the sources supporting it, whereas others might search for a specific data source and see where in the animation it was used. Annotations could be overlaid on videos using existing subtitling systems that can be toggled on or off. To avoid a split-attention effect7, it might
Table 4 | Reference uses Type
Description
Direct import
Information is imported directly or copied Proportions of different species found in an one-to-one with no modification ecosystem
Examples
Adaptation
Information is translated into another format (typically visual)
Interaction of molecules from description of process
Interpolation
Missing information is approximated
Full molecule built from partial crystallographic data
Pruning
Some information is removed
Artery shown with certain branches omitted
Smoothing
Artifacts or other raw characteristics are minimized
3D model from computed tomography scan slices
Sampling
Selected pieces of information are used
Two representative structures shown from database of 40 structures
nature methods | VOL.12 NO.4 | APRIL 2015 | 295
commentary be preferable to provide citations associated with representative still frames or ‘thumbnails’ in an accompanying document. A simple format would be static and linear, yet informative and tightly structured (Fig. 2a). More dynamic and interactive formats can be designed (Fig. 2b), which require more work to produce but allow for quick navigation among thumbnails and elements. In addition to providing the reference type and usage labels that we have suggested (Tables 3 and 4), it can be extremely valuable to add a descriptive note. This kind of informative text is epitomized in the miniseries “Illustrating the Machinery of Life”8, which
describes the scientific literature supporting an illustration. The benefit of a longer article of this type is the freedom to describe in detail exactly how different information was incorporated and why certain decisions were made (for example, among conflicting data). But a problem with this type of article is the lack of predefined structure and utility as a quick reference. We believe that including a brief comment on reference use for each item in the bibliography can be invaluable. Animators are often skilled at implementing their own custom software solutions to expedite common tasks. We anticipate this will naturally result in novel and
Discussion Our primary goal in promoting the inclusion of content-linked bibliographies with
b
npg
© 2015 Nature America, Inc. All rights reserved.
a
improved tools for automated reference management and presentation. These may include custom plug-ins that interface between reference-management software and animation packages or drag-and-drop interfaces for attaching citations to locations on an image. There is also the potential for complex data visualizations with coded information dynamically overlaid on an animation, perhaps highlighting regions of robust scientific support.
Figure 2 | Example output formats for using citation in a scientific animation. (a) Sections from a linear document. The document begins with key references, summarizing the information presented in the animation as a whole. These are followed by thumbnails and associated references. An example thumbnail with shaded and labeled elements is accompanied by a legend. The thumbnail is assigned a numerical identifier along with a time code so that its location in the original piece may be easily found. Each element is listed with properties and associated references. References are numbered sequentially through the entire document to allow indication of references used across multiple thumbnails. Finally, reference type, use and any explanatory comments are presented in italics. A partial set of references informing the elements in this thumbnail are shown. (b) Screen capture of an interactive module. A strip of images along the top allows a user to select from thumbnails representing all elements in the piece. The selected image has labeled and clickable regions outlining each element, a time code, and descriptions of audio elements that may require their own references. The text area below displays references used for a selected element, in this case the dynein molecule. The information is organized as in a. 296 | VOL.12 NO.4 | APRIL 2015 | nature methods
npg
© 2015 Nature America, Inc. All rights reserved.
commentary animated media is to increase the scientific credibility of animation through increased transparenc y and accountability. A thoughtfully constructed bibliography will provide scientists with greater context for a particular animation and allow them to internalize relevant information and investigate areas of speculation rather than dismiss an animation in its entirety because it is based in part upon supposition. Viewers would have a means of comparing citations across animations addressing similar subject matter, and animators would discover sources that colleagues find useful. A general increase in communication and crosspollination of ideas should result from content creators sharing sources with each other and with viewers. Just as research is an iterative process, animations may be improved as scientific evidence becomes available; this process of improvement will likely be aided by the inclusion of citations. Audiences, including scientists, will better understand the storytelling and decisionmaking processes that lead an animator to employ various references in different ways, and content creators would be encouraged to be more intentional in selecting and using reference material. It also becomes important to include a creation or publication date with animations so that future audiences can understand references in the context of when the piece was developed. We believe that the citation process need not necessarily describe every reference used. A viewer would neither expect nor desire access to a large quantity of reference material supporting the design of inconsequential features of the display. The animation production team would be responsible for selecting and sharing the sources that are likely to be relevant or of interest. This type of information filtering is already part of the expertise of biomedical communicators. Animators are tasked with highlighting key points of data, so as to increase the clarity and comprehensibility of complex material. We are aware of potential limitations and drawbacks in using a citation system with animation. From the point of view of an animator, there may be significant overhead in
time (and therefore cost) to the production pipeline through cataloging and maintaining rigorous records of reference use. Although this may be a time-consuming effort, the long-term rewards are worthwhile. There may also be concern among studios that disclosure of sources and other production information would harm them competitively. Weighing the benefit of increased credibility against guarding research methods remains in the hands of the developers. Some may feel that any animation citing “speculation” or “sampling” of data will be called into question or criticized for manipulation. Misrepresenting information should of course be avoided, but it must be remembered that the broad range of uses for scientific visualizations require different degrees of freedom for the use of speculative content. Including clear indication that certain aspects of an animation are speculative should not detract from but rather increase the credibility of the creator. It is outside the scope of this article to discuss how and when references should be used. Instead we assume that references are used and wish to propose ways by which these details can be categorized and presented. Not all bibliographic formats are equally suitable in all situations. A successful bibliography must facilitate an understanding of the references in the context of the visuals. In the case of video hosting sites, one may provide a link to an accompanying document hosted elsewhere that the viewer could peruse while rewatching the film. At a conference, it is more difficult to display all references, but a paper copy of thumbnails and citations could be provided, a URL to access citation information might be visible adjacent to the display, or the presenter may simply highlight a few key references that informed a central point. The availability of an animation for additional viewing will affect the utility of different formats for providing citations. If an audience member sees an animation only once and has no further access to it, an online bibliography would require detailed thumbnails and clear labeling of elements to be of much value (Fig. 2). Complete freedom in the method
of p resenting references will offer the greatest value to viewers, but even venues with the most specific restrictions for media formats should be able to accommodate some form of annotated bibliography and the added support it provides. It is our hope that scientists will encourage the reporting of reference use and that animators will further develop productive ways to share this information. The ideas discussed here may be adapted and implemented for other media, such as illustrations and interactive modules, which suffer many of the same citation pitfalls as animation. Regardless of medium, content creators and audiences alike should consider the merits of adopting more formalized methods for citing resources. By clearly laying out both where and how references are used in visual communications, animations may gain greater credibility, which can lead to more open and productive dialog and dissemination of ideas and knowledge. Note: Any Supplementary Information and Source Data files are available in the online version of the paper (doi:10.1038/nmeth.3334). ACKNOWLEDGMENTS This research is supported in part by grants NSF #DUE1220512 from the US National Science Foundation and SSHRC #SIG-13/14 from the Canadian Social Sciences and Humanities Research Council. The image in Figure 1 was adapted from an animation by S.G.J. (https://vimeo.com/70039214/). AUTHOR CONTRIBUTIONS S.G.J. developed the concepts and wrote the manuscript. J.J. and G.M. provided direction and contributed to the manuscript. COMPETING FINANCIAL INTERESTS The authors declare competing financial interests: details are available in the online version of the paper (doi:10.1038/nmeth.3334). 1. Iwasa, J.H. Trends Cell Biol. 20, 699–704 (2010). 2. Anonymous. Nat. Methods 9, 1127 (2012). 3. Johnson, G.T., Autin, L., Goodsell, D.S., Sanner, M.F. & Olson, A.J. Structure 19, 293–303 (2011). 4. McGill, G. Cell 133, 1127–1132 (2008). 5. Goodsell, D.S. & Johnson, G.T. PLoS Biol. 5, e308 (2007). 6. Goodsell, D.S. J. Biocommun. 35, 52–57 (2009). 7. Chandler, P. & Sweller, J. Br. J. Educ. Psychol. 62, 233–246 (1992). 8. Goodsell, D.S. Biochem. Mol. Biol. Educ. 37, 204–210 (2009).
nature methods | VOL.12 NO.4 | APRIL 2015 | 297
Transparency in film: increasing credibility of scientific animation using citation Stuart G Jantzen, Jodie Jenkinson & Gaël McGill
npg
© 2015 Nature America, Inc. All rights reserved.
Scientific animations have tremendous potential as instruments of insight and dissemination. However, audiences are often unable to determine the degree to which visualizations are informed by scientific evidence. By providing a more detailed account of source use, developers can increase the credibility of animations as scientific tools. Scientific animations are designed for many different venues, audiences and purposes: education, research dissemination, entertainment, outreach, inspiration and hypothesis generation. Whereas static depictions of molecular models can fall short of communicating complex spatiotemporal interactions, animations can more fully reflect our understanding of such models1. These dynamic visualizations draw on a variety of sources and use them in different ways. Visual elements within even a single scene of an animation may be based on direct empirical data, interpretations or pure speculation. Currently, there is no mechanism in place for discerning how the design and behavior of features within an animated display are informed. Even in relatively rare cases where animations include a bibliography in the end credits, there is no way to link these references to individual features of the display. It is left to the viewer to examine all references and make assumptions as to where and how they were used to inform the work. As discussed previously in an Editorial in this journal2, this situation can lead to a lack of clarity regarding the credibility of scientific animation. In peer-reviewed scientific publications, there is an established system of citation; most references are used to provide context Stuart G. Jantzen and Jodie Jenkinson are at the Biomedical Communications Unit, Department of Biology, University of Toronto, Mississauga, Ontario, Canada; Gaël McGill is at the Center for Molecular and Cellular Dynamics, Harvard Medical School, Boston, Massachusetts, USA. e-mail: [email protected]
Arrow (accessory)
Dynein (character)
Retrograde transport (text)
Vesicle (character)
Microtubule (environment)
Figure 1 | Elements in a scene from a scientific animation. The scene has been deconstructed into labeled elements with associated element types listed in parentheses. The character and environment elements could be further characterized with the following properties: Dynein by structure, appearance, motion and interaction; Vesicle by structure, appearance, motion and population; Microtubule by structure, appearance and population. The elements and properties are each informed by a distinct set of references. Figure adapted from an animation created by S.G.J. (https://vimeo.com/70039214/).
for research or to support claims. Though the production of an animation involves an even broader range of reference use, no such system exists for describing source usage. Data may be pruned, extracted or interpolated in ways that would be considered inappropriate in peer-reviewed publications but that are valid for supporting different forms of visual explanation. To add further complexity, data may be used to inform certain properties of an element but not others. For instance, Protein Data Bank structural data may be used to create accurate threedimensional (3D) surfaces3,4, but binding
interactions with another molecule may be more speculative, based on estimates or descriptive sources. Although this may seem an unconventional mix of standards, such treatment is often necessary for explaining a cellular process visually, depending on the availability of data. Nevertheless, it would be informative to explain to audiences that a molecule’s structure was based on a specific data set, whereas its interaction behavior was a combination of speculation and description from other sources. Animations also use a greater variety of references than traditional scholarly works.
nature methods | VOL.12 NO.4 | APRIL 2015 | 293
commentary
npg
© 2015 Nature America, Inc. All rights reserved.
Table 1 | Types of elements Type
Description
Character
Central to narrative
Examples Ribosome, lymphocyte, syringe
Environment
Supportive of narrative; provides context
Lipid bilayer, intestinal wall, surgical field
Text
Written on-screen text
Label, caption
Information graphic
Synthesis and visual communication of data
Bar chart, geographical map
Accessory
Assistive communication device
Arrow, glow, circle
Audio
Auditory track
Narration, music, sound effect
Personal communications with domain experts as well as information from database queries, images, simulations, videos, articles and textbooks may all be incorporated into animated media. Nonstandard reference material should not necessarily be dismissed, yet transparency is important: it should be possible to identify and recognize the class and strength of different sources used in producing an animation. Here we discuss the importance, benefits and difficulties of identifying sources that inform the development of scientific and medical animation. We also describe possible information hierarchies that can be used to establish a standardized citation system. Components of animated displays Throughout a project, and in particular at the research and pre-production stage, developers of scientific animation gather a large selection of sources that represent many types of media. In order to categorically organize these materials and link these sources more directly to specific features and behaviors in an animation, we propose an initial approach in which a ‘cast of characters’ and all supporting elements (Fig. 1 and Table 1) in the animation are described. Different element types have distinct visual and explanatory roles, whether they be characters, environments, text, information graphics, accessories or audio. A scene may be divided into these functional elements partly on the basis of where different references were used. A character could be a single molecule, a cell, a medical device or even an anatomical structure or tissue. The element must simply possess some particular significance in a section of the narrative. A single character might actually encompass a population of a given species, such as a cloud of ions. In this scheme, it is particularly important to cite data sources for characters because they convey the bulk of scientific meaning to the audience. Environments, in contrast, have a supporting role; they would include components
that provide spatial context. Text elements are typically labels or captions and would require references only in cases where there is no visual counterpart to the piece of information conveyed through text. Information graphics typically take the form of animated or static graphical representations derived directly from data. Accessories are auxiliary storytelling tools such as arrows, callouts, light rays, visual analogies or any number of other non-data-driven devices that serve simply to facilitate communication. Rarely would these require citations, but on occasion the need to explain or justify the use of a relevant accessory may arise. In the case of audio content, citations may be needed for elements of narration or sound effects based on specific data that are not also represented visually. We note that we do not refer here to providing credit for content creators (i.e., authors of sound effects, music, etc.); these should be included separately. Seldom does a single reference provide information pertinent to the visualization of all aspects of a character or environment. One source might provide data on structural components and another on connections to surrounding elements. In the case of macroscopic subjects, visual properties might have yet another source. Because different references often inform different properties, we propose subcategorizing elements into properties. We have identified five properties of most character and environment elements— structure, appearance, motion, interaction
and population—each encompassing a number of data types (Table 2). Data informing the structure of an element might come from crystallography or medical imaging, for example. Appearance might include texture or color. Motion data include conformational changes, molecular dynamics simulations and organism locomotion. Interaction relates to the nature of an element’s association with other elements, such as permanent connections, biological ‘machinery’ involved in the creation of the element, transient physical contact, or other associations that result in changes being made to the element. Population data include the quantity, concentration and organization of the element. Other types of elements such as information graphics and accessory elements (Table 1) will not necessarily possess any or all of these properties. Reference types and uses Consideration of all elements and their associated properties, perhaps organized according to the scene in which the element first appeared, allows for a breakdown of an animation into a logical hierarchy of components that indicate where different references are employed. From this point, the question of how references inform visualization can be asked. We broadly classify the different types of sources used in the production of an animation as visual, qualitative and quantitative (Table 3). Animators use different processes when working with these different data types. It might be quite apparent that a visual source matches the relevant element in the animation, but the influence is likely to be more cryptic for other reference types. Depending on the characteristics of the source data, abstraction and conversion must be performed to varying degrees. A benefit of categorizing the type of source is that different subsets of information from a reference could be indicated (for example, figures, descriptions, tables): the resulting
Table 2 | Properties of elements Type
Description
Structure
Structural data, spatial data or subunit Thickness, sequence of monomers, tortuosity composition
Appearance
Visual data; interaction with real or imagined light
Texture, color, transparency
Motion
Spatiotemporal data
Conformational change, trajectory, locomotion
Interaction
Association of an element with other elements
Binding, degradation, social behaviors
Population
Properties of multiples or group; interaction within element
Orientation, aggregation, concentration gradient
294 | VOL.12 NO.4 | APRIL 2015 | nature methods
Examples
npg
© 2015 Nature America, Inc. All rights reserved.
commentary depiction of an element might seem more objective if it were supported by quantitative data rather than descriptive text. Regardless of the data types incorporated, the nature of the scientific support consequently becomes more transparent to the audience. Speculation and artistic license designate cases in which no sources are used to inform properties of an element. Speculative content is common even in peer-reviewed literature when unresolved hypotheses are put forward. Speculation in the context of animation is even more necessary because animated displays require visualization of many aspects of each element despite the inevitable lack of data on certain details. The depiction of elements with missing information poses problems for the animator. Creative decisions depend on the ultimate purpose and focus of the narrative, but the fact that data are missing should also be of interest to certain viewers. Interestingly, it is sometimes during the development of visualizations that it becomes apparent to domain experts exactly what remains poorly understood1; in these situations it is necessary to speculate5. In other cases features are intentionally depicted without the support of references and decisions are made arbitrarily, for simplicity and clarity, or for aesthetic reasons. Sometimes visual information must be supplied that is not physically applicable, such as color and other surface properties at molecular scales. These cases can be categorized under the general notion of artistic license5. We believe that explicit citation of speculation and artistic license adds transparency and credibility to animated works. Furthermore, promoting a single point of view or comparing multiple existing hypotheses is a legitimate use for scientific animation and can involve selectively excluding data to emphasize a specific view1,5. We then categorize how these data sources may be manipulated or interpreted (Table 4) as direct import, adaptation, pruning, smoothing, interpolation and sampling. Most computer animation software has the capacity to import data in a variety of formats. Plug-ins created specifically for biomedical animators (for example, ePMV, Molecular Maya and BioBlender) can simplify the process of using empirical data in animation. Data may be imported and used as is (direct import). However, in some cases data may be missing or noisy or include extraneous information6. For example, outliers may need to be removed or the data may require simplification for
Table 3 | Categories of reference Type
Description
Examples
Visual
Translated from visual sources
Photograph, micrograph, illustration, video
Quantitative
Translated from numerical data
Size, speed, frequency
Qualitative
Translated from descriptive data
Color, process, interaction
Speculation or artistic license
No reference; estimated or imagined information
Missing data, simplification for clarity, visual aid
increased clarity (pruning). Occasionally, sampling artifacts must be addressed or visual noise eliminated (smoothing). There are many situations in which an artist must fill in missing pieces from available data (interpolation). For example, 3D objects are often generated from 2D references or virtual transpositions are performed to present a view different from that of the original source. Interpolation differs from speculation in that the depiction is still primarily based on data. Qualitative sources often require interpretation to render them in a visual format, and quantitative sources sometimes require processing and translation (adaptation). Finally, an animator may simply select pieces of information to supplement an artistic interpretation (sampling). Note that this use of “sampling” is different from the statistical concept of collecting a representative set of data from a population. Our definition includes the intentional choosing of information that supports the desired narrative and exclusion of the remainder. It is worth noting that more than one kind of use may be applicable for a single reference. Although these categories and properties may require future revision to successfully define the constituents of every conceivable animation, we propose that a breakdown of animated elements and their corresponding references using the suggested categories could provide a much-needed approach for hierarchically organizing the sources that inform scientific animation.
Implementation Rigorous citation demands careful management throughout a complex pipeline (Supplementary Fig. 1). During pre- production, production and post-production, an animator should keep references organized and add notes and tags to sources as they are collected and used. Many reference-management applications include the ability to annotate sources with labels or populate custom fields. Whether references are managed with custom software or simple spreadsheets, it is beneficial to review all citations at a point during post-production to confirm their accuracy and relevance, as reference usage and information content may evolve through pre-production and production. Next, the animator must decide how to present this collection of information to audiences. Analogously to the prevalence of different citation styles for written documents, animators and scientists together will need to develop different formats for presenting citation information to audiences, with the goal of providing the most information with the greatest clarity. Viewers should be able to access and navigate the information bidirectionally: one viewer might be interested in a specific visual element and the sources supporting it, whereas others might search for a specific data source and see where in the animation it was used. Annotations could be overlaid on videos using existing subtitling systems that can be toggled on or off. To avoid a split-attention effect7, it might
Table 4 | Reference uses Type
Description
Direct import
Information is imported directly or copied Proportions of different species found in an one-to-one with no modification ecosystem
Examples
Adaptation
Information is translated into another format (typically visual)
Interaction of molecules from description of process
Interpolation
Missing information is approximated
Full molecule built from partial crystallographic data
Pruning
Some information is removed
Artery shown with certain branches omitted
Smoothing
Artifacts or other raw characteristics are minimized
3D model from computed tomography scan slices
Sampling
Selected pieces of information are used
Two representative structures shown from database of 40 structures
nature methods | VOL.12 NO.4 | APRIL 2015 | 295
commentary be preferable to provide citations associated with representative still frames or ‘thumbnails’ in an accompanying document. A simple format would be static and linear, yet informative and tightly structured (Fig. 2a). More dynamic and interactive formats can be designed (Fig. 2b), which require more work to produce but allow for quick navigation among thumbnails and elements. In addition to providing the reference type and usage labels that we have suggested (Tables 3 and 4), it can be extremely valuable to add a descriptive note. This kind of informative text is epitomized in the miniseries “Illustrating the Machinery of Life”8, which
describes the scientific literature supporting an illustration. The benefit of a longer article of this type is the freedom to describe in detail exactly how different information was incorporated and why certain decisions were made (for example, among conflicting data). But a problem with this type of article is the lack of predefined structure and utility as a quick reference. We believe that including a brief comment on reference use for each item in the bibliography can be invaluable. Animators are often skilled at implementing their own custom software solutions to expedite common tasks. We anticipate this will naturally result in novel and
Discussion Our primary goal in promoting the inclusion of content-linked bibliographies with
b
npg
© 2015 Nature America, Inc. All rights reserved.
a
improved tools for automated reference management and presentation. These may include custom plug-ins that interface between reference-management software and animation packages or drag-and-drop interfaces for attaching citations to locations on an image. There is also the potential for complex data visualizations with coded information dynamically overlaid on an animation, perhaps highlighting regions of robust scientific support.
Figure 2 | Example output formats for using citation in a scientific animation. (a) Sections from a linear document. The document begins with key references, summarizing the information presented in the animation as a whole. These are followed by thumbnails and associated references. An example thumbnail with shaded and labeled elements is accompanied by a legend. The thumbnail is assigned a numerical identifier along with a time code so that its location in the original piece may be easily found. Each element is listed with properties and associated references. References are numbered sequentially through the entire document to allow indication of references used across multiple thumbnails. Finally, reference type, use and any explanatory comments are presented in italics. A partial set of references informing the elements in this thumbnail are shown. (b) Screen capture of an interactive module. A strip of images along the top allows a user to select from thumbnails representing all elements in the piece. The selected image has labeled and clickable regions outlining each element, a time code, and descriptions of audio elements that may require their own references. The text area below displays references used for a selected element, in this case the dynein molecule. The information is organized as in a. 296 | VOL.12 NO.4 | APRIL 2015 | nature methods
npg
© 2015 Nature America, Inc. All rights reserved.
commentary animated media is to increase the scientific credibility of animation through increased transparenc y and accountability. A thoughtfully constructed bibliography will provide scientists with greater context for a particular animation and allow them to internalize relevant information and investigate areas of speculation rather than dismiss an animation in its entirety because it is based in part upon supposition. Viewers would have a means of comparing citations across animations addressing similar subject matter, and animators would discover sources that colleagues find useful. A general increase in communication and crosspollination of ideas should result from content creators sharing sources with each other and with viewers. Just as research is an iterative process, animations may be improved as scientific evidence becomes available; this process of improvement will likely be aided by the inclusion of citations. Audiences, including scientists, will better understand the storytelling and decisionmaking processes that lead an animator to employ various references in different ways, and content creators would be encouraged to be more intentional in selecting and using reference material. It also becomes important to include a creation or publication date with animations so that future audiences can understand references in the context of when the piece was developed. We believe that the citation process need not necessarily describe every reference used. A viewer would neither expect nor desire access to a large quantity of reference material supporting the design of inconsequential features of the display. The animation production team would be responsible for selecting and sharing the sources that are likely to be relevant or of interest. This type of information filtering is already part of the expertise of biomedical communicators. Animators are tasked with highlighting key points of data, so as to increase the clarity and comprehensibility of complex material. We are aware of potential limitations and drawbacks in using a citation system with animation. From the point of view of an animator, there may be significant overhead in
time (and therefore cost) to the production pipeline through cataloging and maintaining rigorous records of reference use. Although this may be a time-consuming effort, the long-term rewards are worthwhile. There may also be concern among studios that disclosure of sources and other production information would harm them competitively. Weighing the benefit of increased credibility against guarding research methods remains in the hands of the developers. Some may feel that any animation citing “speculation” or “sampling” of data will be called into question or criticized for manipulation. Misrepresenting information should of course be avoided, but it must be remembered that the broad range of uses for scientific visualizations require different degrees of freedom for the use of speculative content. Including clear indication that certain aspects of an animation are speculative should not detract from but rather increase the credibility of the creator. It is outside the scope of this article to discuss how and when references should be used. Instead we assume that references are used and wish to propose ways by which these details can be categorized and presented. Not all bibliographic formats are equally suitable in all situations. A successful bibliography must facilitate an understanding of the references in the context of the visuals. In the case of video hosting sites, one may provide a link to an accompanying document hosted elsewhere that the viewer could peruse while rewatching the film. At a conference, it is more difficult to display all references, but a paper copy of thumbnails and citations could be provided, a URL to access citation information might be visible adjacent to the display, or the presenter may simply highlight a few key references that informed a central point. The availability of an animation for additional viewing will affect the utility of different formats for providing citations. If an audience member sees an animation only once and has no further access to it, an online bibliography would require detailed thumbnails and clear labeling of elements to be of much value (Fig. 2). Complete freedom in the method
of p resenting references will offer the greatest value to viewers, but even venues with the most specific restrictions for media formats should be able to accommodate some form of annotated bibliography and the added support it provides. It is our hope that scientists will encourage the reporting of reference use and that animators will further develop productive ways to share this information. The ideas discussed here may be adapted and implemented for other media, such as illustrations and interactive modules, which suffer many of the same citation pitfalls as animation. Regardless of medium, content creators and audiences alike should consider the merits of adopting more formalized methods for citing resources. By clearly laying out both where and how references are used in visual communications, animations may gain greater credibility, which can lead to more open and productive dialog and dissemination of ideas and knowledge. Note: Any Supplementary Information and Source Data files are available in the online version of the paper (doi:10.1038/nmeth.3334). ACKNOWLEDGMENTS This research is supported in part by grants NSF #DUE1220512 from the US National Science Foundation and SSHRC #SIG-13/14 from the Canadian Social Sciences and Humanities Research Council. The image in Figure 1 was adapted from an animation by S.G.J. (https://vimeo.com/70039214/). AUTHOR CONTRIBUTIONS S.G.J. developed the concepts and wrote the manuscript. J.J. and G.M. provided direction and contributed to the manuscript. COMPETING FINANCIAL INTERESTS The authors declare competing financial interests: details are available in the online version of the paper (doi:10.1038/nmeth.3334). 1. Iwasa, J.H. Trends Cell Biol. 20, 699–704 (2010). 2. Anonymous. Nat. Methods 9, 1127 (2012). 3. Johnson, G.T., Autin, L., Goodsell, D.S., Sanner, M.F. & Olson, A.J. Structure 19, 293–303 (2011). 4. McGill, G. Cell 133, 1127–1132 (2008). 5. Goodsell, D.S. & Johnson, G.T. PLoS Biol. 5, e308 (2007). 6. Goodsell, D.S. J. Biocommun. 35, 52–57 (2009). 7. Chandler, P. & Sweller, J. Br. J. Educ. Psychol. 62, 233–246 (1992). 8. Goodsell, D.S. Biochem. Mol. Biol. Educ. 37, 204–210 (2009).
nature methods | VOL.12 NO.4 | APRIL 2015 | 297
