Visualization for epidemiological modelling: challenges, solutions, reflections and recommendations.

We report on an ongoing collaboration between epidemiological modellers and visualization researchers by documenting and reflecting upon knowledge constructs-a series of ideas, approaches and methods taken from existing visualization research and practice-deployed and developed to support modelling of the COVID-19 pandemic. Structured independent commentary on these efforts is synthesized through iterative reflection to develop: evidence of the effectiveness and value of visualization in this context; open problems upon which the research communities may focus; guidance for future activity of this type and recommendations to safeguard the achievements and promote, advance, secure and prepare for future collaborations of this kind. In describing and comparing a series of related projects that were undertaken in unprecedented conditions, our hope is that this unique report, and its rich interactive supplementary materials, will guide the scientific community in embracing visualization in its observation, analysis and modelling of data as well as in disseminating findings. Equally we hope to encourage the visualization community to engage with impactful science in addressing its emerging data challenges. If we are successful, this showcase of activity may stimulate mutually beneficial engagement between communities with complementary expertise to address problems of significance in epidemiology and beyond. See https://ramp-vis.github.io/RAMPVIS-PhilTransA-Supplement/. This article is part of the theme issue 'Technical challenges of modelling real-life epidemics and examples of overcoming these'.

Introduction, context and intent

Collaboration between epidemiological modellers and visualization researchers working in the UK in response to the COVID-19 pandemic has been broad, intense and productive. Research and practice in visualization and visual analytics (VIS) have enriched and advanced the models and the modelling process as a result. The efforts and experiences have in turn contributed valuable knowledge and developments to VIS. We report and reflect upon this ongoing interdisciplinary activity and the knowledge generated through our collective experience.

Epidemiological modelling for the COVID-19 pandemic

Epidemiological modelling has played a significant role in informing policy for the COVID-19 pandemic in the UK [1-3] and as evidenced throughout this special issue. It involves the development and use of mathematical and computational techniques to describe the spread, evolution and control of epidemic disease [4,5]. The models in use are enormously varied and employ a wide range of techniques, including mechanistic mathematical approaches [6,7], statistical models trained from disease data [8-11] and computational micro-simulation of agents [12-16] as well as complex model emulators [17]. Each aims to generate outputs to help understand the past, current or future course of an epidemic while considering context-specific strategies for mitigating spread. Mathematical models have limitations and uncertainties, especially when modelling novel infections such as COVID-19. Moreover, different types of models can give different types of insight even when they are describing the same epidemic process [18,19]. For these reasons it is also important to effectively communicate the underlying model assumptions and the uncertainties in their estimates [20].

Designing, constructing and using such models, therefore, comes with significant challenges. In responding to an emergency we need to develop good answers to complex questions at speed: Efforts to quickly and reliably establish plausible answers to these questions during the COVID-19 pandemic have involved teams of academics working in parallel on complementary models and rapidly developing new collaborations and ways of working to support their research. Interdisciplinary expertise in, for example, software engineering, spatial analysis and visualization have been drawn upon as modellers have endeavoured to address the various challenges involved in responding to the pandemic in timely and informative ways.

Which structures and techniques are most effective and informative?

How can we be sure there are no bugs or errors?

How much detail should be included in the model?

What data should be used as inputs for the model and at what spatial and temporal scales?

What are the effects of using different datasets and model parameter settings on model outputs? How do these effects vary?

How can model mechanics and outputs be effectively communicated to diverse audiences?

How can the models be used to inform and compare policies?

How can model-informed decisions be effectively and widely communicated?

Visualization and visual analytics

Visualization emerged as a challenging and enabling application of computer science in the early 1990s. It offered a paradigm shift in the way that processing power and capabilities for rendering were applied to scientific data [21]. Drawing upon long-standing traditions in cartography [22,23], statistics [24,25], graphic design [26-28] and cognitive science [29], and leveraging their developing interactions [30-32], it was perhaps best captured by Mackinlay [33]. This shift sparked cutting edge research in the development of hardware, software, theory and techniques for depicting and interacting with data. The need for tight integration between graphics and analytical capability, interactive rendering speeds for rapid updates and support for analytic workflows for discovery has developed out of this initiative [34] as researchers and designers have supported data analysts in science, government and industry through visualization and visual analytics.

The diverse forms of knowledge [35] that have resulted include techniques, methodologies and epistemologies that enable VIS to contribute meaningfully and effectively to problems, ranging from highly specialized academic domains to urgent and imminent global challenges. Rapid and flexible interactions with rich graphical depictions of data enable us to understand the complexities and nuances of atmospheric models [36-38], poetry composition [39,40], animal ecology [41-43], sporting performance [44-47], transport systems [48-53], evolution [54,55], cyber attacks [56-58], energy consumption [59,60], healthcare [61,62], genetics [63,64] and many other aspects of nature and society including epidemics and epidemiology [65-71].

Rapid assistance in modelling the pandemic

The Royal Society contributed to the interdisciplinary effort to address the COVID-19 pandemic in the UK by convening the rapid assistance in modelling the pandemic (RAMP) initiative [72]. RAMP identified willing volunteers with potential to contribute to the challenges involved in modelling and linked them with candidate epidemiological modelling groups. The Scottish COVID-19 Response Consortium (SCRC) formed as part of this RAMP initiative with a focus on developing a robust understanding of the impacts of different exit strategies from lockdown [73]. Some 150 epidemiologists, software developers, mathematical modellers, data scientists and others came together to do so. SCRC made a call for visualization volunteers to help with the effort in May 2020. As a result more than 20 visualization researchers and developers, from 11 universities and two companies in the UK, offered to volunteer by providing visualization support for the epidemiological modelling in SCRC.

Visualization theory [74] was used to structure this activity, specifically to establish the broad applicability of visualization in modelling workflows and communicate this persuasively to modelling scientists [75]. Activities involving various modelling workflows were supported concurrently (figure 2) by creating seven groups to work in parallel. They provided either model specific or more generic support for visualization and associated analytical capability. Four groups of visualization researchers sought close collaboration with specific modelling teams working on particular models. The intention was to explore opportunities for using graphics to provide rapid feedback on model inputs, outputs and inner workings with the intention of addressing the challenges involved in model development (§1a). Three central visualization support teams provided a complementary more generic focus on: the rapid development of a visualization infrastructure for interactive visualization of the data made available through the SCRC data pipeline [76]; the visual communication and dissemination of the SCRC work to a broader audience; and the provision of analytical capability to the other groups.

Figure 2

Project architecture expressed using the Four Levels of Visualization [140] model. Structuring and planning our engagement in light of the model [98] helped us coordinate the SCRC volunteering effectively, efficiently and flexibly. By identifying needs for disseminative, observational, analytical and model developmental visualization we were able to identify six activities, and plan for the different amounts of effort needed for each. This enabled us to deploy the available VIS volunteers based on their expertise appropriate for each level of tasks and provide flexible visualization support across SCRC [140]: 729. We used the model to develop an iterative approach to establishing opportunities, supporting and developing needs, prototyping solutions and reflecting on action that underpins this paper.

The volunteering took place from June 2020 to January 2021. This initial effort has been consolidated by a follow-on project funded by the UKRI/EPSRC through their COVID-19 Rapid Response programme—RAMP VIS: Making Visual Analytics an Integral Part of the Technological Infrastructure for Combating COVID-19. Running until January 2022, this subsequent 12-month project has helped maintain and develop the activity generated through the volunteering, while providing additional resources to enable further capability, research and communication [75,77].

Supporting epidemiological modelling with visualization research

Much of the knowledge generated in visualization research is encapsulated in distinct visual designs. Munzner [78] adopts the terminology of scientific visualization pioneers Haber & McNabb [79] in describing these designs as idioms—combinations of design decisions that result in particular styles or forms of visual data depiction that are likely to be useful for particular types of user to support particular types of task [80]. These idioms can be considered as graphical templates that offer (sometimes loosely) defined styles of visual communication. They include well known graphic types such as scatter plots [8] (fig. 4); choropleth maps [12] (fig. 2), [15] (fig. 4); bar charts [7], (fig. 5), [81] (fig. 4); heat maps [82] (fig. 2), [81] (fig. 4), [17] (fig. 3); time lines [8] (fig. 8), [15] (figs 5 and 6), [17] (fig. 2) and box plots [83] (fig. 2), as well as many less celebrated and more specific graphical devices [84]. Idioms also involve approaches to generating composite graphics, for example through Small Multiples [8] (fig. 7), [7] (fig. 4), [83] (figs 4 and 5), [17] (figs 5 and 6). Knowledge about how to use visualization effectively in any applied context draws upon these established visual solutions and their use to address particular problems (combinations of task, dataset and user) [80]. It is frequently and beneficially shaped in the crucible of applied work. Some of this knowledge is explicitly captured in a series of complementary efforts to determine and communicate what we know about visualization through established idioms (e.g. [85-88]), but much of it remains implicit and is drawn upon, and accumulated, in applied contexts through the visualization design process. Various models describe this process of understanding tasks, users and data, and applying and developing visualization knowledge in specific beneficial ways. They detail the strategies to apply, the activities in which to engage, the pitfalls to avoid and the qualities expected in rigorous work [89-97]. Munzner [78] emphasizes the importance of these idioms for knowledge transfer between contexts during the design process in noting the crucial role of: ‘existing idioms as [providing] a springboard for designing new ones’.

Our approaches were framed by these ideas, but, in the spirit of transfer, adapted to the rapid nature of our emergency context. Applied visualization research usually involves system design and development that takes place through an ongoing iterative process of redesign, resulting in a usable visualization system. Typically, complimentary processes occur at different time scales [98], including: Much of our initial collaboration involved an iterative process of finding opportunities for beneficially using known graphical approaches, with Design Study Methodology (see fig. 1 of [95]) offering good guidance for identifying the kinds of problems where visualization may be appropriate. We quickly identified known idioms for transfer from the broad body of visualization knowledge that might be applicable in such cases. Working at speed, visualization researchers drew upon tools and techniques in which they had expertise to design and deliver solutions that could be injected into the modelling processes and workflows to contribute to discussion and understanding.

—candidate templates that have been shown to be effective in particular combinations of task, user and data [80] are quickly applied to an established problem in which the characteristics of task, user and data are comparable;

—data rich interactive prototypes are rapidly developed to refine these ideas and iteratively redefine problem (supported by task) and solution (idiom informed visualization design) in light of data, user reactions, learning and reflection on task;

—to address identified task(s) in ways that are transferable to other datasets (and tasks) through stable reliable capability, usually in persistent software that applies, refines, combines and gives access to idioms proven to be effective in earlier stages.

The context of the COVID-19 situation involved disruption, but provided urgency and focus. The digital workplace of lockdown with regular video calls, seminars and other digital knowledge exchange enabled, and indeed required us to Design by Immersion [89]. This helped visualization researchers to quickly develop an understanding of a series of models (data) being developed in parallel by modelling teams (users), and identify questions (tasks) to which known visualization approaches might be usefully applied. They produced designs that ‘springboarded’ existing visualization knowledge and capacity by applying and developing known idioms and appropriate technologies as described above. The epidemiological modellers learned concurrently about visualization possibilities and practice as visualization was applied to their data and tasks. This achieved the kind of reciprocal immersion, shaping and influence [89,91] between epidemiologists, modellers and visualization experts that is so core to effective visualization support and successful applied visualization research [92]. We considered and reassessed needs and capability through this process of iterative transfer and redesign as knowledge of the models and possible disease progression was established at pace.

Challenges, solutions, reflection and recommendations

This activity, and our subsequent consolidation through RAMP VIS, has resulted in a central visualization server that offers: hundreds of plots and composite dashboards depicting data on the core pandemic indicators [99,100] (figure 4); analytical agents that automatically transform raw data to be visualized by the central system; a collection of analytical routines and algorithms that offer generic analytical capability for exploring time series based data; a series of static and interactive visualization prototypes to support the four modelling teams. Work on the modelling support prototypes has resulted in, for instance: new ways of representing and interacting with data for contact tracing and assessing model inputs and outputs (figure 1); improved epidemiological models and understanding of them (figure 3); new connections between research groups and researchers; new attitudes to the use of visualization in epidemiological modelling and shared knowledge about how this can take place; and funding to support ongoing work. In line with SCRC protocols we used open source solutions to ensure transparency, invite scrutiny and prepare for reuse [92]. A series of academic articles, both published and in preparation, in the visualization and epidemiological modelling domains, report on specific results that combine technology, design and data successfully in new and revealing ways, e.g. [75,99,100].

Figure 4

We developed an effective means of generating thousands of viable online interactive visualizations and hundreds of dashboards by developing a means of Propagating Visual Designs from the SCRC data streams [149]. This semi-automated process maps single or multiple datasets to particular visual forms (plot types). Our application also supports quality assurance over the propagation process, to ensure the propagated visual designs are reasonable. The approach uses an ontology [150] to facilitate propagation by formally representing the relationship between dataset and visual device. This provides both flexibility and generalization as the mappings are determined by the ontology, which is established by human experts. The process provides a good example of the kind of human in the loop decision-making that visualization can support, resulting in down-stream efficiencies in this instance by encoding expertise that can be propagated and thus making good use of expert time. Here, visual analysis supports efficient and effective visualization design as we search for appropriate design solutions for observational visualization [140]:729.

Figure 1

Six excerpts of an animation of a modelled disease transmission network [101] are visualized here using DynNoSlice [102,103]. Points are people, with connections showing infection pathways between them and revealing the fragmented nature of a modelled disease outbreak. The final image (right) shows detail, with a close up of the largest components in an infection network, with nodes colour-coded by infection state. Modellers responded to this visualization by lowering the random infection rates used in the modelling.

Figure 3

Nine excerpts (left) of an interactive exploration of a modelled disease outbreak visualized using our Gridded Glyphmap prototype [155]. Cells represent interactively defined areas in Scotland, with areas of higher populations being shown in more vibrant colours. Each glyph shows proportions of population in particular disease states (colours, horizontal proportions) in 10-year age bands (vertical rows), revealing the spatial and age-based characteristics of a modelled disease outbreak. The large image (right) shows a wider spatial overview of a single time-point at a particular scale. Modellers interacted with the output data to reveal patterns that resulted in changes to the model code and deeper understanding of the effects of modification to the model as knowledge of the disease progressed.

This paper does not focus in detail on the specific developments that took place. While we document some of these in ways that make them open for potential transfer in the electronic supplementary materials that are a key contribution in this publication [104], our main intention is to report on the work more broadly and learn from the collective experience of the SCRC volunteering and our subsequent efforts to consolidate. We aim to capture knowledge that emerges by stepping back and sampling from the mass of parallel activity that has occurred to provide a project-wide view. Doing so is intended to develop knowledge about visualization in epidemiological modelling during emergency response that may also be used in other domains and contexts—perhaps other epidemiological emergencies, perhaps other kinds of emergency, perhaps other forms of applied visualization. This kind of ambitious and challenging (probably unique and inherently partial) meta-analysis enables us to shine a light on what visualization can offer as a discipline and on how visualization research and researchers can contribute in epidemiology, and by inference elsewhere, to support and enhance scientific activity with informative interactive analytical graphics. The intention is thereby twofold—to use some of the rich and diverse experiences of the RAMPVIS collaboration: These are not unrelated aims, and we address them by documenting challenges encountered and solutions developed during the collaborative activity. We do so in the context of a series of existing knowledge constructs selected from the VIS literature. We use these to reflect collectively upon the activity and make recommendations to support and develop ongoing collaboration and research. This enables us to assess whether, how and where visualization can be effective, identify open problems emerging from the efforts, provide guidance for future collaborative activity and make recommendations for developing the visualization community and visualization capacity in ways that can be beneficial in epidemiology and beyond. Our intention is to use this paper and its electronic supplementary materials to showcase VIS and its role in observation, analysis, model development and dissemination in epidemiological modelling. In doing so, we demonstrate directly the value of the kind of interdisciplinary collaboration supported by RAMP and by inference, the potential for use of visualization approaches, knowledge and capacity in other domains.

to present visualization knowledge and practice to a broad scientific audience through example, and

to contribute new knowledge to the visualization community through experience.

Approach

Given the rapid, distributed, dynamic, unpredictable and sometimes chaotic nature of the engagement, in which researchers were volunteering their time while interacting with new colleagues and new domains, and while experiencing the pandemic themselves, our knowledge claims are not underpinned by a deeply planned data collection methodology. However, we have taken time and steps to develop and coordinate reflection upon the individual and collective experiences—some of which were not shared until we began this very process. Doing so enables us to make tentative suggestions about visualization, visualization design processes and how visualization might be used more fully in epidemiological modelling and perhaps other domains in the future. This speculation is based upon discussion among the researchers involved, with cross-pollination of ideas and reactions, both to the visceral experiences of the summer of 2020 and the subsequent work in supporting epidemiological modelling through visualization.

Selection and reflection on constructs

We used a structured process to document and reflect on our engagement. The methods are in line with Design by Immersion activities S-2 ‘document’ and P-5 ‘critique’ [89], which resonate with key concepts used in other visualization design process models such as the guided emergence principle of Action Design Research [91] and the record and reflect recommendations for interpretivist design study [105].

In preliminary reflective activity, visualization volunteers were asked to identify and select specific examples of established visualization knowledge that they used in initial transfer as the springboard for supporting epidemiological modelling. We asked the volunteers to select idioms and other knowledge constructs that they applied, and use these as a basis for documenting and reflecting upon their experiences and establishing their learning. We define a visualization knowledge construct, henceforth ‘construct’, as Given the importance of collaboration and communication to the endeavour, and previous positive experience [106,107], we adopted the computational notebook paradigm for documentation of these constructs and reflection upon them. Notebooks encourage explicit reasoning around design choices with explanatory text surrounding embedded code that implements and renders graphics. Based on the ideas of Literate Programming [108] and the computational essay [109], they support activities and principles that are core to effective applied visualization design. Examples include C-2 ‘common knowledge’, C-3 ‘peer-to-peer communication’ and C-4 ‘translate concepts’ [89], along with P-3 ‘reciprocal shaping’ and P-5 ‘authentic and concurrent evaluation’ [91]. Use of these activities and adherence to these principles offers a form of internal validation [92] in supporting information exchange and discussion between collaborators.

Something that is explicitly known and understood in the visualization body of knowledge that was transferred to address a problem in the current context of visualization support for epidemiological modelling.

The result of this collective reflective activity is 29 commonly structured but independently authored digital notebooks. They are listed in table 1 and available for detail and scrutiny through the live links provided in this document. They focus on specific constructs selected by volunteers as a frame for reporting and reflecting on their contributions and experiences. Each explains the context in which the construct was applied and the problem it was intended to solve through a ‘User Story’, before documenting what is known about it, why it was considered a candidate for transfer, how it was used and how it performed. Potential for further transfer is supported in many cases through links to illustrative examples of the construct in use beyond the current context, along with full sources that describe the original research. Critically reflective thinking [110,111] on the transfer and use of the construct to support the epidemiological modelling is then recorded as researchers document their experience and assess what was learned as a result. We use Observable [112] (formerly d3.express) as the computational notebook platform to benefit from its emphasis on reactive literate visualization [107], which supports visualization-rich discourse. This approach and technology encourages reflective reasoning in something approximating a reflective schema [111] and makes it explicit and available both for analysis as part of the design discourse and for wider scrutiny as we assess and synthesize experiences to develop broader claims.

Table 1

RAMP VIS knowledge constructs: selected examples of visualization knowledge used to reflect on visualization support for epidemiological modelling. Interactive notebooks describing each are available as electronic supplementary materials [104].

D3 Prototyping	[133]	an expressive JavaScript library for delivering VIS on the Web
New Data Spaces	[156]	exploring data in transformed data spaces
VIS Design Study	[155]	modifying design study methodology for emergency response
Gridded GlyphMap	[154]	interactive animated maps show model outputs by age group and disease state
Pictograms	[152]	pictorial symbols for informing the public about modelling outputs
GMap	[159]	understanding models by visualizing network clusters as regions
NAPA Cards	[151]	narrative design patterns for data driven VIS storytelling
Linked PPLots	[134]	explore relationships between parameters with parallel coordinates
Alternative Views	[132]	using complimentary visualizations to explore dynamic networks
Visual Faceting	[146]	interactively cut, sort and align time series for comparison
VA for Modelling	[145]	integrate algorithmic tools for VA into modelling workflows
Propagating Designs	[148]	a user interface and workflow for automatic generation of visualizations
Flow Curves	[157]	map flows with direction and quantity between regions
Linked Views	[135]	multiple linked views for parameter space exploration
Ontology for VIS	[149]	a knowledge base that pairs data streams with visualizations
Workflow Analysis	[141]	theory-guided optimisation of VA workflows analyse—cause—remedy—side-effect
Parallel Plots	[136]	parallel coordinates plots for model parameter summary and selection
Error Encodings	[137]	visual representations of error and uncertainty in parameter space
Five Design Sheets	[153]	a structured sketching approach to visualization design
Four Levels of VIS	[139]	a complexity-based scheme to categorize VIS task complexity and effort
OD Maps	[158]	showing the geographic variation of flows in spatial interaction matrices
Scalable Pixel Views	[138]	dense visual representations show variation of each parameter value
Dynamic Data Vis	[150]	dynamic data scheduling agents update VIS systems automatically
Infrastructure	[147]	technology for delivering VIS to ensure readiness in emergency response
Deployment	[131]	developing reliable visualization systems with an agile approach
VIS Guidelines	[140]	a community driven approach to extract, record & transfer VIS knowledge
VIS Theory	[142]	a call for the development and use of theory in VIS
Volunteer VIS	[143]	views on provide VIS support through volunteering in an emergency
Design by Immersion	[144]	opportunities for transdisciplinary VIS design & activities to support it

The diverse forms of contribution made by volunteers in the various roles in the project and the variety of efforts to which they were dedicated (see figure 2) enables us to consider a range of forms of visualization knowledge construct. Some are idioms, visual design templates in the Munzner [78] and Haber & McNabb [79] sense, where the focus is predominantly on transformations from data to graphics. But we capture a wider range of visualization knowledge types, such as enabling technologies [134], theoretical models [148], design methodologies [145,156], narrative devices [152] analytical approaches [157] and system architecture configurations [150] in this rich data source.

Synthesis and reflection across constructs

In a second stage of critically reflective activity we consider, compare and contrast these post-event reflections ‘on action’ [113]. This enables us to suggest key themes that emerge through an inductive process that aligns with the relativist approach to knowledge generation taken here [92,111,114]. The stages of analysis were not entirely separate. Each notebook was developed iteratively, with the individual experiences prompted and informed by the higher level concepts as they emerged as part of the process of collective reflection. Feedback to notebook authors (that we retain as a record in the electronic supplementary materials) enabled us to prompt for ideas, question, refine and improve descriptions, calibrate where possible, communicate effectively and relate experiences to emerging cross-cutting themes as we developed our evidence base, reflected upon its characteristics and synthesized the diverse views and experiences. This reflection on the ‘data’ captured in our notebooks is the basis for a series of cross-project findings.

Sedlmair et al. regard reflection as being ‘where research emerges from engineering’. Implementing our reflective process enables us to document the transfer of some of what is known in visualization to the domain of epidemiological modelling and test and develop this knowledge in the crucible of the pandemic response. Given this approach, we offer no claims of absolute and indefatigable truth, but are careful to relate themes and claimed findings to specific evidence and particularly value cases where multiple experiences converge in our cross-construct meta-analysis.

Findings—results and claims

The diverse array of constructs, listed in table 1 and described fully in our electronic supplementary materials [104], gives us a broad and informative set of experiences of using visualization methods and approaches in emergency response. This is our evidence base. We provide direct access to it through hyperlinks to the cited notebooks themselves (e.g. [159]) or specific cells within them that contain explanations, examples, graphics and reflections (e.g. [159]:1912). It is something of a trrrace [105] through which we record, report and reflect on our activity.

This abundant evidence base of diverse experiences addresses the overarching tacit hypothesis with which we approached the work: It leads to some specific findings about when and where this may be the case and how it might be achieved. Relating experiences across activities through the structured reflection and synthesis described in §2 allows us to develop some more general findings and claims and speculate a little about possible transfer to other related problems, domains, datasets and contexts. These enable us to:

visualization can be effective in epidemiological modelling.

add to the growing body of evidence supporting the utility and applicability of visualization in the analysis of complex datasets across domains, and understand how this can be achieved;

identify open problems for consideration by the visualization research community as it engages in a wide set of problems and domains;

suggest approaches for an effective process of engagement, analysis and design when domain specialists are working with visualization researchers and designers to problem solve with an emphasis on emergency response; and

make recommendations for the VIS and wider scientific and academic communities, in terms of actions that can be taken and priorities that can be established, for more widespread and informed use of visualization in scientific discovery and emergency support in the future.

We begin with six broad claims and evidence to support them: visualization can be effective; visualization is not always appropriate; visualization is experimental; visualization capability is collective; visualization processes evolve; and visualization inspires research.

Visualization can be effective

Our experiences, efforts and reflection show that visualization can contribute effectively at various levels in the epidemiological context. This is the case not only for communicating results, but to address the questions outlined at the outset (§1(a)) by observing, analysing and modelling in line with the Four Levels of Visualization (disseminative, observational, analytical and model developmental) [98] [140] used to frame and coordinate the engagement (see figure 2). For example, by respectively:

presenting information in ways likely to engage and inform broadly [152]: 766, [154];

applying numeric and graphical transformations to reveal structure [159]:2072, [157]:684—see fig. 1;

showing relationships between time series and places at various scales [155]:788, [158]:938, [147]:669—see fig. 3;

comparing large numbers of model inputs and outputs at multiple scales [138,139,146], 170.

Existing visualization idioms provide plenty of potential for rapid deployment and effective application to epidemiology. This holds in the case of considerable complexity [135,139,147], dynamism [151,155] and uncertainty [138]. Much of what was achieved was novel in terms of the designs developed, algorithms produced and the epidemiological context to which the ideas were applied—such as the use of interactive linked views to represent uncertainty and error: ‘a first under the epidemiological setting’ as captured in [138]:427.

We characterize these successes by presenting six ways in which our documented reflection provides evidence to support the view that visualization can be effective.

Achieving important insights

Our collaborations helped us understand the models and the disease, and also the capabilities, limitations and scope of some of the visualization designs and approaches. These insights were often mutually shaped [91] and interdependent.

Domain insights—knowledge about epidemiology and models—involved, among other things:

refining different models, [133]:592, in ways that had not been picked up in standard static graphics [155]:788 (see figure 1);

understanding the effects of varying model inputs [159]:310; and

revealing the complex and subtle geographical characteristics of the modelling [155]:788 (see figure 3).

Equally, design and process insights—knowledge relevant to visualization—that may be reusable were achieved, such as We say more about the visualization design process in §3(e) (with a particular focus in 3(e)(iii)), but can claim with some confidence that:

the population weighted fading of glyphs used to make more populated areas more salient and effectively ‘focus the eye on areas where population numbers are higher for relative changes’ [155]:788 (as explained in figure 3);

a new semi-automated approach to mapping datasets with particular characteristics to plausible visualization designs for dissemination [150]:674 (see figure 4);

a method for validating such recommendations [149]:699;

confirmation of the effective use of narrative patterns [152]; and

an automated process for daily data updates to visualization dashboards for dynamic data dissemination [151].

Visualization resulted in insights about epidemiology and the design and use of data graphics

Capturing and communicating complexity

Much of our reported activity demonstrates scope for showing highly dimensional data in meaningful ways that enable patterns to be detected and models to be understood [135]:867, [136]:427. We used existing approaches to design and develop workable representations of, for example, model outputs that varied 16 parameters with 160 different parameter configurations that were generated through 1000 stochastic runs for eight age groups. This resulted in 20 480 000 time series of 200 days each [137]:339, [139]:346. We applied and refined idioms to reveal structure in complex spatial relationships to help understand and select interaction networks [158]:1375. Our graphics present informative (often interactive—see §3(a)(vi)) visual overviews and allow analysts to interpret input data and parameter configurations—a considerable challenge [82]—and relate them to output patterns and variations. As such:

Complexity is captured effectively with visualization

Detecting variation, difference and change

Graphics that show variations are particularly useful. These may be in scale, over time, in outputs based upon stochastic modelling or in input parameter configurations. Applying idioms to help modellers detect and consider variation and change by rapidly comparing and assessing informative depictions of differences in data sets was particularly valuable and effective. In [155]:788 we hear that: ‘The ability to investigate the differences between different model outputs is critical, and has allowed us to identify problems with the model code itself, and to verify the impact of changes to our understanding of the disease on model outputs.’

We produced coherent graphics designed for comparison [115] broadly, including examples that successfully explore:

individual parameter configurations and clusters of similar configurations [139]:427;

model outcomes and their variation through uncertainty visualization [138]:400;

model outputs under particular conditions—such as location or age group [155]:771; and

the effects of the scale at which data were aggregated, which in one case ‘revealed unexpected dependency on the scale of spatial aggregation on the model behaviour’ [158]:1447.

These and multiple other examples and experiences support a strong claim that:

Visualization is highly effective and particularly valuable for comparison

Delivering detail

Graphics that inform decision-making and act as the basis for reporting often involve reducing datasets to summary statistics for simplicity. As we have seen, visualization offers richer, nuanced, informative alternatives that remain comprehensible while delivering detail [159]:276. We find compelling examples documented in the notebooks in which visualizations ‘unpack’ such indicators or metrics in informative ways to support the decision-making processes at different stages of the modelling workflow.

For instance, in cases where the progression of the pandemic within different simulation runs is compared using their respective values [116,117]. Here, disaggregated visualizations of infection events led to more fine-grained observations such as that ‘the number of random infections were too high which led to several small infection chains’ [133]:592. This not only leads to an improved understanding of the epidemiological model and complex transmission patterns, but also makes ‘comparisons between simulation runs which capture different policies’ a much more informed activity [133]:592. Figure 1 provides an example. Similarly, ensembles of simulations are more closely inspected and fully evaluated through visualizations that depict the input–output relationships ([138]:427) rather than relying on aggregate statistics of indicators such as the total number of hospitalizations that a model is forecasting within a given period. In cases involving the comparative analysis of time series, visualizations served as effective tools to determine the portions of time series that contribute most fully to summary measures such as autocorrelation [157]:676. Heat map visualizations of the distribution of raw time series values for the members of a time series cluster also deliver detail effectively. The graphics reveal patterns that allow the clusters to be described through observations such as ‘Cluster-1, the largest of the groups is the most typical 2-peak UTLAs with a much more pronounced 2nd peak’ [147]:720. In these various examples:

Detailed data are made interpretable with visualization

Transforming data

Deriving new data spaces through informative analytical transformations of data and exploring these graphically ‘reveals relations and structures not easily visible within the raw data’ [157]:684. The Critical Creative Culture that we established (§3(e)(vi)) resulted in various informative graphical transformations. The use of networks to show the similarities and differences between multiple time series is an example [157]:676. Similarly, our visualization of the principal components of a multivariate parameter space enabled informative ‘identification of which parameter settings generate more variations in its simulation and which settings generate more similar outputs’ [146]:427. Network layouts that use space to show structure proved effective [160]:814.

Some transformations are less abstract, such as OD maps [118]. These flow maps address the overplotting associated with spatially concentrated geographical data by stretching and warping geographical space. We used this transforming idiom to reveal differences in flows of people between locations used as an input to the simple network simulation model [159]:1912. In multiple cases, we found that:

New data spaces can be visualized informatively

Rapidly refining and relating perspectives

The ability to interactively change and modify data and views at speed is key to effective visualization [139]:427 (see also [119]). It helps analyse data, refine problems and develop designs [155]:788 by selecting, combining and comparing items of interest; changing focus and scale; or varying the way that information is shown—perhaps through alternative data spaces (§3a(v)). Doing so in rapid and fluid fashion tightens design iterations and analytical work flow loops, which proved effective: ‘Interactivity was one of the most important aspects to me in terms of model development. To be able to have a tool that I could interactively explore model outputs (and inputs) was very useful when adding new mechanisms or datasets’ [155]:788.

Interactivity was crucial to many of our successes—enabling modellers to see aspects of the model that would have been missed in static plots and adjust accordingly—e.g. Gridded GlyphMaps for comparing (§3a(iii)) two deterministic model runs [155], or when using parallel coordinate plots and heat maps to show parameter spaces [136]:400, resulting in modellers being ‘surprised by the amount of information made available by combining them’ [136]:427.

One important interaction involved rapid faceting and grouping that enabled us to explore variation in the data according to selected meta-data: ‘The ability to specify combinations of meta-data for grouping and sorting is effective to generate bespoke criteria aligned with the diverse needs of the application domain’. Doing so, through interactive visual comparison, led to insightful observations [147]:673. Indeed, Factoring in Flexibility (§3(e)(ii)) through rapid interaction was important throughout the collaboration:

Rapid interaction underpins the processes that benefit from visualization

Visualization is not always appropriate

These considerable successes are a subset of those achieved and documented in the supplementary notebooks, which themselves constitute a sample of the total activity. Yet the collaboration also showed that while graphical approaches can be applied to most data, they are neither necessary nor appropriate for all tasks, analysts or combinations of the two. For example, good statistical techniques are more than adequate, for solving problems that are well defined with data that describe phenomena fully. This is captured well by considering task clarity and information location [95], [156]:832.

But even where visualization offers potential, there may be good reasons for not pursuing a visualization engagement. Data may not be forthcoming, collaborators may be unable to prioritize efforts to view, engage with or validate designs and the kind of interpersonal relationships required for success may not be achieved. Visualization is unlikely to be a priority, or a success, in such circumstances. Design Study Methodology [95] captures these pitfalls well: PF-4 ‘no real data available (yet)’; PF-5 ‘insufficient time available from potential collaborators’; PF-11 ‘no rapport with collaborators’. They are particularly likely to occur in the emergency response context and given the time constraints, particularly important to detect.

Those offering visualization support must be sensitive to needs as they work to develop rapport, understand the problem domain, acquire data and establish whether a viable niche exists in which they can contribute: it may not. Equally they must be able to move quickly as opportunities can open and close as the situation develops. Things may change: again highly likely during rapid response. It’s important to be engaged and available, but not disruptive, to assess situations quickly and to know when to contribute and when to move on. Processes for visualization engagement and support need to be sensitive to these issues for visualization to be used successfully across a complex project involving parallel efforts such as the SCRC volunteering. As such, we do not identify individual examples in which visualization opportunities were not established here, but note that the reflection on constructs listed in table 1 offers important insights, particularly through the What Did Not Work? sections.

Visualization is experimental

Just as visualization itself may not be required, so specific instances of it may be ineffective, despite apparent opportunity for transfer and good intention. Existing approaches may not scale well to the problem or data in hand, or show the kinds of structures that are important. They may not be appropriate for computational reasons, perhaps as computational effort impedes the kind of interactivity required (§3a(vi)). An experimental perspective frames the visualization design process as an opportunity to learn about the applicability and scope of visualization techniques—under which conditions do particular idioms apply? It encourages us to use design experiments to determine whether plausible visualization solutions ‘fit’ across tasks and data sets, whether they scale and transfer to the new context and scope. In line with an iterative take on the visualization design processes (e.g. [89,91,95]) it encourages us to use design failures to develop more appropriate and perhaps more transferable solutions.

Some of our well-intentioned efforts to transfer visualization techniques were unsuccessful design experiments, such as the use of colour for location: [133]:596 and offset lines for flow positions [158]:1453. Known approaches may not scale well, such as Hadlack et al.’s Balanced Representation Strategy [120] for multiple-linked view layout [136]:884 or the use of animation to depict large numbers of changes [133]:596. Pre-processing may be required in some cases due to data volumes [137]:733. Specific structure in some datasets made proposed designs ineffective, such as extreme variations in population density [155]:791 or the sizes of the features revealed [160]:705, as did the size of some of the larger datasets. Some designs are sensitive to low level design choices involving alternative layouts and orderings [139]:438. Design proposals may be deemed unsuitable for all sorts of complex and sometimes quite subtle reasons [152]:675. Sometimes technical limitations, and lack of time to achieve computational efficiency, made speeds of computation and rendering inadequate for interactive response—a problem that is hard to fix quickly in the emergency context: ‘given the time we had, the performance is not optimized’ [136]:438, a sentiment shared across our reflections (e.g. [160]:709, [146]:425). An experimental framing can stimulate useful creative thinking to redesign and modify approaches to meet new needs and apply them to new datasets with particular scales, structures or other characteristics. Some of our insights (§3(a)) benefited from this approach.

Recording, reflecting and reporting [105] on visualization design experiments ‘in the field’ contributes beneficially to the discipline of visualization. This is particularly the case in broad projects that require rapid response and involve parallel activity. Computational notebooks (§2a) structured with prompts and strategies for design discourse have great potential in scaffolding this experimental process [106,107].

Visualization capability is collective

The volunteering effort, and any insights or learning achieved from it, have been highly dependent upon the tacit knowledge, skills and experience of a community of researchers. The SCRC volunteers had varying levels of complimentary and overlapping expertise in visualization, epidemiological modelling and visualization for epidemiological modelling. They were able to work together to rapidly share perspectives, and select and apply a range of technologies and techniques from a broad landscape of disciplines and experiences. The repertoire of potential solutions, and knowledge of how to combine, develop and apply them that enabled all of this, is of great value, as is the ability to combine, develop and apply people who have existing loose connections through an active international research community. Coming together at a time of crisis to assess cognitive and computational possibilities, and how particular approaches and technologies are likely to perform and combine in certain circumstances under unfamiliar conditions, demonstrates the importance of this knowledge and the strength of these connections.

The SCRC volunteering reveals that a community of capability exists in the UK, and that in an emergency situation it can be operationalized to rapidly and usefully apply its knowledge, skills and experience to a broad range of problems across a disparate project. With technologies that are mature, flexible, expressive and fast, and people who have the skills to use them quickly and effectively, this can be achieved directly and with immediate success [134]:415, [132]:726. Volunteers came from academia and industry and offered expertise in various compatible development technologies, including D3 [121], JavaScript [134]:841, Java [155]:941, Processing [122], Python [157]:676, Jupyter [133]:787 and Observable [112]. Immediate successes can be extended through longer term creative problem solving and iteration, efforts that can be informed by a theory-driven approach [142]. This is the case despite some instances across a project where visualization is not necessary or appropriate (§3(b)), and during an emergency in which individuals in a complex collaboration, and the connections between them, may be stretched, strained or broken.

Visualization processes evolve

Establishing where and why and how visualization might apply most beneficially in complex new contexts is key to its effective use and core to visualization research. Our collective experiences of COVID-19 visualization support through RAMP revealed plenty about the process of applying visualization knowledge in an emergency response. Close interactions between visualization researchers engaging in Immersive Design [89] with the modelling teams, our central support for analytical processes and our efforts to disseminate information visually through a centralized architecture all contributed to this learning. Just as design solutions apply and adapt known idioms, established design processes adapt and evolve when employed in different circumstances. We, therefore, offer some preliminary ideas about the process through which visualization design, visual analysis and visual problem solving may be conducted in collaborative rapid response projects.

We do so by identifying 10 characteristics of collaboration that resonated in our reflection across the efforts. These are somewhat interdependent, but characterize the kind of collaborative ecosystem of rapid experimental analysis and design that we found to be effective in supporting data analysts with visualization in emergency response. They were key to the successes reported in §3a and we suggest that they be adopted, assessed and adapted in future use of visualization in epidemiology and other applied settings. We consider them to be recommendations for visualization designers and researchers that complement existing guidelines from the perspective of emergency response.

Respond rapidly with analytical visual interventions

Conventional visualization support usually uses and encourages long-term collaboration [91-96] and results in system development (§1d). The end-goal is for data analysts to be provided with reliable systems that give them independent capability post-engagement. Emergency response requires a different approach. The focus of our modelling support teams was on getting viable reliable graphics of data made as quickly as possible [90], to understand data, demonstrate capability and assess needs. Importantly, visual analysts were on hand to develop, use and interpret the bespoke graphics that they injected into the analytical discussion (§3(a)(vi)), factoring in some flexibility (§3e(ii)) through their skills and expertise. Using tools and methods that are to hand to do so by rapidly generating and iterating on potentially useful graphics through analytical visual interventions is an important enabling first step [135]:757, [160]:728 [156]:688. It can result in rapid progress and the kind of mutual understanding of the problem, the data, each other and how visualization can help, that establishes vital capability, focus and trust.

A key aspect of this role is to scan possibilities quickly and Dig Deep (§3e(iv)) to search for high impact low cost solutions by using what is known and what is at hand. Notions and guidance from Lloyd & Dykes [90] and Hall et al. [89] and the kind of loose integration described by Matkovic et al. [123] apply well in this context [146]. Responding rapidly in this way may lead to design ideas and shared knowledge that can be captured and applied subsequently, through further analytical interventions or in usable visualization systems for observation and dissemination as well as analysis [98]. But the combination of a usable UI and a robust system is not an initial intention [138]:438, and this should not hinder progress. It likely requires additional time, resources, reflection and engagement [155]:788, [155]:931 and perhaps different underlying technology that may never become available or indeed be needed.

Factor in flexibility

The ability to rapidly vary data, its resolution, aggregation and visual depiction is key to rapid response. It enables design, analysis, shared knowledge, trust, opportunity and data generation to progress in parallel, as a collaboration is supported by effective visualization: ‘The whole system is very flexible and allows us to do lots of different comparisons and visualizations, which has been really important with such a complex dataset in understanding what is going on’ [155]:788.

Intermittent and unpredictable levels of communication among collaborators during rapid response mean that building flexibility into design is essential: ‘lack of continuous access to epidemiologists throughout the process meant we had to build general/customizable solutions that could be quickly configured and demonstrated’ [147]:681.

Flexible visualization designs get relevant graphics made rapidly. They are robust to (and indeed initiators of) changes in interest, needs, focus and data. As such, technologies and techniques that support them are highly beneficial [150], [134]:671 and processes that enable and encourage them are essential. The concept of the ‘data sketch’ [90]—a ‘loosely bounded collection of data, functionality and ideas’ that scavenges from existing applications and develops through the design process—is valuable here. Computational notebooks that document designs (§3e(vii)) and implement them for analysis offer plenty of scope for developing data sketches, capturing and sharing knowledge and factoring in flexibility. They promote discourse around design, analysis and reuse, and can be used for interactive design experiments [158]:825 [155]:918.

Analyse through design, design through analysis

It seems unhelpful to distinguish between analysis and design for analytical and model-developmental visualization in this context. The two are inextricably linked and seemed increasingly inseparable in our experiences. This symbiosis may be useful in other contexts.

Our rapid analysis, involving injecting visualization into workflows, establishes needs and tests candidate designs. For example, in our efforts to transform abstract data (§3(a)(v)) into two dimensions so that they could be seen and understood, we were collectively forced to prioritize certain aspects of the data for analytical (and graphical) focus. These priorities, in terms of tasks, aspects of data and analytical techniques, changed as the data structures were visually revealed [157]:689. This is a tight and effective design/analysis loop—designs drive analysis, just as analysis drives designs [156]:680. One way forward in this particular case is to fix a space into which data are projected (perhaps through transformation—§3(a)(v)), add data, add visual encodings, apply analytical techniques, interpret graphics and then creatively, analytically change view parameters. If we have Factored in Flexibility (§3(e)(ii)), we can redesign in response to data in real time. The result can be a revealing, iteratively developed, documented, symbiotic analytical design narrative [157]:689.

Dig deep

The rapidity of our response took initial advantage of the deep, often tacit, knowledge in the visualization community (§3(d)). Thinking back can be effective, and is perhaps quicker than searching sideways. As individuals, visualization volunteers predominantly cast a deep rather than wide net—taking ideas that they had used successfully in the past, many of which they had been involved in developing, as candidates for transfer, rather than those that were produced elsewhere.

For example, the architecture for generic support moved through Brodlie et al.’s five-level deployment model for visual computing [124,125] with great speed due to the ‘wonderful knowledge, skill, teamwork, and willingness to help combating COVID-19’ [148]:699 and the tried and tested nature of this useful theoretical construct.

This emergency alternative to the costly Learn phase of Design Study Methodology [95] proved viable and effective given our rapid response emphasis (§3e(i)). The possible disadvantages of experience bias are countered by close collaboration with data experts through design discourse and an Experimental Approach to visualization (§3c) in a Critical Creative Culture (§3e(vi)). Self awareness and reflexivity can further moderate [92].

Refine and combine to design (and analyse)

Refining and combining idioms is powerful, challenging and an essential part of any process of visual analysis and design [154]:981. It requires visualization expertise, technical skill, close collaboration and open communication. It is built on a Deep Dig (§3e(iv)) and helped (accelerated) by Factoring in Flexibility (§3e(ii)). It can result in effective solutions, designed to meet specific challenging needs.

Our collaborations were full of examples including, among others, our visual faceting work [147], approaches that represent error and uncertainty [138], linked parallel coordinates for visual parameter space exploration [135], our gridded bi-directional flow maps [158]:1401 and the development of an ontology-based visualization recommendation system [150]:746.

Create a critical creative culture

A culture of critical creative thinking is an essential factor in achieving much of this, particularly in an emergency and in delivering the speed and flexibility that we advocate (it is core to the ideas presented in §3e(i)–(v)). It underlies the Experimental Approach that we recommend (§3(c)). The visualization knowledge base offers excellent proven methods, tools, functionality and guidelines that can be accessed quickly with a Deep Dig (§3e(iv)). These can be used most effectively if applied and adapted creatively. But high levels of openness, trust and communication are required to develop effective specific solutions to complex problems at speed. The modelling support teams experienced this, feeling at times as though they were part of an ‘idea generation lab’ [157]:684.

In this kind of environment, constraints and lack of resources, or challenges such as a new scale of data or unexpected data structures, can drive creativity, innovation and collaboration (§3c). For example, our approach to propagating visual designs (see figure 4) is a creative solution that enabled us to rapidly produce thousands of viable online interactive visualizations and hundreds of dashboards from limited volunteer developer resources [149]:695. Equally, some inspired thinking enabled us to refine approaches and produce new solutions that may in turn be of use elsewhere. The population-proportional fading used to account for varying population densities in our Gridded Glyphmap [155]:788 is a good example (see figure 3), as is the improved ontology for mapping data types to visual idioms [150]:687. Equally, designing informative new spaces into which data are meaningfully depicted is an analytical/design process (§3e(iii)) that requires us to think deeply and creatively about complex analytical problems—both about what we want to know, and on the computational techniques and algorithms that we can deploy. Here we ask: ‘how can I compute new ‘data’ that is effective, robust, and immediately relevant to the analytical task at hand?’ [157]:689.

Specific methods can be applied to encourage creative thinking (e.g. [59,126]). The five design sheets approach [127] was useful in generating and refining ideas for disseminative visualization [154]. There is scope for further transfer and refinement of creativity methods for visualization as we establish a critical creative culture for design driven analysis and analysis driven design. Such activity can potentially reduce the reliance on, and bias inherent in, the rapid response Deep Dig (§3e(iv)).

Document design discourse

Open discourse around potential solutions that develop through this flexible creative approach to visualization and analysis can help move things forward as collaborators collectively learn about data, domains and designs. Documenting this thoroughly is essential, to engage collaborators who are short of time (§3e(ii)), and record reasons and reactions [106]. This may take time, but establishes a ‘helpful compromise’ [155]:788 that provides pertinent, interpretable visual information, in line with established good practice (e.g. S-2 ‘Document’ [89]) [156]:688.

The aims are various: to diminish knowledge boundaries between visualizers and domain specialists through knowledge sharing [156]:680—(flexible) visual artefacts from both sides of the knowledge divide can be useful boundary objects [128] to help share ideas and perspectives and to inform discourse that leads to design ideas and analysis [145]:910; to develop a fully documented, analytical design narrative that explains progress with examples and rationale that encourage scrutiny [156]:688 [92]; to establish a record that can be shared with new collaborators, across projects or as the basis for post-project reflection, learning, and provenance.

Synchronize and de-synchronize

Establishing a cadence and a commitment to communication is a particular challenge in uncertain and busy times. Understandably, those providing visualization support are sometimes working in the dark, and need to establish ways of getting efficient and effective feedback that informs and does not disrupt. A mix of regular short synchronous meetings (cadence) [135]:966 with a reasonable and negotiable attendance target (commitment) ‘quick, agile cycles’ [156]:680—with documented descriptions that provided scope for asynchronous feedback (with deadlines) offered an effective balance between flexibility and commitment in many cases. This was particularly so when supported by good documentation and searchable asynchronous digital communications streams (§3e(vii)). Regular logging [155]:811 and transparent language [156]:688 help capture progress and keep things moving when schedules, levels of commitment and priorities are dynamic and unpredictable.

Visualization volunteers found it beneficial to get ‘in sync’ with the modeller’s processes where possible [144]:723 but found it important to prepare for and accept down time, and make use of asynchronous methods too. Asynchronous activity works both ways and can involve: One modelling support team found it useful to characterize this as a ‘hurry up and wait’ strategy [140]:725.

direction—asking collaborators for resources through which knowledge can be built offline—papers, examples, datasets, project documentation, then

connection—using rapidly developed visualization as a common boundary object [128] to reconnect either synchronously at planned meetings [136]:877 or asynchronously through materials that document discussion around design and analysis (e.g. [106]).

There are open challenges around scaling this process and making it efficient—teams grow and shrink and team members' priorities and availability change rapidly in an emergency. But we found it a useful set-up in circumstances that differed from more regular visualization design engagements. It may transfer well to non-emergency contexts (see [106]), but is very dependent upon Documented Design Discourse (§3e(vii)) and Factoring in Flexibility (§3e(ii)).

Lurk to learn

Participating in video conference calls in listening-only mode, with permission, can be an excellent non-disruptive means of learning. This is a complex process as priorities and roles are dynamic. We found lurking and listening to work exceptionally well, have little cost and be particularly unobtrusive in the digital workspace. It proved useful for improving understanding, identifying roles for visualization and selecting candidate datasets and problems to which visualization design might be usefully applied: ‘Lurking in these meetings may provide a low-stress way to get ideas for visualization tasks or users, especially when the case for visualization is not clear to those involved.’ [140]:725.

Subsequently dropping candidate designs into discussions that address these problems can help communicate possibilities, stimulate discourse around data that establish roles and priorities, and hopefully contribute to data interpretation and analysis. This is a great role for lurking (friendly) ‘visualization spies’ [135]:966. Perhaps, the post-COVID-19-lockdown, rapid response visualization researcher/designer/analyst immerses [89] by lurking in digital workspaces and meetings? It was very effective in many cases in our project, enabling us to S-1 ‘Observe domain experts practices unobtrusively in situ’. This provided a vital and unexpected means of learning, assessing possibilities and rapidly and effectively conducting design and analysis experiments by injecting visualization into workflows where it might be beneficial [156]:680, [158]:1447.

Embrace digital immersion

Indeed, the pandemic, its various constraints on physical movement (lockdowns), and the associated rapid emergence, acceptance and shaping of screen sharing video meetings, with broad bandwidth and high resolution imagery, have resulted in a rich digital workspace for analysis and collaboration [159]:2529. This suits visualization and visual analysis well and many of these themes that characterize our experience of applying visualization processes to support epidemiological modelling are enabled by this technology. This includes activity that is paper-based [154]:821. It equips us to build on and improve existing approaches to developing engagement between visualization researchers and domain experts, as described variously above.

The objective to ‘Embed VIS expertise in the workflow’ [156]:688 seems an important first step and good precursor to the system building that is typical of applied visualization research. The digital workspace undoubtedly helps with this. But it also enables us to continue throughout a collaboration as we move from Rapid Response (§3e(i)) to something more permanent in ways that that may not involve the post-engagement isolation assumed by system building approaches that pre-date the digital workspace. How many visualization system user guides say ‘just call me on Zoom or ping me on Teams and we can work on this together through screenshare’? This frequently became our mode of providing visualization support in the digital collaboration of SCRC, much of which was undertaken during lockdown when people were unable to leave their homes. Perhaps a situation in which expert analysts work independently with stable software is less necessary or even appropriate in post-pandemic digital workspaces? Perhaps in this context the interdependence of concurrent Analyse through Design, Design through Analysis (§3e(iii)) requires some new thinking about the relationships between visual analyst, data analyst and technology and the roles that support them in the medium and long term. We wonder what happens when we combine Design by Immersion [89] with GitHub, notebooks, Zoom and collaboration tools like Slack, Figma ([134]:852) and Zulip ([145]:933). It may be informative and transformative to find out. Notions from other applied visualization design methodologies such as multi-dimensional in-depth long-term case studies [96], multi-channel visualization engagements [97] and action design research [91] may help us embrace the digital workspace in immersive collaborative analytical visualization design.

Visualization inspires research

Our cross-study analysis shows that the processes and roles and activities associated with applied visualization design are changing. Section 3e offers some insights. But it also reveals some open problems that were either reported in multiple notebooks or that seem relevant to multiple activities across the SCRC engagements. These findings demonstrate the important role of applied work in inspiring and stimulating visualization research. They should be addressed through a combination of applied visualization work in epidemiology and elsewhere, and controlled experimental approaches that complement this activity. In short, we need to establish, develop, assess and refine…

effective and efficient ways of visualizing differences in quantities that allow us to make spatial and temporal comparisons [155]:788 (see also [17], fig. 2 for a temporal view of a spatio-temporal dataset);

methods for comparing quantities and ratios that vary by orders of magnitude [159]:2380 (see also [82]) and that control for population size [155]:788;

effective ways of using layout and colour in combination in dense data graphics [139] (see also [17], fig. 4);

visualization idioms to deal with large numbers of data items and new structures in data that are unexpected or important [160]:705, [150]:687;

consistent visual languages—something that is hard to achieve in a pandemic (in parallel)—that allow us to use colour, icons, other encodings and even interactions in ways that are common, predictable, consistent, effective and understood [153];

narrative patterns for communicating in cases where subjects are sensitive or controversial [152], [153]:706;

approaches that minimize misinterpretation and account for it where it occurs [155]:788 by addressing some of the open issues listed above, and through effective documentation, signposting, training, learning and co-design processes;

effective ways of further embracing the emerging digital workspace for long-term immersive visualization support; and

reliable and effective processes for conducting and supporting research through applied visualization that draw upon the themes identified through this engagement between epidemiological modellers and visualization researchers (§3e).

These open issues demonstrate ways in which deep engagement in data rich problem domains infuses visualization research with potentially impactful opportunities and inspiring challenges.

Recommendations

These experiences and the claims, ideas and findings that they have enabled us to develop, lead us to some tentative recommendations. They address the visualization research community, the epidemiological modelling community, the wider scientific community and the bodies that support and sustain the activities of these groups. Just like the claims and themes that we document, and the parallel projects that have given rise to them, they are highly interdependent.

Reward and invest in reusable functionality

Research that involves working implementations of functionality, with full descriptions and usable code, is hugely valuable. It allows visualization knowledge to be rapidly and successfully applied to and tested in important new contexts, effectively and with efficiency. It is key to many of the characteristics of collaboration that we identify and should be fully supported. Well documented accessible libraries of functionality help with transfer, reliability, creativity and speed [157]:925, [147]:681. They enabled much of what we achieved, underpin many of our findings and are fundamental to much of our guidance.

We need to develop and maintain usable, documented, open libraries of VIS functionality. These are somewhat undervalued outputs in the academic environment and often regarded as an expensive add on to academic contributions. Where they work, they can be enormously effective and influential [121,129,130]. Without them, activity is slowed down by re-implementation [147]:720 and high impact interventions may not be possible [156]:684. They could be supported by efforts to improve reusability [134]:710 and would enable the kind of agile and flexible response advocated here [132]:722. We would like to see such core contributions funded, coordinated, appreciated and rewarded in anticipation of future projects that involve rapid response to unknown emergencies and the less urgent opportunities to adopt and adapt visualization methods to a variety of ongoing problems. The Observable notebooks presented here [104], and the RAMP VIS project outputs, are a small part of this process.

Value and use applied visualization design research

We found out plenty through our applied visualization experiments, and moved the body of knowledge forward somewhat. But we still lack ‘specific guidance about what visualization techniques and interactions will be most effective for which mid-level tasks.’ [135]:713.

Colleagues report, for example, that ‘The [visual faceting] idiom seems highly suited to COVID-19 related analyses due to the complexities of the data points themselves and their limited number. On the negative side the idiom is somewhat poorly explored in existing literature and there isn’t sufficient guidance and support for its deployment’ [147]:669. To make the most of these opportunities for VIS we need to ‘explore the use of the idiom in real analysis settings—how is it used, what analyses does it support, what are the scenarios that it maps to best?’ [147]:681.

We recommend a focus on applied VIS research that will advance our ability to map techniques to data and tasks, to build the visualization knowledge base through experience. Exploratory experimental visualization work in deeply applied contexts will draw attention to open problems in visualization and help us understand the potential of graphics in discovery beyond epidemiology and emergency response.

Document in detail to learn and share knowledge

Clear and comprehensive communication of applied visualization research, involving detailed descriptions of context, with reflective commentary and discussion, is important for knowledge sharing. It offers potential for transferring knowledge between contexts for application in rapid response and elsewhere. It also provides opportunity for cross-study meta-analysis that draws upon diverse experiences that we see so rarely in visualization. This is particularly true if claims and contexts are structured, searchable and findable. This paper and its supplementary digital notebooks offer an example of how this might be achieved.

Reporting in this way is also important within diverse projects such as the RAMP efforts to use visualization in epidemiological modelling. It enables those working in them to share ideas and experiences, supports asynchronous work, justifies decisions, makes alternatives explicit and may contribute to the validity of the work undertaken [92]. It is fundamental to many of the processes of analysis and design that we recommend. Our efforts to do so enabled us to find common activities such as similar task requirements that occur across different modelling workflows [140], share and broaden solutions (e.g. developing consistent colour schemes and shared analytical functionality across the project), and identify cases that required a re-think (e.g. the various compartmental models that required comparable visualtreatment).

Producing informative documentation is time consuming but rewarding. The unpredictable and variable cadence of the emergency effort made it both crucial and (with planning and flexibility) feasible during our volunteering. While our emergency response work was something of a unique opportunity for this kind of activity, the possibilities are broader: for communication between groups, individuals, topics, funding rounds, etc. The current culture of reward structures and systems does not particularly encourage this kind of detailed documentation, but the notebook paradigm (see also [107]) and our adoption of it in the visualization context have provided a means for sharing across the group, prompting reflection and establishing themes with some success. This model could be applied elsewhere for communication and analysis within and between applied visualization projects.

Sustain and strengthen the visualization community

The volunteering activity constitutes valuable progress in terms of the connections, cooperation, capability, knowledge and goodwill that has resulted within and between the epidemiological modelling and visualization communities. In many ways, what we have done has also shaped the visualization community well for future support efforts in the UK. This may involve wider scientific endeavour, future emergency response and the very prescient challenges facing society that involve large, complex, rich datasets. Understandably, we would like to push forwards and build upon what has been achieved, rather than lose it.

We have established important skills, talents, software, infrastructure, teams, connections and ways of working while developing knowledge and capacity. Only some of these are captured in documents such as this one. Much of this rich ecosystem of capability involves tacit knowledge that must be shared, sustained and strengthened through investment in people: ‘We should not let this effort, the resultant software, and perhaps most importantly the accumulated R&D knowledge go to waste …The underlying infrastructure that was generated by the RAMP VIS team should be consolidated and made available for future use.’[148]:705.

Given this, it seems important and wise to maintain, support, develop and apply the body of knowledge, collective capability and expertise that makes this possible. This may involve supporting researchers to work in new areas and to develop their knowledge and skillsets (e.g. UKRI, EP/V033670/1, Visualizing Contact Networks in Response to COVID-19) and increasing Research Software Engineer resources for transforming designs and prototypes to the VIS infrastructure (e.g. UKRI EP/V054236/1, RAMP VIS). But this might be best supported by more substantial coordinated strategic investment.

Adopt visualization design for analysis

We found rapid iterative design experiments to work well in capturing analytical requirements, showcasing possibilities, learning visual languages and prioritizing activity as we apply VIS techniques to complex data challenges in new configurations. They also contribute to the team building, knowledge sharing and development of common understanding that underpin such efforts, through actual analysis: ‘The requirement and design work seem to become part of the visualization creation’ [160]. We should not be too surprised by this (see [90]). But the roles of visualization design in analytical steering and trust building are not adequately acknowledged in the dominant software system development process models that describe applied visualization research [95]. Equally, the wider scientific community may not be fully aware of the potential for VIS in all stages of the scientific process: in model building and analysis as well as for observation and dissemination. VIS is a human analytical process rather than a technology.

To address this, visualization researchers should embrace the perspective of the rapid Visualization Design Experiment (§3(c)) to be less focused on system development. This will enable a sharper focus on the inseparable relationship of discovery between analysis and design (§3e(i) and §3e(iii)) that enables us to assess and refine opportunities for VIS and develop domain knowledge iteratively and experimentally (§3c). It may help us use visualization more widely in the analytical stages of scientific enquiry. Developing a creative culture (§3e(vi)) that allows us to build knowledge through applied analytical design work is key to this.

Conclusion

This paper reports on a complex project, involving parallel activity across a broad range of efforts to provide visualization support for epidemiological modelling. It presents a diverse body of electronic supplementary materials that document this work and capture the iterative structured thinking that has taken place around it [104]. It draws widely upon this data source to propose findings and develop recommendations. The findings are preliminary and partial, but draw upon the knowledge, experience, collaborative capabilities and thinking of visualization researchers, epidemiological modellers, software engineers and other volunteers. The intention is to ‘develop knowledge about visualization in epidemiology that can be applied to other contexts while providing examples that demonstrate how this might be achieved so that visualization can be used effectively and widely in data-driven research’.

We find plenty of opportunity for effective visualization to support epidemiological modelling at all four levels as we observe, analyse, model and disseminate data and its derivatives. We also find calls for more interactive VIS tools for exploration and provenance elsewhere in this special issue [76]. Visualization is not guaranteed to be effective, but applying visualization knowledge and adapting visual analytic approaches in a creative experimental environment of collaborative data analysis and design can be informative. Approaches that involve design by immersion [89] help in the early stages of intense collaboration, where the focus is more on analysis and knowledge sharing than system design. Subsequent software systems may provide proven persistent functionality and benefit from these early interactions, but extending the initial emphasis on knowledge sharing and trust building achieved through rapid analysis and insight marks a shift in current visualization design study thinking [95]. This has been jolted by our emergency context, accelerated by the digital workspace and informed by our visceral volunteering experiences. Changes to visualization support processes that account for this refocus, such as those detailed in §3(e) are helpful. They include a greater emphasis on the deep symbiosis between design and analysis, and the value of early activities in sharing knowledge, understanding data, refining tasks and establishing needs. The recommendations made in §4 build upon these findings in an effort to maintain momentum, make the most of the knowledge achieved and support the VIS and wider scientific communities in developing and applying visualization knowledge. Acting upon them is highly likely to deliver many more successes, such as those listed in §3a.

Importantly, the rich series of interactive reflective notebooks that we offer to supplement this paper acts as a considerable evidence base to support these claims. This ‘data’ documents selected examples of visualization knowledge and comments on its use in the RAMP activity. Providing it enables us to refine, share and discuss our experiences and interpretations of them as knowledge settles and we continue to learn through iteration, reflection and critique. We hope that the collection of interactive notebooks supports our claims well, and in ways that allow others to scrutinize, evaluate, question and build upon them. We see this form of rich documentation as important in the processes of recording, reflecting upon and reporting applied visualization research. The notebooks demonstrate how visualization knowledge was transferred to, shaped in and developed by the particular problem in hand. In some cases they provide working functionality that can be reused and applied elsewhere. We hope that they show how VIS might be applied and developed in other contexts across science—directly with examples, and indirectly through rich description of context and subsequent inference.