Mapping the co-evolution of artificial intelligence, robotics, and the internet of things over 20 years (1998-2017).

Understanding the emergence, co-evolution, and convergence of science and technology (S&T) areas offers competitive intelligence for researchers, managers, policy makers, and others. This paper presents new funding, publication, and scholarly network metrics and visualizations that were validated via expert surveys. The metrics and visualizations exemplify the emergence and convergence of three areas of strategic interest: artificial intelligence (AI), robotics, and internet of things (IoT) over the last 20 years (1998-2017). For 32,716 publications and 4,497 NSF awards, we identify their topical coverage (using the UCSD map of science), evolving co-author networks, and increasing convergence. The results support data-driven decision making when setting proper research and development (R&D) priorities; developing future S&T investment strategies; or performing effective research program assessment.

Introduction and prior work

Advances in computational power, combined with the unprecedented volume and variety of data on science and technology (S&T) developments, create ideal conditions for the development and application of data mining and visualization approaches that reveal the structure and dynamics of research progress. A critical challenge for decision makers is determining how to spend limited resources most productively. To do so, one must have a basic understanding of where the most productive research is being done, who the key experts are, and how others are investing in research. The identification of topics that have recently emerged as increasingly important or that are converging to create new synergies in research can be particularly fertile areas for research and development (R&D).

Different ‘emergence’ and ‘convergence’ definitions, indicators, and metrics have been proposed in previous work in this area. We use the four-attribute model of what constitutes technological emergence [1] and assume that emergent topics should evidence term novelty, persistence, and accelerating growth, and typically show the formation of a research community. Much prior work exists on how to measure emergence. Guo et al. [2] proposed a mixed model that combines different indicators to describe and predict key structural and dynamic features of emerging research areas. Three indicators are combined: 1) sudden increases in the frequency of specific words; 2) the number and speed with which new authors are attracted to an emerging research area; and 3) changes in the interdisciplinarity of references cited. Applying this mixed model to four emerging areas for means of validation results in interesting temporal correlations. First, new authors enter the research area, then paper references become interdisciplinary, and then word bursts occur.

Recent work—including that funded by the US Intelligence Advanced Research Projects Activity (IARPA) Program on Foresight and Understanding from Scientific Exposition (FUSE) [3]—focuses on advanced linguistic techniques for identifying emerging research topics. Contributions from [4] include methods to extract terms from paper titles and abstracts and filter them based on 1) novelty, 2) persistence, 3) a research community, and 4) rapid growth in research activity. Porter et al. [5] developed emergence indicators that help 1) identify “hot topic” terms, 2) generate secondary indicators that reflect especially active frontiers in a target R&D domain, 3) flag papers or patents rich in emergent technology content, and 4) score research fields on relative degree of emergence. Other recent studies introduced several novel NLP methods to measure research diversity and interdisciplinarity. For example, topic modeling has been used to measure the degree of topic diversity [6], Shannon’s entropy measure was applied to compute technological diversity using EU-funded nanotechnology projects data [7]. This paper uses robust and widely used topic identification methods and focuses on visualizing the emergence, co-evolution, and convergence of science and technology areas.

Convergence research was identified by the National Science Foundation (NSF) as one of the 10 Big Ideas for Future NSF Investments [8] that will help advance US prosperity, security, health, and well-being. In this paper, we present a repeatable procedure to characterize emerging R&D topics based on publication and funding data. Using this procedure, we visualize and analyze the convergence of three emerging R&D areas. Our efforts here both build upon and expand work by [9], who used the convergence to study scholarly networks for domains relevant for understanding the human dimensions of global environmental change.

A literature review and stakeholder-needs analysis were used to identify three domains of strategic interest: artificial intelligence (AI), robotics, and the internet of things (IoT). These three areas are of paramount importance not only for global prosperity, but also for defense and security [10]. AI, IoT, and robotics were named top technologies in 2018 with strong arguments and examples of how these technologies will drive digital innovation and completely transform business models [11, 12]. Since AI and robotics will have a major impact towards the future of economy, businesses need advanced preparation to meet these transformational challenges. The 2019 AI annual report pointed to the complexity of the fast-growing AI labor market: “unconditional convergence” and “unconditional divergence” in job demands at the same time [13]. In line with these developments, the White House prioritized funding for fundamental AI research and computing infrastructure, machine learning, and autonomous systems. Also, it argued for the need to work with international allies to recognize the potential benefits of AI and to promote AI R&D [14].

The paper is organized as follows. The following section details the stakeholder needs analysis, which guided the selection of strategic research areas and provided information about stakeholder insight needs. After that, we indicate both the datasets used and the data preprocessing needed for the study. This is followed by an outline of the methods used in the study and the results achieved. After examining validation design and insights in the user study, we end the paper with a discussion of the results and outlook for future research in this area.

Stakeholder needs analysis

A stakeholder needs analysis (SNA) was employed in order to identify insight needs and use cases for using indicators and visualizations of emergence (and indirectly convergence) in daily decision-making environments. This study was approved by the Indiana University Institutional Review Board (protocol number 1807496060). Specifically, the SNA was designed to identify stakeholder demographics, task types, insight needs, work contexts, and priorities to better understand how decision makers might utilize static and dynamic information visualizations, topics of concern, and metrics currently used when making decisions. Twelve decision makers from the Naval Surface Warfare Center, Crane Division, (NSWC Crane) in Martin County, Indiana completed the one-hour survey. Participants included personnel from human resources, corporate operations, engagement, R&D, and various technical specializations. Surveys were conducted both on the Indiana University–Bloomington campus and at WestGate Academy, a technology park located adjacent to the naval base.

Areas of strategic interest

In order to understand what topical areas were of interest, survey respondents were asked to identify which topics, from a list of eight, were most relevant for their work. Additional topics were also solicited from survey participants. The top eight topics are represented in Table 1. Each of the eight topics was queried via NSF award and Web of Science portals to identify the total of funding award and publications between 1998 and 2017.

Table 1

Topical interest and relevant funding and publication data.

Topic	#Experts that expressed interest	#NSF Funding awards active in 1998-2017	#WOS Publications published in 1998-2017
Advanced electronics	10	122	206
Artificial intelligence	10	1,075	7,414
Sensors and sensor fusion	9	145	14
Internet of things	4	348	11,371
Human systems integration	4	0	102
Robotics	3	3,074	13,931
System of systems test and evaluation	3	4	0
Power and energy management	2	0	48

Users study report is available at https://github.com/cns-iu/AiCoEvolution.

Since we used funding and publication data to characterize emerging areas, the number of NSF awards and Web of Science (WOS) publications for each of the top-ranked areas was a factor in final topic selection (see Table 1). Note that for several areas, few funding awards have been made, allowing a human analyst to read through the related abstracts within a week. Artificial intelligence, the internet of things, and robotics are three domains identified as being of national strategic importance that have sufficient numbers of funding awards and publications for rigorous analysis.

Current metrics informing resource allocation decisions at NSWC Crane focus on internal R&D needs, recent calls for funding, and how other federal agencies are focusing their funding. Most respondents identified similar processes for allocating funding, deciding when to bring in outside expertise, and selecting contractors. This unified decision-making approach suggests that visualizations providing greater detail on how others are focusing their funding in strategic areas of interest have an important role to play in the process.

In the second part of the SNA, participants were asked to view three sample visualizations, identify any insights gained from them, and elaborate on how they might use these insights. At the end, participants were asked to identify which of the three figures they found most useful and why. Seven respondents found the network visualization most useful, two found the tree map that identified top funders most useful, and one preferred a visualization showing the number of citations related to one topic over time. One participant responded that all were equally useful in different ways but were each not as useful on their own.

Participants were also presented with three interactive visualizations that they were able to manipulate on laptop computers or tablets. Again, participants were asked to identify insights gained from each visualization, and to elaborate on how they might use these insights. They were also asked to identify which of the three figures they found most useful and why. Six respondents found the co-authorship geospatial visualization most useful, three found the co-authorship network most useful, and two identified the science map as most useful. In sum, co-authorship networks in combination with geographic representations proved most interesting to the stakeholders surveyed. Additionally, stakeholder interest in interactive capabilities suggests that this would be a valuable direction for future efforts.

Data and processing

A majority of publications related to each of the three target domains is captured in the Web of Science (WOS). In the US, much of the funding for these three focus areas is awarded by the National Science Foundation (NSF). The authors acknowledge that some subset of the current research in these areas is conducted through defense organizations and therefore can be difficult to capture in publicly available datasets. Only WOS publications and NSF awards from the last twenty years (1998-2017) were included in the analysis.

Publications

Publication data was retrieved from the Clarivate Analytics Web of Science (WOS Core Collection) web portal and WOS XML raw data (Web of Knowledge version 5) acquired from Clarivate Analytics by the IUNI Science of Science Hub and shared through a Data Custodian user agreement with the Cyberinfrastructure for Network Science Center (CNS) on July 7, 2018. The total number of WOS publications is 69 million and there are more than one billion citation links from 1900 through the early months of 2018. Most publications have title, abstract, and keyword information that can be used for text mining. Publications also have a publication year and author data. Using the AuthorKeywords field, Clarivate Analytics extracted publication identifiers (accession numbers—UT) on October 24, 2019 containing the (compound) query terms “artificial intelligence,” “internet of things,” “IoT,” and “robotics.” We also manually evaluated the 371 keywords where the “IoT” term was semantically and structurally ambiguous (e.g., “interocular transfer (iot),” “antibiotic activity”), resulting in 64 false positives that were removed from the records. Clarivate UT identifiers were then split in eight segments and queried in the WOS web portal on January 24, 2019 to extract the ISI raw data that was then filtered by three keywords. Furthermore, two records have been removed in which the publication year was changed from 2017 to 2018: WOS:000425355100004 (artificial intelligence) and WOS:000425355100017 (IoT). The final number of records can be seen in Table 1. Two of the 7,414 papers (artificial intelligence) had no authors (WOS:000384456000001, WOS:000173337900025) and were excluded from the co-author and geospatial analyses. The number of publications per year and the number of citations for 1998-2017 are shown in Fig 1A. Note that the number of all publications (dashed line) are steadily increasing during the years 1998-2017, and between 2007-2011 we observe a nascent field of IoT (blue solid line) showing a sharp increase in the publication counts, exceeding AI publications (yellow solid line) by 2012 and robotics publications (red solid line) by 2014. The number of citations for all three domains (dotted line) shows a slow increase with a drop by the years 2015-2016. Note that papers published in recent years did not yet have time to acquire a high citation count.

Fig 1

WOS publications and NSF awards.

(A) Number of WOS papers extracted and the number of citations. Yellow solid line represents AI publications; red solid line, robotics publications; and blue solid line, IoT publications. The dotted line represents the total number of citations (total for all three domains). (B) Number of NSF grants and amount of funding awarded by NSF each year. Yellow solid line represents AI awards; red solid line, robotics awards; and blue solid line, IoT awards. The dotted line represents the total amount of funding (total for all three domains).

There are many instances where publication records contain keywords from more than one focus area. For the terms “AI” and “robotics,” for instance, there are 209 overlapping publication records. Between “AI” and “IoT” there are 46 overlapping publications, and between “IoT” and “robotics” there are 38. There were two publications containing keywords from all three areas (“A Novel Method for Implementing Artificial Intelligence, Cloud and Internet of Things in Robots” and “IT as a Driver for New Business”). Over time, there was a statistically significant increase in publications (p < 0.0001), totaling 32,716 during the period of 1998-2017 (see Fig 1A). Table 2 displays the number of unique keywords and authors for the three areas together with the totals.

Table 2

Topical interest and relevant funding and publication data.

Topic	#NSF Unique Investigators	#NSF Unique Keywords	#WOS Unique Authors	#WOS Unique Keywords
Artificial Intelligence	1,297	3,081	17,316	17,534
Internet of Things	575	2,435	23,691	21,204
Robotics	3,275	6,144	30,784	23,561

Funding

The NSF funds research and education in science and engineering through grants, contracts, and cooperative agreements to more than 2,000 colleges, universities, and institutions across the United States. It provides about 20 percent of the federal support academic institutions receive for basic research. More than 500,000 awards—including active, expired, and historical awards from 1952 to today—are available via the NSF award search portal.

NSF funding data for awards containing the (compound) terms “artificial intelligence,” “internet of things,” “IoT,” and “robotics” was downloaded on July 24, 2018. Subsequently, the resulting sets were narrowed to NSF awards that were active in the last 20 years (Jan 1, 1998 to Dec 31, 2017): 1,075 in AI, 3,074 in robotics, and 348 in IoT (see Fig 1B and also Table 1). Note that 325 active awards have their start date earlier than 1998. Table 1 exhibits the number of NSF funding awards active in 1998-2017, and Table 2 illustrates the number of unique investigators and unique keywords for the three areas together with the totals.

There are some instances where awards overlapped: AI and robotics, for instance, received 146 funding awards, robotics and IoT received 17, while AI and IoT received only 2. There was no award for a project involving all three focus areas. The annual distribution demonstrates a statistically significant increase in awards (p < 0.0001), which seems to follow the same pattern as publications, lagging only by 10-fold on the log scale and totaling 4,497 awards over time (see Fig 1B). Note that the total award amount for the 1,075 AI funding awards is $494,310,951, the 3,074 robotics awards is $1,375,299,908, and the 348 IoT awards is $149,498,845. The number of awards and the amount of funding demonstrates a slow increase over time, with the exception of IoT showing a spike between the years 2013 and 2015.

Keyword extraction via MaxMatch

Using results from a linguistic algorithm comparison detailed in [15], the MaxMatch algorithm was used to identify terms in NSF funding awards that match the unique WOS Author Keywords specific to the three topic areas. The MaxMatch algorithm [16] performs word segmentation to improve precision. The algorithm first computes the maximum number of words in the lexical resource (here NSF award titles and abstracts); then it matches long terms before matching shorter terms. Thus, given the text “artificial intelligence” and “intelligence” in a set of relevant terms, and “artificial intelligence” in the title and/or abstract of an award, the algorithm returns “artificial intelligence.” This reduces oversampling of popular, short terms.

All keywords were pre-processed and normalized via Key Collision Fingerprint and ngram methods using OpenRefine [17]. This algorithm finds “alternative representations of the same things,” thus allowing for the normalization of keywords (e.g., “internet of things (iot),” “iot—internet of things,” “internet of thing”). We identified 1,739 clusters for AI; 2,333 clusters for IoT, and 3,201 clusters for robotics, which we then normalized by merging similar terms.

As a result, for WOS publications, we identified 55,946 unique Author keywords: 17,534 unique keywords for AI; 21,204 for IoT, and 23,561 for robotics. There are 2,935 terms that are shared between AI and robotics; 2,204 between AI and IoT; 2,331 between IoT and robotics; 1,117 shared across all three sets (∼2% of all identified WOS keywords).

For NSF awards, we first excluded 325 active records where the start date was earlier than 1998. We then identified 9,185 unique Author keywords terms: 3,081 unique terms for AI; 2,435 for IoT, and 6,144 for robotics. Note that the keywords intersect: There are 1,376 terms that are shared between AI and robotics; 717 between AI and IoT; 914 between IoT and robotics, and 532 are shared between all three terms (∼6% of all identified NSF keywords).

Methods

The research was conducted under IRB protocol 1807496060 and protocol 1809442778.

Burst detection and visualization

A Burst detection algorithm helps identify sudden increases in how often certain keywords are used in temporal data streams [18]. The Kleinberg’s algorithm, available in the Sci2 Tool [19], reads a stream of events (e.g., time-stamped text from titles and abstracts) as input. It outputs a table with burst beginning and end dates and a burst weight, indicating the level of ‘burstiness.’ Burst weights can be used to set thresholds (for example, to keep only the top-10 words with the highest burst weights). For this study, burst was run with gamma at 1.0, density scaling at 2.0, and bursting states and burst length of 1. The weights of terms that burst multiple times were summed up before top-n were picked for visual graphing. Note, however, that the original burst values are used to code the bars by area.

A novel burst visualization was implemented to show the temporal sequence of bursts in funding and publication data as well as co-bursts in both types of data. Using the temporal bar graph visualization as a starting point, each bursting term is represented as a horizontal bar, where the length represents burst duration (defined by a specific start and end date), height (thickness) shows burst strength, and color denotes data type (e.g., funding, publications). Fig 2 explains this new burst visualization using a hypothetical example of six compound terms (T) that burst between 2004-2018. The area of each bar encodes the burst weight equally distributed over all years in which the burst occurs. Note that some burst terms are consecutive—they end in year x and start in year x+1—and might have different burst weights. Bars are color-coded by type with blue indicating bursts in funding, orange bursts in publications, and gray denoting simultaneous bursts in both types of data. While two plots side-by-side would make each easier to examine, this combined visualization makes it possible to compare bursts (start and end dates, burst weights, co-bursts) across datasets.

Fig 2

Burst visualization using horizontal bar graphs.

Top organizations and funding

The Web of Science (WOS) online portal [20] supports searches for specific topics and facilitates the retrieval and examination of top funding agencies and top research organizations. Between September 11 and October 23, 2018, queries were run on the terms “artificial intelligence,” “internet of things,” “IoT,” and “robotics” using topic and title fields for the years 1998-2017. The ‘Organization-Enhanced’ field was used which returns all records including name variants and preferred organization names. Organizations and funding agencies come with a variety of spellings. For example, the NSF and National Science Foundation show up as two different entities making name unification necessary. For each of the three focus areas, we selected the top-10 research institutions and funding agencies, identified their country, and tabulated results (see S1 Table in S1 Appendix).

Co-author networks

The Sci2 Tool was used to extract a co-author network using the co-author column. The resulting undirected, weighted network has one node type: authors. Each node represents a unique author. Edges represent co-occurrence of authors on a paper (i.e., the relationship between pairs of authors to be co-authors or not). Edge weights denote the number of times two authors co-occur on (i.e., co-author) a paper. The example given in Fig 3 illustrates a table with five papers by a total of six authors. Authors A2 and A6, for example, co-occur on papers P2 and P6, so their link is twice as thick. All other edges have a weight of one. For this study, we used perfect match on names in the Authors field using the ‘Extract Co-Occurrence Network’ functionality.

Fig 3

Co-authors listed on papers (left) are rendered as a co-author network (right).

Force directed network layout takes the dimensions of the available display area and a network as input. Using information on the similarity relationships among nodes—for example, co-author link weights—it calculates node positions so that similar nodes are in close spatial proximity. Layout calculations are computationally expensive as all node pairs have to be examined and layout optimization is performed iteratively. Running the Generalized Expectation Maximization (GEM) layout available in Sci2 via GUESS on a simple network of six authors that published five papers (Fig 3, left) results in the layout shown on the right in Fig 3. The network has six co-author nodes that are fully connected. Author nodes are size-coded by the number of co-authors and color-coded by the year of the first publication. Edges denote co-authorship relations and are color coded by year of the first joint publication and thickness coded by the number of joint papers. The legend communicates the mapping of data variables to graphic variables.

WOS publications provide affiliation data (addresses) for authors, making it feasible to generate network overlays on geospatial maps (see Fig 4). We use Make-a-Vis [21, 22] to generate latitude and longitude values for author addresses as well as co-author networks; authors with no US address are excluded from this network. If an author has multiple addresses, the most recent address is used. The co-author network with geospatial node positions is saved out and read into Gephi [23] for visualization using Mercator projection; with node area size indicating number of citations, node color indicating the first year published, and edge thickness representing the number of times two authors are listed on a paper together.

Fig 4

Co-author network overlay on U.S. geospatial map.

See GitHub Appendix for more information on the co-author and geospatial network workflow https://github.com/cns-iu/AICoEvolution. Note: US-Atlas is under an ISC license. Copyright 2013-2019 by Michael Bostock.

Science map and classification system

The UCSD Map of Science and Classification System was created using 2006–2008 data from Scopus and 2005–2010 data from the Web of Science [24]. The map organizes more than 25,000 journals/conference venues into 554 (sub)disciplines that are further aggregated into thirteen main scientific disciplines (e.g., physics, biology), which are labeled and color-coded in the map. For example, the ‘Math & Physics’ discipline in the top left has a purple label and all subdiscipline circles are rendered in purple. The network of 554 (sub)disciplines and their major similarity linkages was laid out on the surface of a sphere and flattened using a Mercator projection, resulting in a two-dimensional map (Fig 5). The UCSD Map of Science wraps around horizontally (i.e., the right side connects to the left side of the map).

Fig 5

A process of mapping scientific journal names into a discipline and (sub)discipline topic (left) and the projection of journal topics into 2D spatial position (right).

In order to create proportional symbol data overlays, a new dataset is “science-coded” using the journals (or keywords) associated with each of the 554 (sub)disciplines. For example, a paper published in the journal Pharmacogenomics has the science-location ‘Molecular Medicine,’ as the journal is associated with this (sub)discipline of the discipline ‘Health Professionals.’ If non-journal data (e.g., patents, grants, or job advertisements) need to be science-located, then the keywords associated with each (sub)discipline can be used to identify the science location for each record based on textual similarity. In this study, multidisciplinary journals such as Science or Nature, which are fractionally assigned to multiple disciplines, were associated with a ‘Multidisciplinary’ discipline. Journals that cannot be classified are automatically associated with an ‘Unclassified’ discipline. Bar graph visualizations showing the number of papers and citations for the 15 disciplines are used to support comparisons (see S1-S3 Figs in S1 Appendix).

Results

The previous sections gave a general introduction to this research, prior work, stakeholder needs, available datasets, and methods used. This section represents results for each of the three areas separately, compares those results, and analyzes, visualizes, and discusses convergence of the three fields.

Artificial intelligence (AI)

The field of artificial intelligence studies the interplay of computation and cognition. It is concerned with subjects such as knowledge representation and retrieval, decision-making, natural language processing, and human and animal cognition. AI research generates tools and artifacts to address problems involving complex computational models, real-world uncertainty, computational intractability, and large volumes of data. It also uses computational methods to better understand the foundations of natural intelligence and social behavior.

Top-funded AI awards include BEACON: An NSF Center for the Study of Evolution in Action led by Erik Goodman at Michigan State University, active 2010-2021, total amount awarded to date $43M; the Center for Research in Cognitive Science led by Aravind Joshi at the University of Pennsylvania, 1991-2002, $21M; and the Spatial Intelligence and Learning Center (SILC) led by Nora Newcombe at Temple University, 2011-2018, $18M.

WOS-top organizations and funding

The top-10 AI-funding organizations most frequently acknowledged in papers (seven were merged, see S6 Table in S1 Appendix) and the top-10 research organizations are exhibited in Fig 6 (top right). The leading top funders are the Natural Science Foundation of China and the National Science Foundation in the U.S., followed by agencies from the UK, Mexico, Europe and Brazil. The top research organizations are the French National Center of Research and the French University of Cote-d’Azure. U.S., India, China, and United Arab Emirates are among countries in the top-10 leading AI research institutions. The list of abbreviations for agencies and institutions and their name variations is exhibited in S6 Table in S1 Appendix.

Fig 6

Bursts of activity and top funding organizations.

Shown on the left are the top-15 keywords with the strongest bursts in funding awards (blue color) and publications (orange color). Gray color indicates a double burst. For example, “Machine Learning” is bursting in both publications and funding awards in 2017-2018. Bar thickness indicates the strength of each burst (weight). Given on the right are the Top-10 funders and research organizations associated with publications (see S5 Table in S1 Appendix).

Burst of activity

Within the 1,075 NSF awards that have the keyword “artificial intelligence,” there are 161 bursts. There are no terms that burst twice. As for the 7,414 WOS publications with the keyword ‘artificial intelligence,’ there are 89 bursts total with no terms bursting twice. There are eight overlapping bursts for NSF and WOS keywords: “Agents,” “Big Data,” “Component,” “Control,” “Deep Learning,” “Expert Systems,” “Machine Learning,” and “Psychology.” The top bursting activity for NSF is “Machine Learning” with the burst weight of 13.04, while for WOS it is “Learning (Artificial Intelligence)” with the burst weight of 39.29. Among the top-10 bursting activities, “Big Data” co-bursts in both sets in 2014-2017 and is rendered in gray (see Fig 6, top left). The other two co-bursting terms are “Deep Learning” and “Machine Learning.” Burst weight is indicated by bar thickness with “Learning (Artificial Intelligence)” having the strongest burst in 2015-2017. Interestingly, the bursting activity between 1998 and 2007 is predominately present in WOS publications (orange color). Starting with the keyword “Web,” NSF awards exhibit bursting activities, culminating by co-bursting with publication from 2014 in “Big Data,” “Machine Learning” and “Deep Learning.”

Key authors and collaboration networks

The complete co-author network for AI has 17,316 unique author nodes. There are 437 authors with more than three papers, 235 authors with more than four papers, 143 with more than five papers. Of these nodes, 901 are not connected to any other node (they are called isolates) denoting that these 901 authors have not co-authored with any others during the 20 years. There are 31,476 co-author edges. The average degree is 3.64. The network has 4,299 weakly connected components, including the 901 isolates. The largest connected component consists of 1,914 nodes and is too large to visualize in a paper. The network was filtered by times-cited ≥ 1 resulting in 11,166 nodes, 21,280 edges, 2,763 weakly connected components and 473 isolates. The largest connected component of this network has 675 co-authors (with 473 isolates removed) and is shown in Fig 7 (top left). Author nodes and node labels are size-coded by the number of citations. Links, which denote co-authorship relations, are thickness-coded by the number of joint publications. The total number of links is 2,028 and they are filtered by the number of co-authored network (≥1). The labels are filtered by the number of times-cited (≥100). Author ‘Zhang, Y’ has the largest number of citations (475) in this network. Zhang is also one of the top-10 cited US authors in the complete co-author network (see Fig 7 top table).

Fig 7

Co-author network and top-10 authors by #Citations for AI (top), robotics (middle), IoT (bottom).

When presented with key author and collaboration networks during the stakeholder needs analysis, users indicated that network diagrams overlaid on top of geographic maps provided greater insights than network diagrams not anchored to geographic space. In response to that feedback, we present here key authorship and collaboration network diagrams that use the United States as a base map.

Co-author U.S. map

To overlay the obtained co-authored network over a U.S. map, we used Make-a-Vis [21]. Fig 8 (top left) shows the number of co-authors for AI with nodes representing the number of citations and a darker hue showing the first year of publication for a given author. The network concentration is noticeable in the eastern states, and Austin and Pittsburgh are the top-two cities in the mid-US, with AI research cited 910 and 888 times, respectively (see Fig 8, top right).

Fig 8

Co-author network extracted from WOS publications and overlaid on the U.S. map.

Note: US-Atlas is under an ISC license. Copyright 2013-2019 by Michael Bostock.

Topical evolution

The topical distribution of 7,414 WOS publications on AI is shown for two 10-year time slices in Fig 9 (top) using Make-a-Vis. The UCSD Map of Science and Classification system used with identical circle area size coding (see discussion in the Methods section). Most of the papers are in the ‘Electrical Engineering & Computer Science’ disciplines and in the ‘Chemical, Mechanical, and Civil Engineering’ disciplines. Note the increase of papers in the ‘Social Sciences’ and ‘Health Professionals’ (see also S1 Fig in S1 Appendix). The top-five most cited papers, along with their publication year and total number of citations, are shown in S4 Table in S1 Appendix.

Fig 9

Topical coverage of AI, robotics, and IoT publications published in 1998-2007 (top) and 2008-2017 (bottom).

Fig 9 illustrates the evolution of topical coverage for each of the key terms. The comparison between left (1998-2007) and right (2008-2017) panels indicates the evolution of each term within scientific disciplines. The topical coverage for AI has increased for all scientific disciplines with the highest publication change in ‘Electrical Engineering & Computer Science’ (10,391), ‘Chemical, Mechanical & Civil Engineering’ (6,283), and ‘Social Sciences’ (2680) (see also S1 Fig and S1 Table in S1 Appendix).

Robotics

The top-10 funding organizations (seven were merged, see S6 Table in S1 Appendix) and the top-10 research organizations are given in Fig 6 (right panel). As can be seen, the U.S. is clearly providing the largest amount of funded publications, while the strongest research institutions are in France. The Chinese funding organization NSFC is the second top-funding agency; however, no Chinese research institution is listed among the top-10. The University of California System (UC System), the University System of Georgia (USG), and MIT are among the top-10 research institutions with the most papers.

In the 3,074 NSF awards, there are 654 total bursts, with 28 double, and two triple bursts (“CPS” and “Impacts”). Summing up burst weights of double and triple bursts results in the top-15 bursts, rendered in blue in Fig 6. As for the 13,931 WOS publications, there are 261 bursts total with seven double bursts. The top-15 bursts are rendered in orange in Fig 6 (middle). Between NSF and WOS keywords, there are 47 overlapped keywords. However, there is no overlapping between the top-15 bursts.

Burst weight is indicated by the bar thickness, with “Soft Robotics” having the strongest burst in 2014-2017 for WOS with the weight of 39.3 and “Law” for NSF with the weight of 31.6. “STEM” is the longest most recent NSF burst between 2010 and 2015.

The original dataset has 30,784 unique authors. There are 2,363 authors with more than three papers, 1,531 authors with more than four papers, and 1,096 with more than five papers. There are 621 isolates, authors who have not co-authored with any others during the 20 years. There are 96,982 co-author edges. The average degree is 6.30. The network has 3,255 weakly connected components, including the 621 isolates. The largest connected component consists of 18,545 nodes. The network was filtered by times-cited ≥ 50 resulting in 2,644 nodes and 10,144 edges. The largest connected component of this network, shown in Fig 7 (middle), has 635 authors with 30 isolates removed. The figure uses the very same size- and color-coding as the co-author network for AI (Fig 7 top). The co-author labels are filtered by the number of times an author was cited (≥ 700). Author ‘Menon, Mani’ has the largest number of citations (3,890) in this network. Menon is also the top-cited US author in the 30,784 co-authors network, along with seven other authors (marked in bold in Fig 7 middle table) that appear both in the largest connected component and listed as the top-cited US authors.

Fig 8 (middle left) shows the co-author network for robotics with nodes representing the number of citations and a darker hue indicating the first year of publication for a given author. The network shows a large concentration in the eastern and mid-U.S. states. Pittsburgh and Cambridge are the top-two cities, with robotics research being cited 6,486 and 5,672, respectively (see Fig 8, middle right).

‘Electrical Engineering & Computer Science’ has been the front-runner discipline in robotics for two decades both in publication and citation amount. It is also noticeable in Fig 9 (middle) that Health Related disciplines (e.g., ‘Brain Research’, ‘Health Specialties’) increased over time. Similar to AI, robotics show a steady increase within each of the 15 disciplines (see also S2 Fig in S1 Appendix). The top-five most cited papers, along with their publication year and total number of citations, are shown in S4 Table in S1 Appendix.

Internet of things (IoT)

The term internet of things (IoT) refers to the network of physical devices such as phones, vehicles, or home appliances that have embedded electronics, software, sensors, actuators, and connectivity allowing them to collect, exchange, and act upon data. The dataset used here starts in 2006. Hence, there are no bursts or authors active before that year.

The top-10 funding organizations and the top-10 research organizations are given in Fig 6 (bottom right). As can be seen, funding from Chinese institutions is most often acknowledged, and three out of the top-10 research institutions are from China. NSF in the U.S. ranks second and two U.S. institutions are listed in the top-10 list. Funding by two European institutions is cited frequently, and four top-10 research institutions are from France.

In the 348 NSF awards, there are 77 total bursts, with no term bursting more than once. The top-15 are shown in Fig 6 (bottom left). Similarly, for the 11,371 WOS publications there are 77 bursts total with no double bursts. There is no term that bursts in both sets. Burst weight is indicated by bar thickness, with “RFID” (Radio Frequency Identification) having the strongest burst of 62.2 in 2006-2013. It is important to point out that there is a clear separation of initial publications bursts, followed by several funding bursts, followed by a new set of publications bursts. The AI and robotics bursts were much more intermixed. Furthermore, there is a difference in terms of the strongest bursting weight between NSF and WOS. The most bursting term for WOS was “RFID” (62.16), whereas NSF had a much smaller value for its strongest burst, “Vehicles” (4.26).

The original IoT dataset has 23,691 nodes and 56,937 edges. In this network, there are 6,979 authors with more than three papers and 5,517 authors with more than five papers. The network has 3,345 weakly connected components, including the 506 isolates. The largest connected component consists of 11,438 nodes. After filtering by times-cited ≥ 5, the network resulted in 6,939 nodes and 12,371 edges with the largest component of 585 with 98 isolates removed. The network was further filtered by times-cited ≥ 5 with 585 authors plotted in Fig 7 (bottom). The figure uses the very same size- and color-coding as the co-author network for AI (Fig 7 top). The labels are filtered by the times-cited (≥ 400). The most cited author is ‘Xu Ld’ with 3,358 citations in the filtered network.

Fig 8 (bottom left) exhibits the co-author network for IoT. The lighter hue indicates more recent first publications by authors in this network. Kalamazoo and Norfolk are the top-two cities with IoT research being cited 902 and 635 respectively (see Fig 8 bottom right).

The topical distribution of WOS publications on IoT is shown in Fig 9. Note that there are only seven papers published in 1998-2007. All other 11,364 journal papers were published in the recent decade. Most of the papers are in the ‘Electrical Engineering & Computer Science’ discipline with some in ‘Chemical, Mechanical and Civil Engineering.’ Note the larger number of papers in ‘Electrical Engineering & Computer Science’ published in venues such as Wireless Personal Communications (74 papers) and International Journal of Distributed Sensor Network (58) dealing with personal and complex implications of IoT. It is also noticeable that IoT increases its topical coverage from three disciplines (1998-2007) to 13 disciplines (2007-2018), supporting evidence that IoT is a nascent field (see also S3 Fig in S1 Appendix). The top-five most cited papers plus publication year and total number of citations are shown in S4 Table in S1 Appendix.

Convergence

Over the last 20 years, the three areas of “artificial intelligence,” “IoT,” and “robotics” have been merging. That is, there are more and more publications, funding awards, and keywords that are shared between pairs or even among all three of these areas. Fig 10 shows the increase in inter-citation linkages. Citations from papers in AI to papers in robotics and IoT are given in yellow; arrows are thickness-coded by the number of citations. Note that early arrows are rather thin while more recent citation links are thicker. As expected, only papers from earlier or the same year can be cited (i.e., arrows either point downwards or down-left). Citations from papers in robotics are given in red and many cite papers in AI.

Fig 10

Temporal convergence between AI (yellow), robotics (red), IoT (blue).

Citations from papers in IoT are given in blue and they go back to AI and robotics papers as early as 2000; as the IoT dataset only covers papers published in 2004-2017, they only start citing papers from other domains in more recent years (2012-2017), particularly heavily citing robotics in 2013 and 2015 and AI in 2009. Of interest here, we see a spike in NSF awards for IoT research during those years (see Fig 1B).

Validation: User studies

Decision makers from a variety of areas at NSWC Crane were invited to examine and help interpret initial versions of the visualizations related to AI in terms of readability, memorability, reproducibility, and utility. This study was approved by the Indiana University Institutional Review Board (protocol number 1809442778). This user study was administered as an online survey delivered via Qualtrics which took 30-50 minutes for participants to complete, see S1 Appendix for survey instrument. Five expert decision makers participated in the user study. Participants were presented with five visualizations on the topic of artificial intelligence. They were asked to complete tasks demonstrating their ability to interpret the visualizations and were asked to provide feedback on the utility of the visualizations in their particular line of work. Feedback collected during the user studies helped optimize algorithm and user interface implementations and improve documentation of the results. Three visualizations are relevant for the work presented here.

The first visualization is a burst diagram, showing bursts of terms in funding awards and in publications. Burst diagrams can be particularly useful for understanding temporal relationships between funding and publication streams. Users speculated that burst rates may be tied to larger economic conditions, which affect funding streams and R&D investment. The visualization can also confirm strategic direction and identify subtle shifts in focus. For example, one user described how the “earlier burst was related more to neural networks, algorithms, and knowledge/expert systems. The recent burst seems to be related to large datasets, computer vision, and deep learning (more “big data” topics).” One can gain insight into areas that are receiving funding now, and may therefore see research advances in the future, which can help drive the development of proposals that will be relevant to granting organizations.

The second visualization showed top-10 subnetworks with the largest number of authorships. Users noted that the ability to understand which researchers are most prolific and which are working across disciplines was valuable. In the words of one user, “having an understanding of major authors in each area and how they interrelate allows me to determine who to work with in a given topic area and who may have a larger breadth of knowledge.” The visualization could be used to identify potential collaborators as well as to deepen a general understanding of how researchers in this topic area are related.

The third visualization illustrated topical evolution visualized on a map of science. Users felt this map had the least direct application to their daily work. When asked to identify which visualizations were most relevant for their work, study participants identified both burst analysis and key authors and collaboration networks as highly relevant. This is consistent with results from the stakeholder needs analysis. Table 3 summarizes the feedback on the utility of each visualization for strategic planning, building potential partnerships, setting research agendas, determining national importance, and making hiring decisions.

Table 3

Summary from the expert qualitative opinions for three types of visualizations.

Topic	Top Organization	Top Agencies	Burst of Terms	Co-author Network	Topical Evolution
Strategic planning	✓	✓	✓	✓	✓
Potential partnership	✓			✓
Research			✓	✓	✓
National importance	✓	✓
Hiring			✓	✓

Discussion and outlook

This project used large-scale publication and funding data to support the analysis and visualization of key experts, institutions, publications, and funding in strategic areas of interest that were identified via a formal user needs analysis. Results of the study can be used to identify leading experts and potential investigators or reviewers; to detect emerging or declining areas; and to understand the role that funding agencies play in different countries and various topic areas.

Specifically, this paper used a formal stakeholder needs analysis to identify key insight needs together with strategic areas of interest. A detailed analysis of topic bursts in publication and funding data was performed; funding by international research organizations was compared; major authors in the U.S. were mapped geospatially; and the topical evolution was mapped for all three areas. Results were validated through a formal user study with professionals working in these areas. Novel visualization algorithms such as the double-burst visualization in Fig 6 and the convergence visualization in Fig 10 led to actionable insights. The data, code, and workflows developed for and used in this paper have been documented and published on GitHub (https://github.com/cns-iu/AICoEvolution) so results can be reproduced and other topic areas can be studied.

The presented research has several limitations. First, the analysis uses two high-quality, high-coverage data sources (Web of Science and NSF Awards database) but other data (e.g., publication data from the arXiv preprint repository [25] or patents to capture technology evolution [7]) could be added in future studies to capture science and technology developments. Note that an inclusion of additional data sources would require disambiguation and cleaning of author names and geographical locations. While Web of Science provides information on paper citations, arXiv does not. Going forward, we are interested to explore additional datasets such as Federal Business Opportunities (FBO) data that would make it possible to gain a more comprehensive understanding of the S&T funding landscape. Secondly, we used a variety of metrics such as the number of publications and citations, publication sources (journals), disciplines and subdisciplines, authors and their co-authorship relations, authors’ geographical locations, keywords, extracted terms (NLP MaxMatch feature engineering method), award $ amounts, type of organization and funding agency. Future work might also like to consider additional metrics such as authors’ diversity proposed by [7] or topic diversity recently suggested by [25]. Finally, we focused on a small, static subset of keywords and their alternatives (IoT vs Internet of Things) found in abstracts, titles, or keywords and did not examine contextual or semantic changes of terms over time.

Experts that participated in the survey and user study expressed a strong interest in interactive data visualizations that would make it possible to zoom into specific subareas or to retrieve details on a specific author or paper or funding award. Going forward, we plan to develop interactive visual interfaces to near real-time datasets to support experts managing and evaluating research portfolios and making strategic decisions related to the systematic growth of different research areas.

This appendix contains S1-S6 Tables and S1-S3 Figs.

(PDF)

Click here for additional data file.

11 Aug 2020

PONE-D-20-16726

Mapping the co-evolution of artificial intelligence, robotics, and the internet of things over 20 years (1998-2017)

PLOS ONE

Dear Dr. Scrivner,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

First, I'd like to commend your team of collaborators for considering such a complex meta-study. I concur with both reviewers that, as is, the manuscript is not technically sound, which is a critical issue for a journal like PLoS ONE. I therefore ask you to thoroughly revise their manuscript to address the flaws in methodology, as well as all the other detailed comments from the reviewers.

Please submit your revised manuscript by Sep 25 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Roland Bouffanais, Ph.D.

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. We note that [Figure(s) #] in your submission contain [map/satellite] images which may be copyrighted. All PLOS content is published under the Creative Commons Attribution License (CC BY 4.0), which means that the manuscript, images, and Supporting Information files will be freely available online, and any third party is permitted to access, download, copy, distribute, and use these materials in any way, even commercially, with proper attribution. For these reasons, we cannot publish previously copyrighted maps or satellite images created using proprietary data, such as Google software (Google Maps, Street View, and Earth). For more information, see our copyright guidelines: http://journals.plos.org/plosone/s/licenses-and-copyright.

We require you to either (1) present written permission from the copyright holder to publish these figures specifically under the CC BY 4.0 license, or (2) remove the figures from your submission:

1.    You may seek permission from the original copyright holder of Figure(s) [#] to publish the content specifically under the CC BY 4.0 license.

We recommend that you contact the original copyright holder with the Content Permission Form (http://journals.plos.org/plosone/s/file?id=7c09/content-permission-form.pdf) and the following text:

“I request permission for the open-access journal PLOS ONE to publish XXX under the Creative Commons Attribution License (CCAL) CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). Please be aware that this license allows unrestricted use and distribution, even commercially, by third parties. Please reply and provide explicit written permission to publish XXX under a CC BY license and complete the attached form.”

Please upload the completed Content Permission Form or other proof of granted permissions as an "Other" file with your submission.

In the figure caption of the copyrighted figure, please include the following text: “Reprinted from [ref] under a CC BY license, with permission from [name of publisher], original copyright [original copyright year].”

2.    If you are unable to obtain permission from the original copyright holder to publish these figures under the CC BY 4.0 license or if the copyright holder’s requirements are incompatible with the CC BY 4.0 license, please either i) remove the figure or ii) supply a replacement figure that complies with the CC BY 4.0 license. Please check copyright information on all replacement figures and update the figure caption with source information. If applicable, please specify in the figure caption text when a figure is similar but not identical to the original image and is therefore for illustrative purposes only.

The following resources for replacing copyrighted map figures may be helpful:

USGS National Map Viewer (public domain): http://viewer.nationalmap.gov/viewer/

The Gateway to Astronaut Photography of Earth (public domain): http://eol.jsc.nasa.gov/sseop/clickmap/

Maps at the CIA (public domain): https://www.cia.gov/library/publications/the-world-factbook/index.html and https://www.cia.gov/library/publications/cia-maps-publications/index.html

NASA Earth Observatory (public domain): http://earthobservatory.nasa.gov/

Landsat: http://landsat.visibleearth.nasa.gov/

USGS EROS (Earth Resources Observatory and Science (EROS) Center) (public domain): http://eros.usgs.gov/#

Natural Earth (public domain): http://www.naturalearthdata.com/

Additional Editor Comments (if provided):

First, I'd like to commend the authors for considering such a complex meta-study. I concur with both reviewers that, as is, the manuscript is not technically sound, which is a critical issue for a journal like PLoS ONE. I therefore ask the authors to thoroughly revise their manuscript to address the flaws in methodology, as well as all the other detailed comments from the reviewers.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: N/A

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The article analyzes three science and technology areas, namely AI, robotics, and IoT, and their coevolution over twenty years. Using data on funding, publications, and citations, the study produces visualizations of burst topics, co-author networks, and inter-citations over time, etc. A user study was conducted to examine the usefulness and utility of related visualizations.

On page 3, Table 1, 0 NSF awards for "human systems integration"? Is this related to HCI? Is it a coding error?

On page 6/7, is figure 2 a hypothetical example or based on real data? What are the parameters of the burst analysis? How were the exact keywords (both single words and phrases) identified? It is interesting to see Support Vector Machines had a 10-year burst (2008 - 2018) when other innovations such as deep learning should have gained more popularity in the 2010s.

On page 9/10, the top authors and co-author component analyses are useful, though it is difficult to read and interpret co-author networks (figure 7) and those with geographical overlay (figure 8).

Figure 9 is very interesting. However, it is again visually challenging to compare the two decade periods of 1998-2007 vs. 2008-2017. Perhaps there is a way to overlay the two maps or create another map to contrast the differences (changes/growth) between the two decades.

Figure 10 is useful in depicting the coevolution and inter-citations of the three fields. It would be better (and simpler) to run the overall statistics and show the total number of inter-citations over time.

How many subjects participated in the user study? I cannot find the number in the manuscript. Also, the user study analysis is anecdotal and lacks details about the structure and coding of related questions.

Overall, the paper is exploratory and provides interesting results about the development of three important research areas in science and technology. However, it lacks significant results and major findings as a research article.

Reviewer #2: The authors aim to demonstrate the emergence and convergence of three fields of research: Artificial intelligence (AI), robotics and Internet of things (IoT), using publication, funding and scholarly network metrics. The authors also aim to show how the novel visualizations and concepts introduced in this work could better inform directions for interdisciplinary research or new research areas.

While the article is generally well written, and is clear with respect to its intended contributions, I think it could benefit from a lot more clarity in terms of the methods considered, the presentation, the interpretation of the results and how they tie back to the primary claims of the paper. That is, how exactly do the results create a tangible potential for interdisciplinary research? In addition, the current results consider AI, robotics and IoT (fields that are known to be inter-related) and establishes their inter-dependencies over the years through various metrics/visualizations. However, it would have been more interesting to examine randomly selected fields, and determine retrospectively, whether there exists a potential for interdisciplinary research in the future. I also suggest restructuring certain portions of the article to enhance clarity. The quality and presentation of all figures must be improved. I have provide my detailed comments below:

1. line 12: Please define emergence and convergence, and clearly state which one (or both) is the focus of the paper

2. line 23: Is this statement based on your data or some previous study? I could also imagine new authors entering a research area and word burst occurring without it being very interdisciplinary.

3. lines 38-43: It is better to clearly state in a single paragraph the novelty and contributions of your work. What is it that separates it from previous work?

4. lines 58-63: Section numbers are mentioned, but sections are not numbered. Same issue throughout the article.

5. line 78: Was there any basis for selecting those 8 topics?

6. Wouldn’t it be better to analyze all 8 fields and show how your visualizations/metrics inform directions for future research?

7. Line 83-85: Why use publication and funding data to select from the list of fields? Doesn’t it make more sense to select fields that show the most growth most recently? For example, if a field X has a very high number of total funding awards and publications, it need not necessarily mean that the field is currently relevant. Most of the funding may have been obtained say, before 2005. So if one must select emerging areas, it must be based on the current trend of funding/publications and not the total number.

8. Lines 98-115: It is not clear whether these visualizations are the ones that you have shown in the figures of this paper. I think they are not. If that is the case, why not include sample of the visualizations that you refer to in line 99?

9. Line 137- why capitals?

10. To me it is not clear how these query terms would work. For example, there are probably several AI papers that were written in 1998 which would not contain the words ‘Artificial Intelligence’ or ‘Deep learning’. How would these papers be classified? Would they be ignored for the purposes of this paper? If so, it should be clearly mentioned.

11. In Fig 1, do the colored curves correspond to citations or publications?. Figure looks grainy. Quality should be enhanced.

12. Line 148: Should be 2004, not 2014.

13. In fig 1, why is the dotted line (total citations) showing a downward trend, while none of the individual fields (colored curves) seem to be decreasing? Again, are colored curves citations or publications?

14. Line 178: “There was no award for a project in all three focus areas.” - Change to “There was no award for a project involving all three focus areas.”

15. Fig2: Why not also show ‘burstiness’ through line thickness or by using colors?

16. Fig2 : Why plot both funding and publication bursts in the same plot? Wouldnt it be clearer to have separate plots for each?

17. Fig 5: What determines the position of sub-disciplines?

18. Fig 5 caption should be more descriptive.

19. Fig 3: Publication year shows 1970-1970

20. Fig3: Does the edge color represent the year of the latest co-authored papers?

21. Fig 4: In the bottom table of the figure, papers are referred to as A1,A2, which is inconsistent with the notation for papers in Fig 3 (P1,P2 etc.,)

22. Fig 5: The text written in yellow is impossible to read. In general, the figure is too grainy.

23. Fig 5: Why aren’t all colors mentioned in the legend?

24. Results section: Introduction to the fields of AI, robotics and IoT can be moved to after tge stakeholder analysis. There is no point introducing these topics in page 8, when it has been continuously been mentioned/discussed from pages 1-8. Same goes for the sub-section WOS-top organizations and funding. Better to talk about this immediately after the stakeholder analysis.

25. Fig 6: Thickness is used to depict ‘burstiness’, but this was not mentioned in the ‘Burst detection and visualization’ section

26. Lines 383-481: Why not reorganize this to talk about “Burst of activity” for all three fields, then move to “Key authors and collaboration networks” for all three and so on? This way, it would be easier to compare and show the correlations and contrasts between different fields.

27. Line 488: I did not understand the sentence about the arrow thickness. Do you mean arrows corresponding to earlier works are thinner? What if the arrow connects an early work to one of the latest works? Also, there is no perceivable difference in thickness in the figure.

28. Lines 528-535: Questions asked to users could be included in the appendix

29. Line 543: bursts

30. Clarity of all figures need to be improved

31. Fig 9: ‘Brain research’ is missing in the legend.

32. Fig 9: It would be much easier to see the differences between the sub-figures if the names of the fields are removed from the figure. They are color coded with the legend anyway, so you can do this.

33. Figure 10: Did not understand the significance of the legend ‘1 and 9’

34. Line 381: Cannot see the steady increase in the figure.

35. In the discussion, also clearly mention the drawbacks and challenges of your proposed visualizations and approach. What would be the challenges in scaling this up, say, if one wanted to examine 100 fields instead of 3? How would the inclusion of international collaborations affect your approach? What would need to be considered while taking into account currency differences when it comes to funding on a global scale?

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: Thommen George Karimpanal

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

28 Sep 2020

Dear Editor:

We are grateful for the extremely helpful review of our manuscript on Mapping the co-evolution of artificial intelligence, robotics, and the internet of things over 20 years (1998-2017).

We submit here a revised version of our article, which incorporates the changes recommended by the reviewers. We also include a point-by-point response to each of their comments.

Thank you again for your continued consideration of our work. We hope this revision is sufficiently responsive to the first round of feedback to merit publication, but we stand ready to make whatever adjustments you deem necessary.

In response to comments from Reviewer 1:

1. On page 3, Table 1, 0 NSF awards for "human systems integration"? Is this related to HCI? Is it a coding error?

Response

Thank you for raising this question. The NSF award portal was utilized to query for terms suggested by the stakeholders. “Human systems integration” was one of the suggested terms and the result of the query was zero award with selected filters applied.

2. On page 6/7, is figure 2 a hypothetical example or based on real data? What are the parameters of the burst analysis? How were the exact keywords (both single words and phrases) identified? It is interesting to see Support Vector Machines had a 10-year burst (2008 - 2018) when other innovations such as deep learning should have gained more popularity in the 2010s.

Response

Thank you for helping to clarify this. We added the following text to line 228: “Fig 2 explains this new burst visualization using a hypothetical example of six compound terms (T) that burst between 2004-2018." The algorithm used to extract keywords is described in section Keyword Extraction via MaxMatch (page 6). The parameters for burst analysis are explained in section Burst detection and visualization (page 7).

3. On page 9/10, the top authors and co-author component analyses are useful, though it is difficult to read and interpret co-author networks (figure 7) and those with geographical overlay (figure 8).

Response

Thank you for raising this concern. Our aim is to display the overall structure of the co-author network by showing major nodes and connections; network clusters, backbones, and density; and author impact via node size (#citations) and co-author network evolution (the darker a node, the older the authorship). In the Discussion and Outlook section, we discuss the need for interactive visualizations where hovering over an author would reveal his/her name, #papers, #citations, among others.

4. Figure 9 is very interesting. However, it is again visually challenging to compare the two decade periods of 1998-2007 vs. 2008-2017. Perhaps there is a way to overlay the two maps or create another map to contrast the differences (changes/growth) between the two decades.

Response

Thank you for suggesting this. The values for each period are provided in Supplemental material.

We also added to line 382 (p.14): (see also S1 Fig and S1 Table in S1 Appendix).

5. Figure 10 is useful in depicting the coevolution and inter-citations of the three fields. It would be better (and simpler) to run the overall statistics and show the total number of inter-citations over time.

Response

Thank you for suggesting this. We now added a convergence summary table (excel file) to our GitHub repository which is available at https://github.com/cns-iu/AICoEvolution/.

6. How many subjects participated in the user study? I cannot find the number in the manuscript. Also, the user study analysis is anecdotal and lacks details about the structure and coding of related questions.

Response

Thank you for this question. In the revised version, we added the number of subjects in the user study (line 503, p.18). Demographic information and survey instruments are provided in S1 Appendix and our GitHub repository available at https://github.com/cns-iu/AICoEvolution/.

7. Overall, the paper is exploratory and provides interesting results about the development of three important research areas in science and technology. However, it lacks significant results and major findings as a research article.

Response

With due respect, we would like to argue that the paper presents a rather comprehensive analysis and comparison of three strategically important research areas that are co-evolving. It presents a novel visualization of co-bursts and a novel visualization of convergence. As confirmed by the formal user study, the study results and the novel visualizations are informative and actionable by experts that need to prioritize investments and make strategic decisions in these areas.

In response to comments from Reviewer 2:

1. While the article is generally well written, and is clear with respect to its intended contributions, I think it could benefit from a lot more clarity in terms of the methods considered, the presentation, the interpretation of the results and how they tie back to the primary claims of the paper. That is, how exactly do the results create a tangible potential for interdisciplinary research? In addition, the current results consider AI, robotics and IoT (fields that are known to be inter-related) and establishes their inter-dependencies over the years through various metrics/visualizations. However, it would have been more interesting to examine randomly selected fields, and determine retrospectively, whether there exists a potential for interdisciplinary research in the future. I also suggest restructuring certain portions of the article to enhance clarity. The quality and presentation of all figures must be improved. I have provide my detailed comments below:

Response

The original submission includes high resolution vector versions of all figures. Please kindly use those when reviewing the paper.

Using 3 topics of interest as an illustration, we identified leading international academic organizations, funding agencies, top researchers relevant to each topic as well as convergence between topics enabling new pathways for a stakeholder to develop future investment and funding opportunities. We revised the abstract to make this clear.

We agree that this is an alternative approach to conduct valuable research. However, the work presented here is more pragmatic--it starts with a set of research areas that are of strategic interest to different governmental labs as well as industry representatives so that study results can directly support data-driven decision making.

2. line 12: Please define emergence and convergence, and clearly state which one (or both) is the focus of the paper

Response

The paper focuses on emergence using a 4-attribute model: novelty, persistence, growth and research community (line 15, p.1) and convergence of three emerging areas. Emerging areas (topics) were identified based on publication and funding data (line 35-42, p.2).

3. line 23: Is this statement based on your data or some previous study? I could also imagine new authors entering a research area and word burst occurring without it being very interdisciplinary.

Response

Yes, relevant prior work is cited in line 17: (Gao et al., 2011).

4. lines 38-43: It is better to clearly state in a single paragraph the novelty and contributions of your work. What is it that separates it from previous work?

Response

Our research presents novel visualizations to examine the emergence, growth and convergence of scientific research areas.

5. lines 58-63: Section numbers are mentioned, but sections are not numbered. Same issue throughout the article.

Response

Thank you. We changed section numbers to section names as suggested.

6. line 78: Was there any basis for selecting those 8 topics?

Response

A user needs analysis (see Section Stakeholder needs analysis) was conducted to select these 8 topics.

7. Wouldn’t it be better to analyze all 8 fields and show how your visualizations/metrics inform directions for future research?

Response

The top-three, strategically most valuable fields were selected to fit the comparison in a Plos One paper. All workflows are well documented and code is available on GitHub and they can be re-run for the remaining five or any other field.

8. Line 83-85: Why use publication and funding data to select from the list of fields? Doesn’t it make more sense to select fields that show the most growth most recently? For example, if a field X has a very high number of total funding awards and publications, it need not necessarily mean that the field is currently relevant. Most of the funding may have been obtained say, before 2005. So if one must select emerging areas, it must be based on the current trend of funding/publications and not the total number.

Response

Thank you for your suggestion. We selected fields that are of strategic interest to different governmental labs as well as industry representatives so that study results can directly support data-driven decision making. From the list of strategic areas, we selected the top three based on their NSF awards and publications.

9. Lines 98-115: It is not clear whether these visualizations are the ones that you have shown in the figures of this paper. I think they are not. If that is the case, why not include sample of the visualizations that you refer to in line 99?

Response

We included these visualizations in the GitHub repository with questions used for user needs and user studies. We used an Artificial Intelligence topic for user studies and we made some adjustments to figures as they were distributed as a paper-based questionnaire.

10. Line 137- why capitals?

Response

These are the examples of unprocessed keyword terms. We replaced them by lower character strings.

11. To me it is not clear how these query terms would work. For example, there are probably several AI papers that were written in 1998 which would not contain the words ‘Artificial Intelligence’ or ‘Deep learning’. How would these papers be classified? Would they be ignored for the purposes of this paper? If so, it should be clearly mentioned.

Response

Excellent point. As a field evolves, there are changes in terminology. In addition, not all authors would adhere to providing standard (widely accepted) terms. However, in dealing with a large scale dataset, we focused only on 3 words and their alternatives (IoT vs Internet of Things) found in abstracts, titles, or keywords. The extracted publications served as a baseline dataset that was used to collect pertinent keywords. That is, any term that at that time was deemed important to include as a keyword becomes a part of a compound term set.

12. In Fig 1, do the colored curves correspond to citations or publications?. Figure looks grainy. Quality should be enhanced.

Response

The original submission includes high resolution vector versions of all figures. Please kindly use those when reviewing the paper.

The “colored curves” correspond to the number of publications per each individual domain. This information is also available via the Figure 1A description which reads: “Yellow solid line represents AI publications, red solid line - robotics publications, and blue solid line - IoT publications.”

Text description for Figure 1B: ” Yellow solid line represents AI funding, red solid line - robotics funding, and blue solid line - IoT funding.

We added this information to the main paper text as well.

13. Line 148: Should be 2004, not 2014.

Response

Thank you for pointing this out. The text has been changed to “between 2007-2011” we observe a nascent field of IoT (blue solid line) showing a sharp increase.

14. In fig 1, why is the dotted line (total citations) showing a downward trend, while none of the individual fields (colored curves) seem to be decreasing? Again, are colored curves citations or publications?

Response

Excellent question. We added the following text to the main text and the figure caption: “The dotted line in Figure 1A represents the total number of citations for all three domains. Note that papers published in recent years did not yet have time to acquire a high citation count.”

15. Line 178: “There was no award for a project in all three focus areas.” - Change to “There was no award for a project involving all three focus areas.”

Response

This was changed. Thank you for suggesting.

16. Fig2: Why not also show ‘burstiness’ through line thickness or by using colors?

Response

Thank you for your suggestion. Burstiness (strength) is already encoded in the bar height (thickness). The hypothetical example in Figure 2 uses the equal bar height for each term, as described in line 229. To clarify this, we added the following text “the height (thickness) shows a burst strength.”

17. Fig2 : Why plot both funding and publication bursts in the same plot? Wouldnt it be clearer to have separate plots for each?

Response

Two plots would make each easier to examine. However, we are interested to compare bursts across datasets. This novel visualization makes it possible to do just this--the start and end dates but also burst strength can be compared. In addition, the co-occurrence of bursting terms can be easily discovered. We added this explanation to the main text.

18. Fig 5: What determines the position of sub-disciplines?

Response

The position of sub-disciplines is prescribed by the UCSD Map of Science classification system, please see details in [1]. The reference is also provided in the text.

19. Fig 5 caption should be more descriptive.

Response

Thank you for your suggestion. We added the following text to the caption: "A process of mapping scientific journal names to discipline and (sub)discipline topic (left) and the projection of journal topics into 2D spatial position (right)."

20. Fig 3: Publication year shows 1970-1970

Response

Thank you for pointing this out. We changed the legend to First Year Last Year.

21. Fig3: Does the edge color represent the year of the latest co-authored papers?

Response

In this simple network performed via Sci2, edges denote co-authorship relations and are color coded by year of the first joint publication and thickness coded by the number of joint papers.

22. Fig 4: In the bottom table of the figure, papers are referred to as A1,A2, which is inconsistent with the notation for papers in Fig 3 (P1,P2 etc.,)

Response

Thank you for this suggestion. We updated the table in Fig 4.

23. Fig 5: The text written in yellow is impossible to read. In general, the figure is too grainy.

Response

The original submission includes high resolution vector versions of all figures. Please kindly use those when reviewing the paper. We also added a shadow to the yellow labels.

24. Fig 5: Why aren’t all colors mentioned in the legend?

Response

This figure in the Method section explains the general process of computing a science map from publication data using hypothetical data.

The map has 13 disciplines that are color coded and labelled in the same color. We added explanatory text to the main paper to make this easier to understand.

25. Results section: Introduction to the fields of AI, robotics and IoT can be moved to after tge stakeholder analysis. There is no point introducing these topics in page 8, when it has been continuously been mentioned/discussed from pages 1-8. Same goes for the sub-section WOS-top organizations and funding. Better to talk about this immediately after the stakeholder analysis.

Response

The very brief introductions to specifics of the three fields serve as an introduction to each of the three research areas. We now explain this structure and intent in the very beginning of the Results section.

26. Fig 6: Thickness is used to depict ‘burstiness’, but this was not mentioned in the ‘Burst detection and visualization’ section

Response

Thank you for your suggestion. To clarify this, we added the following text “the height (thickness) shows a burst strength.” in Burst detection and visualization section.

27. Lines 383-481: Why not reorganize this to talk about “Burst of activity” for all three fields, then move to “Key authors and collaboration networks” for all three and so on? This way, it would be easier to compare and show the correlations and contrasts between different fields.

Response

We thank the reviewer for this suggestion. We presented topics separately to provide a more holistic picture of topic evolution and then merged them to depict their convergence.

28. Line 488: I did not understand the sentence about the arrow thickness. Do you mean arrows corresponding to earlier works are thinner? What if the arrow connects an early work to one of the latest works? Also, there is no perceivable difference in thickness in the figure.

Response

The thickness of arrows corresponds to the number of citations (line 488). The visualization shows that the early papers do not have many converging citations (across disciplines), that is they are thinner. The legend shows the thickness on the scale between 1 and 9. We added a legend title “#TimesCited”.

29. Lines 528-535: Questions asked to users could be included in the appendix

Response

User needs and study questions are included in the GitHub repository created for this paper.

30. Line 543: bursts

Response

This was changed. Thank you for suggesting.

31. Clarity of all figures need to be improved

Response

The original submission includes high resolution vector versions of all figures. Please kindly use those when reviewing the paper.

32. Fig 9: ‘Brain research’ is missing in the legend.

Response

Thank you for pointing this out. We updated the legend in Fig 9.

33. Fig 9: It would be much easier to see the differences between the sub-figures if the names of the fields are removed from the figure. They are color coded with the legend anyway, so you can do this.

Response

The original submission includes high resolution vector versions of all figures. Please kindly use those when reviewing the paper.

34. Figure 10: Did not understand the significance of the legend ‘1 and 9’

Response

The thickness of arrows corresponds to the number of citations (line 488). The legend shows the thickness on the scale between 1 and 9. We added a legend title “#TimesCited”.

35. Line 381: Cannot see the steady increase in the figure.

Response

We made the following modification to line 381-382: “The topical coverage for AI has increased for all scientific disciplines with the highest publication change in Electrical Engineering & Computer Science (10,391) and Chemical, Mechanical & Civil Engineering (6,283), and Social Sciences (2680) (see also S1 Fig and S1 Table in S1 Appendix)”

36. In the discussion, also clearly mention the drawbacks and challenges of your proposed visualizations and approach. What would be the challenges in scaling this up, say, if one wanted to examine 100 fields instead of 3? How would the inclusion of international collaborations affect your approach? What would need to be considered while taking into account currency differences when it comes to funding on a global scale?

Response

We thank the reviewer for these questions. First, Gephi is suitable for large-scale network visualizations (see Yifan Hu and OpenOrd plugins) and widely used with large biomedical data (e.g., genes, proteins) [2].

Secondly, the Web of Science is a global repository of scientific papers. Our top funding agencies and organizations analyses show who is leading in a particular topic of interest. For instance, AI research leading institutions are global: U.S., India, China. Similarly, the co-authors network combines both national and international connections. US map was selected as a geographic layout in this analysis, however, the international map is available as long as there are authors’ coordinates in the metadata.

Finally, including international funding will require an additional investigation whether such data is available, and then converting funding via currency exchange.

References

1. Börner K, Klavans R, Patek M, Zoss AM, Biberstine JR, Light RP, et al. Designand update of a classification system: The UCSD map of science. PLoS ONE.2012;7(7):e39464. doi:10.1371/journal.pone.0039464.

2. Pavlopoulos, G. A., Paez-Espino, D., Kyrpides, N. C., & Iliopoulos, I. (2017). Empirical Comparison of Visualization Tools for Larger-Scale Network Analysis. Advances in Bioinformatics. Hindawi Limited. https://doi.org/10.1155/2017/1278932

Submitted filename: Response to Reviewers.pdf

Click here for additional data file.

16 Oct 2020

PONE-D-20-16726R1

Mapping the co-evolution of artificial intelligence, robotics, and the internet of things over 20 years (1998-2017)

PLOS ONE

Dear Dr. Scrivner,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

Please submit your revised manuscript by Nov 30 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Roland Bouffanais, Ph.D.

Academic Editor

PLOS ONE

Additional Editor Comments (if provided):

Reviewer #2 still has some issues that should probably be addressed in a second round of revision.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Partly

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: No

Reviewer #2: N/A

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: (No Response)

Reviewer #2: 1. While I agree with the authors’ argument that the work examines areas of strategic interest, my concern is that these selected areas happen to be very correlated with each other. This raises the question of how useful the presented approach would be when the selected areas are not correlated with each other. For example, if a policy maker wanted to make inferences about say, ‘Power and energy management’ and ‘robotics’ (or any two fields that don’t seem to be immediately/obviously correlated), I wonder how the approach would fare.

My point is that currently, the authors show how their approach can be useful when related fields are considered. While this is useful, I think it is also equally important to show what the results would look like when your approach is used to evaluate seemingly unrelated fields of research. I believe replicating at least some of the experiments on other non-correlated fields would serve to strengthen the contribution of this paper, which is also a concern raised by Reviewer 1. Otherwise, it would seem like the authors only considered the convenient choice of correlated fields of research.

2. The authors’ response to point 8. from the first round of reviews did not really address my concern. I am not suggesting the analysis be re-done based on the current trend (which could be quantified by say, the rate of change in the citations and publications), but I think the authors need to justify why using the total citations and publications is more relevant than using the current trends of these quantities, when it comes to the strategic interests of governments and industries. At the very least, the authors should acknowledge that other measures (other than the total pubs.+cites.) could be also be used.

3. The point about changes in terminology is an important one (comment 11. from the first round of reviews). I hope the authors include the points mentioned in their response, to the main manuscript.

4. The text overlaid on top of the graph in Fig 9 can be removed (point 33. from 1st round of reviews). This could make the figure significantly less cluttered, without any loss of information, as the corresponding color codes are already mentioned in the legend.

5. Regarding point 36., I still think it would be valuable to add a section on the limitations and future issues that remain to be addressed. This could be useful for those aiming to develop similar visualization tools in the future.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: Yes: Thommen Karimpanal George

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

11 Nov 2020

Dear Editor:

Thank you for the detailed review of our manuscript on Mapping the co-evolution of artificial intelligence, robotics, and the internet of things over 20 years (1998-2017).

We submit here a revised version of our paper, which implements many of the suggestions by Reviewer 2. We also include a point-by-point response to each of the comments.

Thank you again for your continued consideration of our work. We hope this revision addresses all remaining concerns and merits publication, but we stand ready to make whatever adjustments you deem necessary.

In response to comments from Reviewer 2:

1. While I agree with the authors’ argument that the work examines areas of strategic interest, my concern is that these selected areas happen to be very correlated with each other. This raises the question of how useful the presented approach would be when the selected areas are not correlated with each other. For example, if a policy maker wanted to make inferences about say, ‘Power and energy management’ and ‘robotics’ (or any two fields that don’t seem to be immediately/obviously correlated), I wonder how the approach would fare. My point is that currently, the authors show how their approach can be useful when related fields are considered. While this is useful, I think it is also equally important to show what the results would look like when your approach is used to evaluate seemingly unrelated fields of research.

Response

Thank you for sharing your concern. Many strategic decision makers (research team leads, funders, teachers) are very interested to understand the interplay of research areas that co-evolve and impact each others’ growth). The paper presents general workflows and an exemplary analysis of three research areas that have high R&D impact and value.

Please do note that the same workflows can be used to understand the evolution of unrelated research areas. For example, data on ‘Power and energy management’ and ‘robotics’ can be compiled via the Web of Science and NSF award portals using relevant query terms. The NLP MaxMatch algorithm for keywords extraction and other workflows on Github https://github.com/cns-iu/AICoEvolution/ can be used to analyze and visualize these research areas. Even if the research areas are unrelated and no matching terms are found (aka null convergence), emerging topical areas in published work and awards, bursts, and co-author networks can be studied. However, this is not the main focus of the present paper.

2. I believe replicating at least some of the experiments on other non-correlated fields would serve to strengthen the contribution of this paper, which is also a concern raised by Reviewer 1. Otherwise, it would seem like the authors only considered the convenient choice of correlated fields of research.

Response

Just to reiterate, the paper introduces two novel visualizations (co-bursts and convergence) that are particularly valuable for studying the convergence of research areas. Both visualizations were exemplarily used to map three research areas of strategic importance and both were evaluated by domain experts.

As we write in our original response to Reviewer 2, “the work presented here is more pragmatic--it starts with a set of research areas that are of strategic interest to different governmental labs as well as industry representatives so that study results can directly support data-driven decision making.”

Reviewer 1 was originally concerned about the “lack of significant results and major findings”) and not the choice of the research areas. In our recent revision, we pointed out that the paper introduces two novel visualizations (co-bursts and convergence) that were evaluated by domain experts which fully addressed the concerns of Reviewer 1.

3. The authors’ response to point 8. from the first round of reviews did not really address my concern. I am not suggesting the analysis be re-done based on the current trend (which could be quantified by say, the rate of change in the citations and publications), but I think the authors need to justify why using the total citations and publications is more relevant than using the current trends of these quantities, when it comes to the strategic interests of governments and industries. At the very least, the authors should acknowledge that other measures (other than the total pubs.+cites.) could be also be used.

Response

Thank you for clarifying your concern. In our approach, we used a variety of metrics in addition to #publication and #citations, such as publication source (journals), discipline and subdiscipline, authors and their co-authorship relations, authors’ geographical locations, keywords, extracted terms (NLP MaxMatch feature engineering method), award $ amount, type of organization and funding agency.

We added the following text (p.2): “Recent studies introduced several novel NLP methods to measure research diversity and interdisciplinarity. For example, topic modeling has been used to measure the degree of topic diversity (Yegros-Yegros et al., 2015), Shannon’s entropy measure was applied to compute technological diversity using EU-funded nanotechnology projects data (Paez-Aviles et al., 2018).

4. The point about changes in terminology is an important one (comment 11. from the first round of reviews). I hope the authors include the points mentioned in their response, to the main manuscript.

Response

Thank you for suggesting this. We added to the study limitation (p.15): “Finally, we focused only on a small subset of keywords and their alternatives (IoT vs Internet of Things) found in abstracts, titles, or keywords and did not examine contextual or semantic changes of terms over time.”

5. The text overlaid on top of the graph in Fig 9 can be removed (point 33. from 1st round of reviews). This could make the figure significantly less cluttered, without any loss of information, as the corresponding color codes are already mentioned in the legend.

Response

The visualization was computed using the Make-A-Vis tool. We prefer to keep the current figure to support easy and complete reproducibility of the presented workflow. This is analogous to a map of the world which would also show names of major geographic areas, e.g., continents or countries.

6. Regarding point 36., I still think it would be valuable to add a section on the limitations and future issues that remain to be addressed. This could be useful for those aiming to develop similar visualization tools in the future.

Response

Thank you for this suggestion. We added the following text to (p.15): “The presented research has several limitations. First, our analysis is based on two data sources: Web of Science and NSF Awards projects. Future studies might like to include publication data from the ArXiv preprint repository (Klinger et al., 2020) or patents to capture technology evolution (Paez-Aviles et al., 2018). Secondly, we used a variety of metrics such as the number of publication and citations, publication source (journals), discipline and subdiscipline, authors and their co-authorship relations, authors’ geographical locations, keywords, extracted terms (NLP MaxMatch feature engineering method), award $ amount, type of organization and funding agency. Future work might also like to consider authors’ diversity as proposed by (Paez-Aviles et al., 2018) and topic diversity (Weitzmn and Rao-Stirling) proposed by (Klinger et al., 2020). Finally, we focused on a small, static subset of keywords and their alternatives (IoT vs Internet of Things) found in abstracts, titles, or keywords and did not examine contextual or semantic changes of terms over time.”

References

Klinger, J., Mateos-Garcia, J., & Stathoulopoulos, K. (2020). A narrowing of AI research. ArXiv.

Páez-Avilés, C., Van Rijnsoever, F. J., Juanola-Feliu, E., & Samitier, J. (2018). Multi-disciplinarity breeds diversity: the influence of innovation project characteristics on diversity creation in nanotechnology. Journal of Technology Transfer, 43(2), 458–481. https://doi.org/10.1007/s10961-016-9553-9

Submitted filename: Response to Reviewers.pdf

Click here for additional data file.

13 Nov 2020

Mapping the co-evolution of artificial intelligence, robotics, and the internet of things over 20 years (1998-2017)

PONE-D-20-16726R2

Dear Dr. Scrivner,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Roland Bouffanais, Ph.D.

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

18 Nov 2020

PONE-D-20-16726R2

Mapping the co-evolution of artificial intelligence, robotics, and the internet of things over 20 years (1998-2017)

Dear Dr. Scrivner:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Professor Roland Bouffanais

Academic Editor

PLOS ONE