How Visualization PhD Students Cope with Paper Rejections.

We conducted a questionnaire study aimed towards PhD students in the field of visualization research to understand how they cope with paper rejections. We collected responses from 24 participants and performed a qualitative analysis of the data in relation to the provided support by collaborators, resubmission strategies, handling multiple rejects, and personal impression of the reviews. The results indicate that the PhD students in the visualization community generally cope well with the negative reviews and, with experience, learn how to act accordingly to improve and resubmit their work. Our results reveal the main coping strategies that can be applied for constructively handling rejected visualization papers. The most prominent strategies include: discussing reviews with collaborators and making a resubmission plan, doing a major revision to improve the work, shortening the work, and seeing rejection as a positive learning experience.

Introduction

Facing paper rejections in academia is common and sometimes has adverse effects on individuals. Existing research studies the effect of paper rejections on scholars [4, 7] and provide some guidelines on how to cope with such experiences (e.g., [3, 9]). However, the studies were conducted for management and statistical sciences and the results might not necessarily generalize to the visualization community due to the diversity in reviewing models and culture. Additionally, these studies did not focus on PhD students specifically. We are interested in this group because we want to understand, among other things, if and how PhD students are negatively affected by paper rejections as well as what are their strategies to cope with it. PhD students, as new and inexperienced members of the community, might suffer more from negative reviews and paper rejections. If badly timed, a single rejected paper can lead to significant delays in completing the PhD dissertation or even threaten its successful completion.

Within visualization research, there exists literature that outlines common reasons of a paper rejection and provides guidelines for preparing a manuscript for submission [8]. Additionally, senior researchers of the visualization community have made suggestions on how to deal with paper rejections. Elmqvist [5,6] described some strategies on how to handle paper rejections while acknowledging that young PhD students are most vulnerable. Taking a wider perspective, Shneiderman [10] laid his thoughts on a new paradigm of research providing guidance to the students, junior and senior researchers, and policymakers. However, we lack research that studies how PhD students in the visualization research community perceive and handle paper rejections.

In this paper, we aim at understanding and revealing the strategies of visualization PhD students in coping with paper rejections. We conducted an online study to gather responses from our target user group, i.e., PhD students working in visualization research community. Based on the responses, we performed a qualitative data analysis and report our findings. We provide our questionnaire, raw data, and coded responses as a supplemental material [1].

Study Design

We decided to use a short online questionnaire for the study to reach visualization PhD students internationally. To not bias the participants towards certain answers and to also discover unexpected strategies, the questionnaire used open-ended questions for addressing the main research questions. Hence, a mostly qualitative analysis is required to identify common themes and noteworthy exceptions. To capture the full breadth of the community, we strove towards diversity of participants with respect to the country of affiliation and level of experience (among PhD students).

Questionnaire

The online questionnaire consists of three sections: general information, publication history, and handling paper rejections. The questions in the first section focus on knowing the stage of participants’ doctoral studies, countries of their universities, and field of study during their bachelor and master studies. The second section has two questions and aims at gauging the research experience of participants in terms of submitted and rejected number of peer-reviewed research articles. The third and final section captures the experience of participants with the papers that were rejected and re-submitted. All questions in the third section were open-ended questions and participants could write as much as they wanted. In particular, we asked the following four questions:

Q1: How did your lab-mates/colleagues/supervisors help to handle paper rejections?

Q2: What strategic decisions did you take to re-submit the rejected papers?

Q3: How did you handle the papers that got rejected more than once? (if any)

Q4: How did your impression of paper reviews change over time?

These questions focus on capturing the possible ways or steps that students may have taken to cope when they receive a paper rejection. The questionnaire was designed to be completed in approximately 25–30 minutes.

Target Participants and Recruiting

The target participants were the PhD students from the visualization community who have submitted research articles to peer-review venues and have already experienced one or more paper rejections. This target group ranges from first year PhD students to the ones who have recently (in the past 6 months) graduated or left the PhD program. The questionnaire was posted to the IEEE VIS and Infovis.org mailing lists as well as to the Discord channel used by EuroVis 2020 and Euro Graphics 2020. We also forwarded the questionnaire within our professional network by email and advertised it on Twitter.

In total, we received 43 responses until the submission date. In this paper, as responses have been still coming in during data analysis, we focus on the first 24 responses and plan on including the later responses in an extended version of this paper. The following analysis and descriptions are purely based on the first 24 responses.

Analysis Methodology

To analyze the responses, we followed an open coding process. The process began with two of the authors independently coding the first five responses, followed by a discussion on a merged set of authoritative codes. This resulted in a list of 19 codes. In the next stage, we divided the remaining responses into two equal parts (9 and 10 responses in these groups) and each coder used the list of already established 19 codes to do the coding. During this second stage of coding, both coders still came across new types of comments and occasionally had to define new codes. These codes were then immediately added to the shared list of codes. Notifying the other coder allowed him to use the newly defined code as well. This process, once again, resulted in some partly redundant codes that were again merged and resolved in another follow-up meeting. This refinement left us with a set of 28 different codes. Now, both coders used this set of codes to validate each other’s coding. The resulting conflicts during the validation phase were then resolved in subsequent meetings, which also led to a further consolidation to 25 codes (cf. Table 1).

Table 1:

List of codes along with their frequency of occurrence per question and total (sorted by total).

Codes	Frequency
	Q1	Q2	Q3	Q4	Total
improvement	4	13	11	4	32
downgrade work	–	16	5	–	21
discuss reviews	19	–	1	–	20
impression gets better	–	–	–	12	12
see rejection as positive	–	1	4	5	10
resubmission plan	9	–	–	–	9
similar venue scope	–	9	–	–	9
painful reviews at first	–	–	–	9	9
informal discussion	6	–	–	2	8
impression remains the same	–	–	–	7	7
submit to journal	–	6	–	–	6
fair reviews	1	–	–	5	6
high-level reflection of reviews	3	–	–	2	5
discuss reviews quickly	3	–	–	–	3
contradictory reviews	–	–	–	3	3
different venue scope	–	2	–	–	2
disagree with reviews	–	–	–	2	2
no special discussion	1	–	–	–	1
lack of support from coauthors	1	–	–	–	1
encouragement from supervisor	1	–	–	–	1
dealing with rejections meeting	1	–	–	–	1
supervisor’s decision	–	1	–	–	1
post on arxiv	–	1	–	–	1
unclear reviews	–	–	–	1	1
get early feedback	–	1	–	–	1
Total	49	50	21	52	172

Results

Based on the assigned codes (Table 1), we report the results per question (Q1–Q4), each discussing the most frequently mentioned types of answers and noteworthy individual examples. We begin with summarizing the general information and publication history of the participants. We numbered the participants with identifiers P1–P24. Individual participants are listed when referring to groups of five participants or fewer; else, we just provide the count of participants, but details can be found in the supplemental material [1].

Participants

Among the 24 participants, about 84 percent of the participants had done a master degree in computer science, and almost 63 percent had completed a bachelors in computer science. As shown in Fig. 1, a majority of the participants (22) were PhD students ranging from first year to sixth year and the ones who graduated in the past six months. One respondent (P2) had left the PhD program and one participant (P22) had graduated two years ago. Participants were enrolled at universities in 10 different countries, with half of the responses coming from Germany and USA (Fig. 2). None of the participant claimed of being enrolled with multiple universities. Among the participants, the number of submitted and rejected peer-reviewed articles ranged from 0 to 26 and 0 to 12 respectively (Fig. 3). One participant (P8, a second year student) did not submit any article to date. Another participant (P21, a first year student) had not experienced any rejection. But since both of them provided valuable comments in their responses, we decided to still include them in our analysis.

Figure 1:

Stage of PhD studies of the participants.

Figure 2:

Participants by country of affiliation.

Figure 3:

Number of submitted and rejected articles per participant.

Support of Collaborators (Q1)

As seen from Table 1, the most mentioned support strategies within the collaboration groups of participants were to discuss reviews (19 participants) and make a concrete resubmission plan (9 participants) with collaborators of the paper. Six participants reported to have had informal discussions (e.g., vis-à-vis during a coffee or lunch break, or via instant messaging) with their colleagues. They either discussed the reviews quickly (P3, P9, and P14) or celebrated the success of their peers and commiserated with the others who faced a rejection (P15 and P24). For getting support within the work group, the similarity of research topics seems to play a role as P14 commented: “I just talked briefly with my supervisor about it. Since I’m the only person in my lab doing visualization research, the other ones don’t know about the problems so much in detail.”

Towards improving the submission, participant P7 complained about the lack of support from coauthors in revising or reworking the paper and mentioned that they did not have background in visualization research. Four participants (P2, P7, P10, and P12) discussed the exact improvements and feasibility of these improvements for their paper with coauthors. Three experienced students (P10, P13, and P19) stated that they did a high-level reflection of reviews to understand what went wrong and to find areas of improvement.

Strategies for Resubmission (Q2)

When it comes to resubmitting rejected papers, prominent strategies are either downgrading work to resubmit in a lower-ranked venue (16 participants) or making improvements in the work (11 participants). The improvement might involve considerably extending the work with additional contributions (P12) or by including a user study (P1, P3, and P6). Both P1 and P12 stated that they converted short papers to full papers. The participants who decided to go for an improvement (11 participants) either resubmit their papers to a similar scoped venue (9 participants) or to a different scoped venue (P7 and P15). It could be better to switch to a different venue for better aligning contributions with the target audience as P15 stated: “In the case of one paper the reviewers commented that the novelty/contribution to the visualization community was limited, so here we submitted the application-based paper to a workshop with a better audience fit […].”

Being afraid that somebody else might publish similar research first, participant P24 mentioned that they had posted the paper on arXiv and described it as: “[…] if we were scared of being ‘scooped’ we would put it on arXiv and go for the next venue. If no external pressure, we hold on to next year’s visualization-related conference.”

The responses also indicated few submission strategies of the papers. Participant P7 reported his/her tendency of submitting a poster to get early feedback: “I tend to like to do posters, because I can try to think about how to present the work visually, and in verbal discussions with others. This is generally helpful for me.” Another strategy is to submit to a journal (6 participants) rather than waiting for the next conference cycle. P4 responded that he/she prefers to directly submit to journals as it provides more review cycles (major and/or minor revisions) in contrast to conferences where a major revision is considered as a reject. Participant P14 also favored this strategy as some journals in visualization community (e.g., TVCG) offers the presentation of accepted work at any major visualization conference.

Handling Multiple Rejects (Q3)

Nine participants mentioned that they had made further improvements before resubmitting the papers that were rejected more than once. Similar to handling first rejections of their papers, the participants reported to downgrade work by resubmitting to lower-rated venues (P15, P19, P20, and P23). None of the participants mentioned to have abandoned their work after getting it rejected multiple times. However, participant P7 highlighted that shortening the work could be a good strategy to avoid spending more time on a project. Four participants (P7, P14, P15, and P20) mentioned that they had seen rejection as a positive learning experience while handling papers that got multiple rejects. The response of participant P7 summarizes this positive outlook:

“My attitude is ‘this will get in somewhere, eventually’. I have seen my papers improve dramatically after revising and resubmitting, so I tend to think of rejections as a good thing, because the paper wasn’t ready and will only get better each time I try again. If it were to be on a topic where I didn’t feel like making substantial changes, I might stop at submitting a short paper or a poster, but I haven’t encountered this yet.”

However, participant P14 also highlighted the difficulty while handling papers that got multiple rejections as: “It was really hard to not give up. There were many tears. But in the end, you have to stand up again and improve your work.”. Eleven participants either did not provide any answer to the question or mentioned that, until now, they had not faced multiple rejections of the same paper.

Impression of Rejected Paper Reviews (Q4)

Nine participants highlighted that it was painful to read the reviews at first, especially, on the notification day. Considering how the subjective impression of the reviews changes over time, half of the participants (12) mentioned that their impression got better, while seven participants reported that their impression remained the same. Four participants (P5, P6, P7, and P15) said that they saw rejection as positive. A response from participant P6 summarizes a strategy to avoid taking reviews personally: “[The impression of the reviews] fairly remained the same. I always took it constructively and know that rejections are part of the science process. It’s not [a] reflection on me personally or my skills.”.

Five participants (P3, P12, P14, P20, and P22) reported that overall they had gotten fair reviews while two of them (P3 and P22) disagreed with reviews on some points. Three participants (P1, P7, and P14) reported that sometimes reviewers had provided contradictory reviews, while participant P18 mentioned receiving unclear reviews. Participant P13 mentioned a strategy of not considering each point in a review; instead he/she had learned to do a high-level reflection of reviews with experience:

“My perception of specific reviews did not change much with time, but my general view on reviews over the years changed a lot, in such a way that I am not any longer simply considering each point in the review, but am asking myself more often, what went wrong in the communication of the content.”

Two participants (P15 and P24) reported doing informal discussion of the reviews. Elaborating on their responses, participant P24 said that “[…] Talking with peers who also submitted papers on review day also adds to stress if your paper wasn’t accepted.” On the other hand, participant P15 mentioned that going through the reviews with friends as a social event helps him/her to remain positive, especially when some comments in the reviews are disheartening.

Study Limitations

The geographical diversity of the participants is limited to 10 countries from Europe and North America only. Moreover, 12 out of 24 (50%) responses are from Germany and USA. We believe this might be the result of using our personal networks to circulate the questionnaire. Therefore, the analysis results may be dominated by the work culture of visualization PhD students in Europe and North America.

Since the questions in the third section of our questionnaire were open-ended and broad, we provided some examples for each question to specify along which lines participants can think of while answering the questions. On the one hand, those examples were good to encourage participants to share their thoughts in case they felt lost. But this might have influenced the responses to some extent on the other hand.

Along similar lines, we might have biased the participants by advertising the study in a way that implied that they might feel frustrated about rejected papers—this was done to attract the attention of more participants. In general, just focusing on rejected papers takes a perspective that cuts short examples where papers were improved in a short revision cycle and then directly accepted. Since we had to compromise on the extensiveness of the questionnaire, we cannot make statements about this or other aspects of the submission process and strategies.

Discussion

Based on the responses, we can infer that PhD students in the field of visualization research take the reviews seriously. The students seem to discuss reviews and formulate a resubmission plan and continue to make further improvements in their work. The students frequently use the strategy of downgrading work while resubmitting, either by shortening the length or opting for a lower-rated venue. It seems that the strategies work well as they eventually get their papers published. This also indicates that visualization research community has many venues and tracks where the improved work eventually gets published. The strategies reported by visualization PhD students—discussing reviews, improving the work, and resubmitting it to another venue—align well with the suggestions made by Shneiderman [10, Section 6.6] and Elmqvist [5].

Although the responses of PhD students indicate that reviews are painful at first, they try to see rejection as a positive event. Some students mentioned that they try not to take the reviews personally and highlighted that rejections do not mean incompetency, which resonates with the advice by Elmqvist [6]. Other students also mentioned that engaging in informal discussions with colleagues and friends helps them cope with paper rejection. Knowing about these strategies can be helpful especially for young PhD students who have limited knowledge about the normalcy of receiving rejections in a peer-review research process. It maybe a good idea to talk about rejections, both formally and informally, and discuss strategies on how to cope with them. The discussions potentially creates a social support system that helps in coping with rejections and matches well with the recommendation by Day [3].

There were few responses which highlighted the occurrences of unclear and contradictory reviews. Even though these factors contribute to the painful experience on the day of notification, PhD students in visualization research usually handle them positively. As a result, many students highlighted that their impression of reviews gets better with time. Some students reported that they had used their judgment and sometimes had disagreed with few comments in the reviews. Additionally, some experienced students mentioned doing a high-level reflection of reviews trying to understand the issues that might have misdirected the reviewers. These strategies can also be helpful for finding areas of improvement and formulating a plan of action, especially when the reviews are unclear, have contradictory views, or contain insufficient information to back the review rating. The high-level reflection of reviews can be done in different ways. Elmqvist suggested summarizing the reviews [5] while Shneiderman shared personal experiences of handling difficult reviews and highlighted using one’s own judgment to continue improving the work [10, Section 6.6].

Since nine respondents (37.5%) indicated to have experienced emotional distress on the day of rejection notification after reading the reviews, it may be indicative of a harsh textual connotation being used in the reviews. We argue that conscious efforts can be made by reviewers to proactively minimize the painful rejection experience. A more polite text tone over a harsh comment may be used to communicate the same review. Although not always possible, concrete suggestions for improving the work can be clearly highlighted. Moreover, alternative reviewing models can be explored and adopted that have helped in reducing the harsh tone; Besançon et al. [2] concluded in a recent survey that reviewers use a more polite tone in open and non-anonymized peer reviewing model.

Conclusion and Future Work

Based on the responses we can infer that most PhD students in the visualization research have a positive outlook to cope with paper rejections. However, sometimes the positive outlook does not correlate with the absence of a painful experience while facing rejections (e.g., P14). We hope that knowing about the strategies of others can help the students who are facing paper rejections. Also, we hope that initiating discussions on the normalcy of paper rejections in academia and talking about known strategies in coping with rejections can be helpful especially to the young PhD students joining a research team.

As part of future work, we would be interested in discerning the strategies of students having different levels of experience. Another aspect would be to validate the results of this analysis by getting expert feedback of experienced reviewers from the community. Also, we plan to extend the analysis to the complete set of responses.