Do the Criteria of Our Best-Paper Awards Need Revision?

In India, where not many psychiatrists pursue PhD and few opportunities exist for funded research in psychiatry, most research in the specialty is conducted as postgraduate theses. The only recognition available to those who do exceptional theses is the awards instituted by professional organizations, such as the Indian Psychiatric Society (IPS). Hence, the criteria to decide the winners of such awards should be balanced and equitable and help pick the best studies from a scientific perspective. We surveyed the websites of the major professional organizations in Indian psychiatry to check if their awards criteria are valid enough. Here are our findings:

Weightage for Various Components

For most awards, 60% of the total marks are allotted for the written manuscript and the remaining 40% for the oral presentation during the conference (n = 11, Box 1)). Only the North Zone of the IPS (IPS-NZ) has provided details of how these marks are further divided (Table 1). Four awards follow a division of 75% and 25%, respectively, for the written version and the presentation. In the J. C. Marfatia Award of the Indian Association for Child and Adolescent Mental Health (IACAM), 50% of the marks are for the written paper, 30% for the presentation, and 20% for answering the post-presentation queries.

Table 1.

The Breakup of the Marks in the Awards of Indian Psychiatric Society North Zone

Written Manuscript		60 Marks
1	Topic, title, its relevance, and methodology	12 marks
2	Survey of literature, references	12 marks
3	Presentation of results and discussion	12 marks
4	Conclusions and how far the study substantiates them	12 marks
5	Clarity, lucidity, precision of language, and overall elegance of the paper	12 marks
Presentation During the Conference		40 marks
1	Style, clarity, compactness of expression, and presentation	20 marks
2	Use of audiovisual aids (if any): appropriateness, quality, visibility, comprehensibility, and novelty	10 marks
3	Response to the points raised in the discussion	10 marks

1. 60% for the written version and 40% for the presentation

a. Indian Psychiatric Society (https://ancips2020.com/awards.php)

i. Marfatia Award

ii. Bhagwat Award

b. Indian Psychiatric Society North Zone* (http://ipsnz.org/awards-fellowship/)

i. Dr. A.K. Kala Award

ii. Dr. Buckshey Award

iii . Dr. G.C. Boral Award

c. Indian Psychiatric Society West Zone (http://www.ipswzb.net)

i. Dr. Anil. V. Shah Best Paper Award

d. Indian Association of Biological Psychiatry (https://anciabp.com/ashawards.php)

i. The Asha Award

e. Indian Association for Child and Adolescent Mental Health (https://www.childindia.org/award.php)

i. The Luke Clack Award

ii. The Niloufer Award

2. 75% for the written version and 25% for the presentation

a. Indian Psychiatric Society

i. The Bombay Psychiatric Society Silver Jubilee Year Award

b. Indian Association of Social Psychiatry (https://iasp.org.in/awards/)

i. Dr. G C Boral Award

ii. Dr. J K Trivedi Award

iii.Dr. Anil Malhotra Award

iv. Dr. B B Sethi Award (poster)

*Details of the breakup are available (Table 1).

For IPS Poona Psychiatrists Association Awards I & II, given for the best published papers, and for which the conference presentation part is absent, the division is as follows:

Topic, title, its relevance, and methodology: 20 marks.

Survey of literature, references, and bibliography: 20 marks.

Presentation of the results and discussion: 20 marks.

Conclusions and how far the study substantiates them: 20 marks.

Clarity, lucidity, precision of language, and overall elegance of the paper: 20 marks.

The two poster awards of IPS (Professor K.C. Dube Poster Session I award and Professor M. Murugappan Poster Session II award) also follow the above criteria. No marks are granted for the presentation, unlike in the poster awards of the IPS West Zone and Indian Association of Social Psychiatry (IASP).

Two Undervalued Vital Aspects

Contribution to the Field

The essential criterion in determining the value of a study, in any field, is the extent to which it advances knowledge, i.e., its originality or novelty.[1] Science grows incrementally; each new study should improve upon the previous research on that topic and advance the knowledge in that area to some degree. The role of the entire introduction section of an article is to emphasize and argue how well the study achieves this. Unfortunately, our survey reveals that this important criterion is either entirely ignored or given only minor importance. (For the two poster awards of IPS, when the contestants initially submit the abstract, the suggested section headings are “aims/objectives”, “methodology,” etc.; the much important “Background” is not asked for.)

As per our experience, when students plan their thesis topic, more importance is given to the ease of collecting the sample and finishing the study on time. Many guides, too, consider the thesis to be merely an exercise in teaching the student the basics of medical research and statistics. However, as we mentioned in the beginning, postgraduate theses form the bulk of the psychiatric research conducted in India. It is only fitting, therefore, that we ensure that every such study also advances knowledge. Including this as an awards criterion may encourage the students and guides to pay more attention to this aspect. Moreover, when we recently analyzed why the manuscripts submitted to this journal get rejected, lack of novelty emerged as the commonest reason.[2] Hence, giving novelty its due importance in the awards criteria may help enhance the publication worthiness of our studies as a whole.

As the contestants may miss to cite or even intentionally hide better quality studies on their topic, to assess the originality of a study, the judges may have to search online academic databases. Hence, it may not be possible for the floor judges to assess novelty – this task should be entrusted to the judges who evaluate the written manuscript.

The Quality of the Methods

The scientific rigor and appropriateness of the study methods, too, are not given sufficient prominence. It is highly unfair to bestow any award to studies with flawed methods. Judges of the written manuscripts should screen them for errors in methods, specifically major ones such as wrong study design, and including the ones we had earlier identified as common in submissions to this journal, such as assessing the prevalence of a disorder using a screening tool, using a cross-sectional design to determine causality, lack of a priori sample size calculation, using instruments that are not validated, absence of a control group when one is essential, etc.[3]

Most organizations stipulate a benchmark---papers will be allowed floor presentation or granted the award only if they get a specified minimum percentage of the marks for the written version or the total marks (Table S1). Likewise, a provision can be added that papers with grossly erroneous methods will not be moved to the second stage.

While it may be fine to choose judges for the floor presentation from a larger pool, comprising both people inside and outside academia, for the written draft, the judges should be carefully handpicked from experienced researchers, preferably with expertise in the particular subspecialty (e.g., child psychiatry, social psychiatry). To facilitate this, suggestions may be sought from journal editors as they would know good peer reviewers competent in research methodology and statistics. Also, those who are nominated as judges should be given the option upfront to voluntarily recuse themselves if they feel they are not competent in research methodology or do not have the requisite sub-specialty expertise.

Further Steps Needed in This Regard

The exact percentage of marks to be allotted for novelty and methods must be determined through broader discussions involving reputed researchers, senior practitioners, journal editors, experienced peer reviewers, and past contestants and winners. Systematic, interactive methods to arrive at a consensus, such as the Delphi method, or other techniques, may be suitable for this purpose. We feel that these two aspects together deserve at least 50% of the total marks. Moreover, as it may not be easy to quantify and accurately rate the degree of advancement of the field in studies on diverse areas,[1] a guideline in this regard too has to be developed through detailed discussions.

Restrictions on Contestants

IPS’ The Poona Psychiatrists Association Awards I & II and the Professor K.C. Dube Poster Session I award demand that “no more than two papers where a particular member appears as an author or co-author may be submitted for either award in a year.” The IPS-NZ stipulates that “no paper shall be eligible to contest for an award where a member who has won that award in the immediately preceding year appears as an author or co-author.” Though well intended, such restrictions may prevent students and those who recently passed from major institutions from contesting, as some other paper in which their guide is a co-author may be contesting in the same year or have won in the previous year. People who win awards may mentor others; why deprive their mentees from contesting just because of their names on the papers? In this regard, the clauses by The Luke Clack Award of IACAM (“The award can only be won once by any member of the society as the principal author.”) and most research awards of IASP (“No person shall win the award more than once as principal author.”) are apt.

For the Bhagwat Award of IPS, the presenting author and all co-authors have to be under 35 years of age. This is unfortunate, as, in most centers, it will not be feasible for younger faculty and residents to conduct research without the involvement and support of the more senior and experienced faculty.

To be considered for any award, a study should have ethics approval, and the contestants should submit a copy of the approval certificate along with the written paper---this was not specifically mentioned in any of the websites we surveyed.

The award committees should do a plagiarism check of all submissions.

Only IPS-NZ says that if most of the judges feel that no paper is of high enough merit, there will be no award that year. This clause can be adopted by other organizations too.

The Indian Academy of Pediatrics (https://iapindia.org/pdf/8297-IAP-AWARD-RULES-2020.pdf) specifies that the papers will be judged by a panel of four judges, of which two members will be experts in the particular field. This can be taken up by psychiatric organizations too---the written manuscripts, at least the ones selected for floor presentation, may be sent to experts on the topic the paper covers. The help of respective zonal or national journal editors may be sought to identify appropriate subject experts.

The judges of the written version may check the manuscripts against the equator network’s relevant checklist (https://www.equator-network.org/reporting-guidelines/) to assess the completeness of reporting.

For some of the awards for which detailed breakup is available, disproportionately high marks are given for the conclusions and how far they are substantiated by the study---including a whopping 20% for the two Poona Psychiatrists Association Awards for published papers. This should be reduced to about 5%, to free up room for allotting more marks for novelty and study methods.

In most awards, the response to questions in floor presentation is given disproportionately high marks. (J. C. Marfatia Award of IACAM even says that “If no questions are asked then the presenter will get at least 10 marks in this area.”) We feel that the question-answer session needs not be allotted more than 5% marks. In any case, there are practical difficulties in standardizing the difficulty level of questions posed to different candidates. Moreover, our observation is that more than the accuracy of the answers, it is the candidate’s personality that usually gets assessed, just as in a conventional viva voce session.[4] A possibility of the contestants and their buddies rigging the discussion section exists, too.

Criteria for the posters should be different from those for the papers presented orally. Here too, some marks should be allotted for study novelty, innovation, and scientific quality, just as in the oral papers. In addition, marks should also be awarded for the poster’s ability to stand alone, its organization, balance between text and figures, clarity, brevity, visual appeal, quality of graphics, and oral presentation and discussion of the findings in the poster.[5] For the poster award of the west zone of IPS, the judges assess the full written papers for a total of 60 marks (out of the total 100). This is an imitable policy, we feel.

IASP insists that the winning articles should be submitted to its journal within two weeks of the conference. IPS-NZ even specifies that all the papers submitted for awards will become the property of the organization for publication in its journal. Such clauses may be essential for many of our journals to survive, as they may be finding it difficult to fill the pages and bring out the issues on time. However, this thwarts the major aim of the awards, i.e., identifying and appreciating the best work done in the organization’s geographical area or subspecialty, because researchers who feel that their articles are good enough to get published in journals of better repute may decide to stay away.

Details of the breakup are not available online for the awards of the Indian Association of Private Psychiatry, the Dr. V K Varma Award of IASP (which is for best published paper), and the awards given by many zonal branches of IPS. Hence, we were unable to include their details in this analysis.

The respective award committees may be sharing more information about the division of marks to the judges, in addition to what is mentioned on the websites. We did not have access to those details and also did not actively try to procure them for inclusion in this analysis. (We indeed had access to some of them as we have been judges on some occasions; but, to respect confidentiality, we did not include them in our analysis.)

Click here for additional data file.

Supplemental material for this article is available online.