Measuring the impact of allied health research.

BACKGROUND: Excellence in Research for Australia (ERA) rankings are given to academic journals in which Australian academics publish. This provides a metric on which Australian institutions and disciplines are ranked for international competitiveness. This paper explores the issues surrounding the ERA rankings of allied health journals in Australia.
METHODS: We conducted a broad search to establish a representative list of general allied health and discipline-specific journals for common allied health disciplines. We identified the ERA rankings and impact factors for each journal and tested the congruence between these metrics within the disciplines.
RESULTS: Few allied health journals have high ERA rankings (A*/A), and there is variability in the impact factors assigned to journals within the same ERA rank. There is a small group of allied health researchers worldwide, and this group is even smaller when divided by discipline. Current publication metrics may not adequately assess the impact of research, which is largely aimed at clinicians to improve clinical practice. Moreover, many journals are produced by underfunded professional associations, and readership is often constrained by small numbers of clinicians in specific allied health disciplines who are association members.
CONCLUSION: Allied health must have a stronger united voice in the next round of ERA rankings. The clinical impact of allied health journals also needs to be better understood and promoted as a research metric.

Introduction

There is a worldwide impetus to develop and measure metrics of research excellence.1 For instance, research benchmarking activities have been established over the last 7 years in Australia,2 the UK,3 South Africa,4 Europe,5 and the USA.6 Research metrics have been expressed in different forms in different countries. Despite this lack of standardization, research metrics have implications for universities in terms of their international academic standing, and their competitiveness for infrastructure and research funding. These metrics also have implications for individual researchers in terms of their academic status and credibility, their competitiveness for grants, and academic promotion.

In Australia, lobbying recently commenced for the Excellence in Research for Australia (ERA) ranking assigned to individual journals in the second ERA rankings, due to be published in 2012.7 The history of ERA is described later in this paper. ERA objectives2 are to:

Establish an evaluation framework that gives government, industry, business, and the wider community assurance of the excellence of research conducted in Australia’s institutions

Provide a national stocktake of discipline areas of research strength and areas where there is an opportunity for development in Australia’s higher education institutions

Identify excellence across the full spectrum of research performance

Identify emerging research areas and opportunities for further development

Allow for comparison of Australia’s research nationally and internationally for all discipline areas.

Based on the initial 2010 round of ERA rankings for journals, we have already observed the impact within our own university, in terms of researchers’ perceived competitiveness for promotion, their success in applying for nationally competitive grants, and their academic standing. Allied health researchers in our School of Health Sciences have been advised that unless they publish in A* or A journals (the top two rankings of ERA journals) they may not be competitive for grants or academic promotion, despite the fact there may be no relevant journals in their discipline in these categories. This has raised concerns that if allied health researchers seek to publish in these high-ranked journals, they may well improve their individual academic standing and contribute to their university’s research ranking, but their work may not be read by the appropriate audience, and therefore it may have minimal impact.

Allied health is a complex and evolving collective of disciplines. It is widely accepted as an umbrella term referring to all health services which are not medicine or nursing.8 Pharmacy and dentistry may or may not be included. Usually, the term “allied health” covers a range of health disciplines with different tasks, ranging from the physical therapies (such as physiotherapy, occupational therapy, and podiatry), to counseling services (such as social work and psychology), manufacturing services (such as orthotics and prosthetics), and diagnostic and support services (such as audiology, pathology, medical physicists, and imaging).8 Because there is no standard definition for allied health, research can be published variably in discipline-specific, multidisciplinary, and health services journals. The discipline-specific journals are often academic vehicles sponsored by professional associations, and thus they can have varied financial, organizational, and academic supports, and readership and publication constraints.

Within the allied health research community, the main reason for publishing is to improve clinical uptake of practices that are evidence based.8,19 Thus, clinicians are often the primary research endpoint, and the findings from translational (applied) research would seem to be best disseminated through clinician-focused journals.2,9,10,19 However, as previously indicated, allied health academics in Australia are now being strongly encouraged to publish in journals which are valued by the academic community as having high publication metrics.2 This potentially creates a tension for allied health researchers internationally, regarding whether they primarily seek academic recognition, or rather clinical impact which might change service quality and health outcomes. Consequently, the ERA ranking process for journals poses issues for Australian allied health researchers and clinicians. There is no one peak allied health academic body which could lobby effectively, so the ERA journal lobbying process raises questions about who is in a position to lobby for overall recognition of allied health research. Moreover, should lobbying be for an improved ERA rank for each discipline-specific allied health journal, or whether there should be a united Australian allied health voice which considers discipline-specific journals, as well as journals in which multidisciplinary health research might be published?

The importance of appropriately disseminating research information in a manner that will influence clinical practice decisions and change practice behaviors has been increasingly noted over the past decade.9–12 Such is the importance of improving the safety and quality of health care by understanding the most appropriate mechanisms of knowledge transfer (from research into clinical practice) that a whole new field of research has emerged (implementation science).9–12 As a consequence, the capacity of publication metrics to express appropriately how the research evidence was disseminated has been questioned in recent international publications.13,16

This paper investigates the ERA publication metrics in allied health research in six core allied health areas for the journals relevant to common fields of research.

Publication metrics

There is ongoing international debate about the best type and use of publication metrics.13–16 In Australia, there are currently two systems used to measure and rank the value of a journal title. The first system relates to the journal impact factor. This is an indicator used to rank, evaluate, and compare journals. Journal impact factors have been used internationally to assist authors when considering journals in which to publish and for libraries to identify potential quality journals to add to their collections. As a result of the work of Garfield and Sher in the early 1960s, journal impact factors have been updated annually and published in the Journal Citation Reports produced by Thomson Reuters since 1975.17 Derived from an analysis of the data held in the Thomson Reuters citations, these have become a way of measuring the relative quality of a journal within a particular subject area or field. The journal impact factor is described as being “based on 2 elements: the numerator, which is the number of citations in the current year to items published in the previous two years, and the denominator, which is the number of substantive articles and reviews published in the same 2 years.”15 Expressed as a calculation, the 2010 impact factors would be determined as follows.

The 3-year time frame for the calculation means that it takes time for new journals to be accepted and have an impact factor. Since the production of Journal Citation Reports in 2007, 5-year impact factors are starting to be provided. An issue for some disciplines is that impact factors are only available for the journals accepted to be in Journal Citation Reports, and the coverage of some fields is limited. It is recognized that there has been an effort to increase coverage in recent years, for instance the number of titles in the Journal Citation Reports for nursing science has gone from 32 titles in 2005 to 72 titles in 2009. As a result of the Thomson Reuters regional content expansion program, the coverage of Australian journals is increasing, with 52 new Australian journals across all subject categories added to the 2009 Journal Citation Reports.

The second metric is the one which is the subject of this paper. Whilst journal impact factors have been used internationally since 1975, changes have occurred in recent years, with individual countries looking to evaluate their research quality. In February 2008, the Australian government announced the new initiative for Australian research quality and evaluation, ie, ERA.2 The Australian Research Council was given the responsibility by the Australian government for the ERA initiative to assess research quality within the Australian higher education institutions. The information gained from this exercise would indicate institutions and disciplines that were internationally competitive and identify emerging areas of research. By August 2010, the full ERA data collection process was completed and submitted for the evaluation of the research in the eight discipline clusters, by committees consisting of experienced and internationally recognized experts. The discipline clusters comprise physical, chemical and earth sciences, humanities and creative arts, engineering and environmental sciences, social, behavioral and economic sciences, mathematical, computing and information sciences, biological sciences and biotechnology, biomedical and clinical health sciences, and public and allied health sciences. The data collection process included the creation of a list of ERA-ranked journals. In formulating the list, the Australian Research Council worked with over 400 Australian and international reviewers. The draft list was then available for universities and other interested groups to review. The final listing categorizes journals by fields of research and divides each category into A*, A, B, or C rankings, with A* being the top 5% of the titles, A the next at 15%, B the next at 30%, and C the bottom at 50%. Journals may be assigned up to three fields of research codes.

Briefly, there is a third approach, ie, actual article citation counts. Listings of these can be found at the Institute for Scientific Information Web of Knowledge Database.17 Garfield15 comments that: “The use of JIFs instead of actual article citation counts to evaluate individuals is a highly controversial issue.” There is the recognition for researchers that their article was accepted for publication in a high impact journal, but there may be no correlation between the citations for the researcher’s article, which could even be zero, regarding the citations for the journal. Although actual article citation counts are not a metric of journal evaluation, it is worth mentioning as a tool for evaluating scientific integrity without use of journal impact factors or ERA ratings. Variations of this measure include the h index, which is used as a measure to evaluate an individual author’s performance. The h index is based on the number of papers published by the researcher and on how often these papers are cited in papers written by other researchers.

Dilemma of allied health research

For Australian allied health researchers, the ERA ranking of a journal is now becoming the most important consideration when selecting journals in which to publish. When collaborating with international authors, selecting those titles that rank high in ERA but also have a significant impact factor can be a challenge for some fields. Within the allied health journals, many of which appear in the journal citation report science category for rehabilitation, important differences between the two metrics can be observed. In Table 1, the 30 rehabilitation journals listed in a recently published analysis of rehabilitation research by Shadgan et al18 (for which Thomson Reuter’s Web of Knowledge database was used) are compared against their ERA rankings and 2009 impact factor rankings (see Table 1). The 100 top cited articles were published in the first eleven journals on the list.18 These journals included: IEEE Transactions on Neural Systems and Rehabilitation Engineering, European Journal of Cancer Care, Archives of Physical Medicine and Rehabilitation, Journal of Rehabilitation Medicine, Journal of Neuroengineering and Rehabilitation, Physical Therapy, American Journal of Physical Medicine and Rehabilitation, Brain Injury, Journal of Orthopaedic and Sports Physical Therapy, Australian Journal of Physiotherapy, and Manual Therapy. Only one of these ranks in the ERA journal list as A and the others are either ranked as B or C. From the full list, there are only two A* and one other A ranked journal (Table 1).

Table 1

A list of ERA rankings, their fields of research, and impact factors for rehabilitation journals

ERA rankings 2010	ERA FoR	Titles in order of number of top cited articles from Shadgan et al18	2009 impact factor ranking in rehabilitation category (33 titles)
A*	09030906	IEEE Transactions on Neural Systems and Rehabilitation Engineering	3
A*	11101112	European Journal of Cancer Care	19
A	11031106	Archives of Physical Medicine and Rehabilitation	6
A	11031106	Journal of Rehabilitation Medicine	11
B	09031109	Journal of Neuroengineering and Rehabilitation	7
B	1103	Physical Therapy	9
B	11031106	American Journal of Physical Medicine and Rehabilitation	14
B	11031702	Brain Injury	16
B	11031106	Journal of Orthopaedic and Sports Physical Therapy	2
B	11031106	Australian Journal of Physiotherapy (now Journal of Physiotherapy)	13
B	1103	Manual Therapy	5
B	110311091702	Neurorehabilitation and Neural Repair	1
B	1106	Journal of Electromyography and Kinesiology	10
C	11031106	Clinical Rehabilitation	12
C	1103	Journal of Burn Care and Rehabilitation (now called Journal of Burn Care and Research)	Surgery ranked 62/166 (Surgery)
C	1103	Journal of Rehabilitation Research and Development	17
C	1103	Physiotherapy	22
C	1103	Journal of Manipulative and Physiological Therapeutics	20
C	11031106	Disability and Rehabilitation	15
C	1103	Folia Phoniatrica et Logopaedica	Not in 2009 JCR
C	1103	Journal of Musculoskeletal Pain	30
C	1103	Top Stroke Rehabilitation (Topics in Stroke Rehabilitation)	18
C	1106	Adapted Physical Activity Quarterly	21
C	1106	Athletic Therapy Today	33
C	1106	Journal of Back and Musculoskeletal Rehabilitation	32
C	1106	Journal of Sport Rehabilitation	26
C	1106	Physical Therapy in Sport	23
C	1106	Prosthetics and Orthotics International	25
C	1110	Rehabilitation Nursing	24
C	1117	Support Care Cancer (Supportive Care in Cancer)	8

Notes: 0903 biomedical engineering, 0906 electrical and electronic engineering, 1103 clinical sciences, 1106 human movement and sports science, 1109 neurosciences, 1110 nursing, 1112 oncology and carcinogenesis, 1117 public health and health services, 1702 cognitive sciences, JCR Science category: Rehabilitation.

Abbreviations: ERA, Excellence in Research for Australia; FoR, field of research; JCR, Journal Citation Reports.

This highlights the low ranking of premier rehabilitation journals, for example, Australian Journal of Physiotherapy (now called Journal of Physiotherapy) and Manual Therapy, which are provided to most physiotherapists as part of their professional association membership. Because these journals would have the greatest exposure to practicing clinicians, they are potentially most likely to influence clinical practice.

A specific ranked list for allied health sciences or rehabilitation is not available. This is in contrast with nursing, which has a specific ERA field of research category, and for which the Council of Deans of Nursing and Midwifery (CDNM, Australia and New Zealand) worked closely on providing information for the ERA process. Of the journal titles in the list originally prepared by the CDNM for the ERA process, all are either A* or A in ERA ratings, except for Health Expectations, which does not fall into the nursing category but is in the field of research category of public health and health services (field of research 1117). Other allied health sciences, such as occupational therapy, are mainly placed in the broad ERA field of research category of clinical sciences (field of research 1103), consisting of 1365 journals, as compared with the field of research category of nursing (field of research 1110) with 263 journals in total.

Methods

Data sources

From a range of fields of research, we collated peer-reviewed journal titles relevant to allied health research. The journal titles were then broadly divided into categories which reflected allied health discipline interests. The journal titles were collated independently by two allied health researchers (with different backgrounds, ie, general health sciences [OT], and speech pathology [KW]). The results were compared to ensure completeness. The final list was then discussed with the consultant liaison health librarian (JH). Where differences were identified, the researchers arrived at consensus through discussion. The current impact factor for each journal was then identified by searching the current journal home page, Web of Science, and/or Scopus search engines. We then sourced the current ERA ranking for each journal from the 2010 Australian Research Council ERA list.2

Analysis

All data were entered into a Microsoft Excel® spreadsheet for analysis. We charted the relationship between impact factors and ERA rankings for those journals which had both metrics. We also reported on the percentage of journals which had one metric only. To overcome the issue that some distributions of discipline-specific impact factor values were not normally distributed, we reported the median impact factor (25–75th percentile) in each ERA field of research category. We looked for differences in impact factor values within each discipline, using Kruskal–Wallis one-way analysis of variance tests. Differences were significant at P < 0.05. For ease of reporting, we rounded the impact factor median and percentile values to one decimal place.

Results

Publications associated with six core allied health disciplines were considered in this paper, ie, evidence-based practice in allied health, allied health therapy research (reflecting mostly physiotherapy, occupational therapy, podiatry), medical radiation, nutrition, speech pathology, and social work.

Evidence-based practice in allied health

Of the 13 journals relevant to allied health, only two had impact factors (one A ERA ranked journal, Evidence-based Complementary and Alternative Medicine) and one B journal (Clinical Trials, see Table 2). The ERA A ranked journal had a slightly higher impact factor than the ERA B ranked journal. The remaining journals had been assigned ERA ranks without any measure of citations or readership. We undertook no statistical analysis on this group of journals because of the small number of journals with impact factors.

Table 2

A list of evidence-based practice journals publishing allied health research

ERA rankings	Impact factor	Journal title
A	2.064	Evidence-based Complementary and Alternative Medicine
B	1.917	Clinical Trials
B		Evidence-based Child Health: a Cochrane review journal
B		Evidence-based Mental Health
B		International Journal of Evidence-based Healthcare
B		International Journal of Evidenced-based Healthcare
B		PLoS Clinical Trials
C		American College of Physicians Journal Club
C		Applied Clinical Trials: your peer-reviewed guide to global clinical trials management
C		Clinical Evidence (Print 1999–2006)
C		Evidence-based Healthcare and Public Health
C		Evidence-based Integrative Medicine
C		Evidence-based Preventive Medicine

Note: Journal Citation Reports 2009 Edition was used.

Abbreviation: ERA, Excellence in Research for Australia.

Therapies

We identified 145 journals that published allied health therapy research, particularly physiotherapy, occupational therapy, and podiatry. Every journal had an ERA ranking (see Table 3). However, the percentage of ERA ranked journals with impact factors was 100% for A* and A ranked journals, 97% for B ranked journals, and 41% for C ranked journals. Figure 1 highlights the incongruence between ERA rankings and impact factors, particularly between A*, A, and B ranked journals. Figure 1 also highlights a broad overlap of lower impact factors in B and C ranked journals. There was a significant difference between the ranked impact factors across the ERA categories (P < 0.05), with the significant difference occurring between the A*, A, and B ERA categories. The median A* ERA journal impact factor was 3.4 (3.3 [25th percentile] to 5.2 [75th percentile]). For the A journals, the median impact factor was 2.6 (2.1 [25th percentile] to 3.0 [75th percentile]), for the B ranked journals it was 2.3 (1.8 [25th percentile] to 2.7 [75th percentile]), and for the C journals it was 1.3 (0.8 [25th percentile] to 1.8 [75th percentile]).

Table 3

A list of journals which could publish allied health therapy research

ERA rankings	Impact factor	Journal title
A*	7.041	Stroke
A*	3.427	Journal of Bone and Joint Surgery – American Volume
A*	3.112	Journal of Orthopaedic Research
A	5.371	Pain
A	4.014	Movement Disorders
A	2.624	Spine
A	2.594	Rheumatic Disease Clinics of North America
A	2.576	Gait and Posture
A	2.184	Archives of Physical Medicine and Rehabilitation
A	1.909	Acta Orthopaedica
A	1.882	Journal of Rehabilitation Medicine
B	5.622	Ageing Research Reviews
B	5.398	Neurorehabilitation and Neural Repair
B	4.255	Journal of Neurotrauma
B	3.914	Experimental Neurology
B	3.8	Mental Retardation and Developmental Disabilities Research Reviews
B	3.612	European Journal of Pain
B	3.131	Age and Ageing
B	2.786	Headache
B	2.66	Journal of Bone and Joint Surgery – British Volume
B	2.626	Journal of Trauma: Injury Infection and Critical Care
B	2.507	Scandinavian Journal of Rheumatology
B	2.505	Clinics in Chest Medicine
B	2.482	Journal of Orthopaedic and Sports Physical Therapy
B	2.423	Journal of Pain and Symptom Management
B	2.391	Journal of Head Trauma Rehabilitation
B	2.324	Journal of the Neurological Sciences
B	2.319	Manual Therapy
B	2.283	Head and Neck
B	2.08	Physical Therapy
B	2.065	Clinical Orthopaedics and Related Research
B	1.956	European Spine Journal
B	1.95	Burns
B	1.88	BMC Musculoskeletal Disorders
B	1.88	Gerontologist
B	1.709	Australian Journal of Physiotherapy
B	1.556	American Journal of Physical Medicine and Rehabilitation
B	1.42	American Journal of Occupational Therapy
B	1.38	Journal of Occupational Rehabilitation
B	1.277	Neurological Research
B	1.245	Orthopedic Clinics of North America
B	1.206	Journal of Spinal Disorders and Techniques
B	0.438	Australian Occupational Therapy Journal
B		Physical and Occupational Therapy in Pediatrics
C	4.152	Arthritis Care and Research
C	3.643	Occupational and Environmental Medicine
C	3.205	Stress: the international journal on the biology of stress
C	2.871	International Journal of Stroke
C	2.383	Injury: International Journal of the Care of the Injured
C	2.25	Joint Bone Spine
C	1.95	Neuro Rehabilitation: an interdisciplinary journal
C	1.934	Journal of Shoulder and Elbow Surgery
C	1.837	Journal of Palliative Medicine
C	1.825	International Orthopaedics
C	1.783	Spinal Cord
C	1.777	Journal of Orthopaedic Trauma
C	1.767	Clinical Rehabilitation
C	1.639	Journal of Communication Disorders
C	1.617	Journal of Burn Care and Research
C	1.592	Journal of Child Neurology
C	1.555	Disability and Rehabilitation
C	1.41	Physiotherapy Canada
C	1.367	Journal of Rehabilitation Research and Development
C	1.36	Archives of Gerontology and Geriatrics
C	1.326	Journal of Hand Surgery (American Volume)
C	1.303	Clinical Neurology and Neurosurgery
C	1.226	Topics in Stroke Rehabilitation
C	1.185	Journal of Clinical Rheumatology: practical reports on rheumatic and musculoskeletal disease
C	1.17	Journal of Clinical Neuroscience
C	1.126	Knee
C	1.117	Archives of Orthopaedic and Trauma Surgery
C	1.059	Journal of Manipulative and Physiological Therapeutics
C	1.037	Clinical Neuropathology
C	0.914	Journal of Aging Studies
C	0.844	Journal of Developmental and Physical Disabilities
C	0.812	Journal of Hand Surgery (European Volume)
C	0.773	Physiotherapy
C	0.734	Topics in Geriatric Rehabilitation
C	0.717	Geriatrics and Gerontology International
C	0.687	Hand Clinics
C	0.674	Physical Therapy in Sport
C	0.671	Australasian Journal of Ageing
C	0.612	Hand Therapy
C	0.612	Journal of Hand Therapy
C	0.592	Foot and Ankle International
C	0.358	International Journal of Rehabilitation Research
C	0.184	Journal of Physical Therapy Science
C		Acta Orthopaedica et Traumatologica Turcica
C		Acta Ortopedica Brasileira
C		Acute Pain
C		Advances in Neurology
C		Advances in Physiotherapy
C		Ageing International: information bulletin of the International Federation on Ageing
C		Annals of Thoracic Medicine
C		British Journal of Occupational Therapy
C		Canadian Journal of Occupational Therapy
C		Canadian Journal on Aging
C		Clinical Neuroanatomy
C		Diabetic Foot
C		Foot and Ankle Surgery
C		Geriatrics and Aging
C		Geriatrics: better medicine for midlife and beyond
C		Headache Currents: a journal for recent advances in headache and facial pain
C		Hong Kong Physiotherapy Journal
C		Intellectual and Developmental Disabilities: journal of policy, practice, and perspectives
C		International Journal of Occupational Medicine and Environmental Health
C		International Journal of Rheumatic Diseases
C		International Journal of Therapy and Rehabilitation
C		International Musculoskeletal Medicine
C		Journal of Applied Gerontology
C		Journal of Clinical Neuromuscular Disease
C		Journal of Foot and Ankle Surgery
C		Journal of Musculoskeletal Medicine
C		Journal of Musculoskeletal Pain: innovations in research, theory and clinical practice
C		Journal of Musculoskeletal Research
C		Journal of Neurological Sciences
C		Journal of Occupational Science
C		Journal of Orthopaedic Surgery
C		Journal of Orthopaedic Surgery and Research
C		Journal of Prosthetics and Orthotics
C		Journal of Rehabilitation
C		Journal of Stroke and Cerebrovascular Diseases
C		Journal of the American Society for Surgery of the Hand (absorbed by Journal of Hand Surgery, Philadelphia)
C		Journal of the Japanese Physical Therapy Association
C		Journal of Whiplash and Related Disorders
C		Modern Rheumatology
C		New Zealand Journal of Occupational Therapy
C		New Zealand Journal of Physiotherapy
C		Occupational Therapy in Healthcare
C		Occupational Therapy in Mental Health: a journal of psychosocial practice and research
C		Occupational Therapy International
C		Orthopaedics and Trauma
C		Pain Research and Management
C		Pediatric Physical Therapy
C		Physical and Occupational Therapy in Geriatrics: current trends in geriatric rehabilitation
C		Physical Therapy Reviews
C		Physiotherapy Research International
C		Physiotherapy Singapore
C		Physiotherapy Theory and Practice: an international journal of physical therapy
C		Research on Aging
C		Scandinavian Journal of Occupational Therapy
C		Scandinavian Journal of Trauma, Resuscitation and Emergency Medicine
C		The Foot
C		The Journal of Manual and Manipulative Therapy
C		The Pain Clinic

Note: Blank cell indicates no Impact Factor identified. Journal Citation Reports 2009 Edition was used.

Abbreviation: ERA, Excellence in Research for Australia.

Figure 1

A comparison of the impact factors and ERA rankings for therapy journals (occupational therapy, physiotherapy, and podiatry).

Notes: Numbers in parentheses represent the number of journals with both metrics of journal evaluation by the total.

Abbreviation: ERA, Excellence in Research for Australia.

Medical radiation

We identified 52 journals in the medical radiation field (see Table 4), all with ERA rankings. The ERA A* and A journals all had impact factors. Impact factors were documented for 94.0% of the B ranked journals and 44.4% of the C ranked journals. Figure 2 charts the relationship between impact factors and ERA rankings, highlighting the overlap between impact factors and ERA rankings in all classifications. The one A* journal impact factor (impact factor 6.3) was analyzed with the A ERA ranked journals, when considering ranked impact factor differences across the ERA categories. The median A ERA journal impact factor was 4.7 (2.0 [25th percentile] to 3.3 [75th percentile]), the B ranked journals median impact factor was 2.6 (2.0 [25th percentile] to 3.3 [75th percentile]) and for the C journals, was 1.3(1.0 [25th percentile] to 1.8 [75th percentile]). There was a significant negative trend in ranked impact factors across each of the ERA categories (P < 0.05).

Table 4

A list of medical radiation journals, their ERA ratings and impact factors

ERA rankings	Impact factor	Journal title
A*	6.341	Radiology
A	6.424	Journal of Nuclear Medicine
A	6.424	Nuclear Medicine
A	4.85	Investigative Radiology
A	4.592	International Journal of Radiation: oncology biology physics
A	4.343	Radiotherapy and Oncology
A	1.842	International Journal of Radiation Biology
B	3.962	Seminars in Nuclear Medicine
B	3.915	Clinical Nuclear Medicine
B	3.589	European Radiology
B	3.435	Psychiatry Research: Neuroimaging
B	3.296	American Journal of Neuroradiology
B	2.877	The Quarterly Journal of Nuclear Medicine and Molecular Imaging
B	2.77	Journal of Magnetic Resonance Imaging
B	2.645	European Journal of Radiology
B	2.616	Neuroradiology: a journal devoted to neuroimaging and interventional neuroradiology
B	2.456	Nuclear Medicine and Biology
B	2.105	British Journal of Radiology
B	2.092	Academic Radiology
B	2.026	Magnetic Resonance Imaging
B	1.805	Journal of Vascular and Interventional Radiology
B	1.719	Journal of Neuroimaging
B	1.645	Clinical Radiology
B	1.041	Computerized Medical Imaging and Graphics
B		Journal of Radiology Nursing
C	4.531	European Journal of Nuclear Medicine
C	2.467	Molecular Imaging and Biology
C	1.949	Cardiovascular and Interventional Radiology
C	1.791	Abdominal Imaging
C	1.568	Skeletal Radiology: journal of radiology, pathology and orthopedics
C	1.333	Clinical Physiology and Functional Imaging
C	1.315	Nuclear Medicine Communications
C	1.186	Pediatric Radiology: roentgenology, nuclear medicine, ultrasonics, CT, MRI
C	1.044	Brain Imaging and Behavior: an international journal
C	0.956	Journal of Diagnostic radiography and Imaging
C	0.917	Annals of Nuclear Medicine
C	0.602	Journal of Digital Imaging
C		Applied Radiology: the journal of practical medical imaging and management
C		Australian and New Zealand Nuclear Medicine
C		Biomedical Imaging and Intervention Journal: a multidisciplinary open access online journal
C		BMC Medical Imaging
C		Cancer Imaging
C		International Journal of Computer Assisted Radiology and Surgery
C		Journal of Medical Imaging and Radiation Oncology
C		Journal of Medical Imaging and Radiation Sciences
C		Journal of Nuclear Medicine Technology
C		Journal of Radiotherapy in Practice
C		Neuroscience Imaging
C		Radiography
C		The Internet Journal of Nuclear Medicine
C		The Internet Journal of Radiology
C		The Neuroradiology Journal

Note: Journal Citation Reports 2009 Edition was used.

Abbreviation: ERA, Excellence in Research for Australia.

Figure 2

Impact factors and ERA rankings of medical radiation journals.

Notes: Numbers in parentheses represent the number of journals with both an ERA ranking and impact factor by the total.

Abbreviation: ERA, Excellence in Research for Australia.

Nutrition

We identified 38 journals in this area, where there was clear congruence between impact factors and ERA rankings (See Table 5 and Figure 3). In this discipline, we found 16 journals with impact factors but no ERA ranking. For those journals with both ERA and an impact factor, apart from considerable overlap between the impact factors for the A* and A journals, the impact factors generally reflected the ERA rank. There was a significant negative trend in impact factors and each ERA ranking, with the median A* ERA journal impact factor being 4.3 (3.1 [25th percentile] to 6.3 [75th percentile]). For the A journals it was 3.1 (3.1 [25th percentile] to 3.7 [75th percentile]), for the B ranked journals was 2.6 (2.4 [25th percentile] to 2.6 [75th percentile]), and for the C journals was 1.4 (1.1 [25th percentile] to 1.7 [75th percentile]). For the journals without ERA rankings, the median impact factor was 1.5 (0.5 [25th percentile] to 2.2 [75th percentile]), suggesting that these journals should have mostly attracted C rankings (Figure 3).

Table 5

A comparison of the ERA ratings and impact factors for nutrition journals

ERA rankings	Impact factor	Journal title
A*	8.783	Annual Review of Nutrition
A*	6.307	American Journal of Clinical Nutrition
A*	4.288	Journal of Nutritional Biochemistry
A*	3.146	Food Chemistry
A	4.343	International Journal of Obesity
A	3.725	Critical Reviews in Food Science and Nutrition
A	3.128	Journal of the American Dietetic Association
A	3.072	European Journal of Clinical Nutrition
A	2.797	International Journal of Eating Disorders
B	2.866	European Journal of Nutrition
B	2.627	International Journal of Behavioral Nutrition and Physical Activity
B	2.582	Appetite
B	2.362	Journal of the American College of Nutrition
B	1.356	Journal of Nutrition Education and Behavior
C	1.97	Annals of Nutrition and Metabolism: European journal of nutrition, metabolic diseases and dietetics
C	1.921	Journal of Human Nutrition and Dietetics
C	1.588	Food and Nutrition Bulletin
C	1.39	Journal of Medicinal Food
C	1.313	International Journal of Food Sciences and Nutrition
C	1.214	Asia Pacific Journal of Clinical Nutrition
C	0.897	International Journal for Vitamin and Nutrition Research
C	0.256	Current Topics in Nutraceutical Research
	4.291	Current Opinion in Clinical Nutrition and Metabolic Care
	4.091	Journal of Nutrition
	3.446	British Journal of Nutrition
	3.274	Clinical Nutrition
	1.831	European Journal of Lipid Science and Technology
	1.712	Journal of Nutrition, Health and Aging
	1.672	Journal Of Clinical Biochemistry And Nutrition
	1.606	Food Policy
	1.324	Food Reviews International
	1.229	International Journal Of Sport Nutrition And Exercise Metabolism
	1.112	Genes and Nutrition
	0.526	Food And Drug Law Journal
	0.481	Archivos Latinoamericanos De Nutricion
	0.278	Canadian Journal Of Dietetic Practice And Research
	0.184	Ecology Of Food And Nutrition
	0.164	Ernahrungs-Umschau

Note: Journal Citation Reports 2009 Edition was used.

Abbreviation: ERA, Excellence in Research for Australia.

Figure 3

Impact factors and ERA rankings of nutrition journals.

Notes: Numbers in parentheses represent the number of journals with both an ERA ranking and impact factor by the total.

Abbreviation: ERA, Excellence in Research for Australia.

Speech pathology

We identified 48 speech pathology/language and communication journals, all with ERA rankings. In total, 100% of the A* journals, 83% of the A journals, 86% of the B journals, and 64% of the C journals had impact factors (see Table 6 and Figure 4). There was considerable overlap of impact factors in the different ERA categories, and there were no significant differences in ranked impact factors between the ERA categories. The median A* ERA journal impact factor was 1.3 (1.1 [25th percentile] to 2.3 [75th percentile]), for the A journals was 1.1 (0.7 [25th percentile] to 1.6 [75th percentile]), for the B ranked journals was 1.3 (0.7 [25th percentile] to 2.3 [75th percentile]), and for the C journals was 0.9 (0.6 [25th percentile] to 1.3 [75th percentile]).

Table 6

A comparison of the ERA ratings and impact factors for speech pathology journals

ERA rankings	Impact factor	Journal title
A*	3.221	Journal of Memory and Language
A*	1.323	Studies in Second Language Acquisition
A*	0.968	The Linguistic Review
A	2.973	Brain and Language
A	2.188	Journal of Fluency Disorders
A	1.726	Memory and Cognition
A	1.273	Language Speech and Hearing Services in Schools
A	1.196	Speech Communication
A	1.034	Computer Speech and Language
A	0.791	Language and Communication
A	0.635	Text and Talk: an interdisciplinary journal of language, discourse and communication studies
A	0.615	Language Sciences
A	0.561	Linguistics: an interdisciplinary journal of the language sciences
A		International Journal of Speech-Language Pathology
A		Linguistic Analysis
B	2.531	Language Learning and Technology
B	2.391	Journal of Head Trauma Rehabilitation
B	2.347	Journal of Speech, Language, and Hearing Research
B	2.283	Head and Neck
B	1.879	American Journal of Speech-Language Pathology
B	1.587	Journal of Voice
B	1.33	International Journal of Language and Communication Disorders
B	0.831	Aphasiology
B	0.8	American Speech: a quarterly of linguistic usage
B	0.73	Language and Speech
B	0.73	Language and Speech
B	0.688	Discourse and Communication
B	0.34	Topics in Language Disorders
B		Acquiring Knowledge in Speech, Language and Hearing
B		Language Learning and Development
C	1.639	Journal of Communication Disorders
C	1.577	Dysphagia
C	1.016	American Journal of Otolaryngology – Head and Neck Medicine and Surgery
C	0.851	Journal of Laryngology and Otology
C	0.75	International Journal of Speech, Language and the Law
C	0.424	Ear, Nose and Throat Journal
C	0.395	Quarterly Journal of Speech
C		Communication Disorders Quarterly
C		Contemporary Issues in Communication Science and Disorders
C		Journal of Medical Speech-Language Pathology
C		Seminars in Speech and Language

Note: Journal Citation Reports 2009 Edition was used.

Abbreviation: ERA, Australian Excellence in Research.

Figure 4

Impact factors and ERA rankings of speech pathology journals.

Notes: Numbers in parentheses represent the number of journals with both an ERA ranking and impact factor by the total.

Abbreviation: ERA, Excellence in Research for Australia.

Social work

We identified 143 social work journals, of which all had ERA rankings. All the A* journals, 77.8% of the A journals, 58.9% of the B journals, and 21% of the C journals had impact factors. Table 7 and Figure 5 outline the associations between ERA rankings and impact factors. As anticipated from the data distribution, there was no association between ERA ranking and impact factor, and no significant difference in ranked impact factors between the ERA categories. The impact factor of the only A* journal was 1.1, the median impact factor of the A journals was 0.7 (0.5 [25th percentile] to 1.0 [75th percentile]), for the B journals was 1.0 (0.4 [25th percentile] to 1.2 [75th percentile]), and for the C journals was 0.6 (0.4 [25th percentile] to 1.1 [75th percentile]).

Table 7

A list of the social work journals, their ERA rankings, and impact factors

ERA rankings	Impact factor	Journal title
A*	1.139	The British Journal of Social Work
A	2.162	American Journal of Community Psychology
A	1.682	Journal of Research on Adolescence
A	1.034	Journal of Social Policy
A	1.032	Ageing and Society
A	0.952	Child and Family Social Work
A	0.762	Disability and Society
A	0.683	Social Choice and Welfare
A	0.675	Social Service Review
A	0.603	Journal of Family Therapy
A	0.494	International Journal of Social Welfare
A	0.478	Social Work in Health Care
A	0.458	International Social Work
A	0.426	Affilia: journal of women and social work
A	0.294	Journal of Social Work Practice
A		Australian Journal of Family Law
A		Australian Social Work
A		Disability Studies Quarterly
A		Journal of Aging and Social Policy: a journal devoted to aging and social policy
A		New Directions for Evaluation
A		Social Policy and Society
B	2.613	Child Maltreatment
B	2.537	Trauma Violence and Abuse
B	2.339	Child Abuse and Neglect
B	1.397	Social Work
B	1.275	Family Process
B	1.152	Journal of Behavioral Health Services and Research
B	1.101	Health and Social Care in the Community
B	1.048	Journal of Community Psychology
B	1.039	Research on Social Work Practice
B	0.892	Journal of Marital and Family Therapy
B	0.857	Social Policy and Administration
B	0.745	Journal of Social Work Education
B	0.667	Child Welfare
B	0.512	Administration in Social Work
B	0.341	Revue Internationale de Securite Sociale
B	0.317	Families in Society: the journal of contemporary social services
B	0.235	Journal of Family Studies
B	0.167	Australian Journal of Guidance and Counselling
B	0.056	Asia Pacific Journal of Social work and Development
B		Australasian Dispute Resolution Journal
B		Bereavement Care: an international journal for those who help bereaved people
B		Child Care in Practice
B		China Social Welfare
B		Hong Kong Journal of Social Work
B		International Social Security Review
B		Journal of Immigrant and Refugee Studies
B		Journal of Poverty
B		Journal of Social Work
B		Journal of Social Work Values and Ethics
B		Journal of Sociology and Social Welfare
B		Journal of Teaching in Social Work: innovations in instruction, training and educational practice
B		Practice
B		Practice Reflexions
B		Psychoanalytic Social Work
B		Qualitative Social Work: research and practice
B		Social Work Research
C	1.318	Family Relations
C	1.166	Children and Youth Services Review
C	1.127	Aging and Mental Health
C	1.043	Journal of Intellectual and Developmental Disability
C	0.85	Health and Social Work
C	0.581	Clinical Social Work Journal
C	0.455	Journal of Social Service Research
C	0.396	British Journal of Guidance and Counselling
C	0.37	Childhood: a global journal of child research
C	0.337	American Journal of Family Therapy
C	0.097	Smith College Studies in Social Work
C		Advances in Social Work
C		Advances in Social Work and Welfare Education
C		Amsterdams Sociologisch Tijdshrift
C		Aotearoa New Zealand Social Work Review
C		Australian and New Zealand Journal of Family Therapy: innovative and contextual approaches to human problems
C		Canadian Journal of Counselling
C		Canadian Journal of Program Evaluation
C		Canadian Social Work
C		Child Abuse Review
C		Child and Adolescent Social Work Journal
C		Children Australia
C		Communities, Children and Families Australia
C		Critical Social Work
C		Developing Practice: the child youth and family work journal
C		Ethics and Social Welfare
C		European Journal of Psychotherapy, Counselling and Health
C		European Journal of Social Work: the forum for the social work professional
C		Grief Matters
C		Groupwork
C		Indian Journal of Social Work
C		International Journal of Diversity in Organisations, Communities and Nations
C		International Journal of Narrative Therapy and Community Work
C		International Journal of Qualitative Methods
C		Journal of Baccalaureate Social Work
C		Journal of Children and Poverty
C		Journal of Comparative Social Welfare
C		Journal of Ethnic and Cultural Diversity in Social Work: innovations in theory, research and practice
C		Journal of Evidence-Based Social Work: advances in practice, programming, research, and policy
C		Journal of Family Social Work
C		Journal of Feminist Family Therapy
C		Journal of Gay and Lesbian Social Services: issues in practice, policy and research
C		Journal of Gerontological Social Work
C		Journal of Law and Social Work
C		Journal of Policy Practice
C		Journal of Practice Teaching in Health and Social Work
C		Journal of Progressive Human Services: successor to catalyst: a socialist journal of the social services
C		Journal of Religion and Spirituality in Social Work: social thought
C		Journal of Social Work in Disability and Rehabilitation
C		Journal of Social Work in End-of-Life and Palliative Care
C		Journal of Vocational Rehabilitation
C		Peace Review: a journal of social justice
C		Rural Social Work and Community Practice
C		School Social Work Journal
C		Scottish Youth Issues Journal
C		Social Development Issues: alternative approaches to global human needs
C		Social Security: journal of welfare and social security studies
C		Social Work Abstracts
C		Social Work and Christianity: an international journal
C		Social Work and Social Sciences Review: an internal journal of applied research
C		Social Work and Society
C		Social Work Forum
C		Social Work in Mental Health: the journal of behavioral and psychiatric social work
C		Social Work in Public Health
C		Social Work with Groups: a journal of community and clinical practice
C		Social Work: a professional journal for the social worker
C		Socialno Delo
C		The Australian Journal of Rehabilitation Counselling
C		The Clinical Supervisor: the journal of supervision in psychotherapy and mental health
C		Voluntas: International Journal of Voluntary and Nonprofit Organizations
C		Youth and Policy: the journal of critical analysis
C		Youth Studies Ireland

Note: Journal Citation Reports 2009 Edition was used.

Abbreviation: ERA, Australian Excellence in Research.

Figure 5

Comparison of impact factors and ERA rankings of social work journals.

Note: Numbers in parentheses represent the number of journals with both metrics of journal evaluation by the total.

Abbreviation: ERA, Excellence in Research for Australia.

Summary of results

Table 8 provides a summary of the analysis, and highlights the disparity within six selected allied health areas with respect to ERA rankings and impact factors. Table 8 reports the raw numbers of journals in each discipline, in each ERA category, the number of journals with impact factors, and the median impact factor for each category. Significant differences in impact factors are also reported as P-values. In five of the six categories, there were a large number of journals in which allied health research was published, under the appropriate field of research codes. Significant differences and trends in decreasing impact factors in the decreasing ERA categories were found only for therapies, medical radiation, and nutrition. There were insufficient numbers of journals with impact factors to make these calculations in the evidence-based practice area, and there were no differences in impact factors across ERA categories for speech pathology or social work journals. There were 16 journals (42% of total journals) in nutrition without any ERA rank, although their impact factors suggest an equivalency with ERA rank C.

Table 8

Summary of ERA categories, median impact factors, and significant differences

Discipline		Total	A*	A	B	C	No ERA rank	P value
Evidence-based practice	N	13	0	1	6	6
	N journals with impact factor	2		1	1	0
	Median IF score			2.1	1.9	NA		NA
Therapies	N	145	3	8	33	101
	N journals with impact factor	76	3	8	32	33
	Median IF score		3.4	2.6	2.3	1.3		<0.05
Medical radiation	N	52	1	6	18	27
	N journals with impact factor	36	1	6	17	12
	Median IF score			4.7	2.6	1.3		<0.05
Nutrition	N	38	4	5	5	8	16
	N journals with impact factor	22	4	5	5	8	16
	Median IF score		4.3	3.1	2.6	1.4	1.5	<0.05
Speech pathology	N	48	3	12	15	11
	N journals with impact factor	33	3	10	13	7
	Median IF score		1.3	1.1	1.3	0.9		>0.05
Social work	N	143	1	27	39	76
	N journals with impact factor	60	1	21	22	16
	Median IF score		1.1	0.7	1.0	0.6		>0.05

Note: Journal Citation Reports 2009 Edition was used.

Abbreviations: ERA, Australian Excellence in Research; IF, impact factor.

Discussion

This paper highlights the difficulties for allied health academics in establishing research excellence via publication in Australia. The limitations of this research are its focus on selected fields of research, only five clinical areas of allied health, and one general area (evidence-based practice). Other allied health researchers may have selected different journals for inclusion in the list or different fields of research. However, 439 journals were considered, the liaison librarian provided an independent view of the validity of journal inclusion, and the pattern of impact factor disjoint with ERA ranking was relatively consistent. The strengths of this research are that it is the first paper we know of that has raised the issue of the need for a collaborative allied health voice to consider publication metrics appropriately to overcome the small clinical and research allied health workforce and the allied health focus on knowledge translation. Our research highlights how little is known about who is publishing in allied health journals, why these journals are chosen, who is reading these publications, and what difference the research is making to clinical practice decisions.

We have previously reported the relatively small clinical workforce in allied health disciplines in Australia compared with medicine and nursing,8 as well as the even smaller number of Australian allied health academics who are publishing their research.19 Given these workforce constraints, the question arising from the ERA ranking process seems to be: Should Australian allied health researchers’ focus be on lobbying the Australian Research Council2 for improved rankings of individual allied health journals, should their focus be on allied health journals as a collective, or should questions be asked about how to measure clinical impact and knowledge translation as elements of publication metrics?

Despite finding over 400 journals in which selected allied health disciplines might publish (either in discipline-specific research or multidisciplinary research), there are potentially many hundreds more journals, had we considered other allied health research areas.3,4 The relationship we report between ERA rankings and impact factors in allied health generally suggests that the higher the ERA ranking, the higher the academic impact of the journal is likely to be. However, not every journal had an impact factor and an ERA ranking, and the expected negative linear relationship between ERA categories and impact factors did not occur in every allied health discipline (eg, speech pathology and social work). This suggests that more negotiation is required to rank allied health journals appropriately in terms of their visibility, research quality, impact, and clinical importance for the 2012 round of the ERA process.

It is unknown what the impact of any allied health journal is on clinical practice, because there has been no research in this area. Therefore, there is no current metric to estimate impact in a clinical environment or to measure changed behaviors. With the ERA focus being on international benchmarking and competitiveness, it seems essential for allied health to estimate the extent of the clinical readership of allied health journals, and to measure the difference the current evidence-base makes to translation of knowledge into clinical practice.8–16,19

Effective evidence (knowledge) translation is the key to improved health care processes and outcomes.9–12 It has been reported that translational (applied) research does not necessarily lead to multiple citations, because the clinicians who are digesting the research may not be researching or publishing themselves.9–12,19 We contend that because of its clinical origins, allied health research aims to stimulate clinical change first and research directions second, whereas basic and bench research is about stimulating the next wave of preclinical research. Therefore, allied health clinicians may never cite a publication themselves, even though it may have had considerable impact on their practice, because so few are likely to publish. Therefore, the current publication metrics may not be the best way to promulgate information on “true” impact in allied health, ie, knowledge translation, improved health care practices, safety, and health outcomes.2,9–12 With the present allied health ERA metrics, it does not seem possible to achieve both academic prestige and a measure of knowledge translation, because the low citation rates of allied health journals reported in this paper are often the most widely read clinical allied health journals. This situation is in direct contrast with journals commonly read by general practitioners (ie, Medical Journal of Australia, and Australian Family Physician), which both attract ERA A rankings.2

Conclusion

We provide three key messages from this research. The first is that there are inconsistent relationships between impact factors and ERA rankings in five selected allied health disciplines, and that many ERA rankings have been made in the absence of much information about journal size, readership, or likely impact. The second message is that, to assign the 2012 ERA rankings to journals appropriately, allied health disciplines need to present a united voice, which does not just lobby for individual research areas or discipline journals. This voice could comprise leading allied health researchers and clinicians who have an understanding of publication metrics, allied health tasks and roles, knowledge translation, and clinical uptake of evidence. This group could use the results presented in this paper as a starting point to identify discipline-specific allied health professional journals and key multidisciplinary journals which might publish allied health research and influence clinicians. This group could then lobby the Australian Research Council to address the incongruity between impact factors and ERA rankings, and to promote the journals which publish good quality papers and are likely to influence allied health clinical practice. The third message is that it is essential that publication and research excellence metrics go beyond academic impact, and include measures of translational research. Only by doing this will the true impact of allied health research be measured. Therefore, measures of knowledge translation are urgently required, and should be components additional to the current metric calculations.

We agree that excellence in allied health research must be a goal, and that it must be measurable. However, any metric should include the relevance of the research to health care, and include knowledge of who uses the research, why, and how. Users may not just be other researchers, and therefore measures of knowledge translation and clinical impact are essential in comprehensive and high quality publication metrics, to integrate research and clinical imperatives.