An evaluation of the content of hematology and medical oncology fellowship websites.

Introduction: The growing demand for Hematology and Oncology services has greatly piqued the interest of potential residents towards this specialty. Since the programs' official websites are now becoming the primary source of information that potential residents turn to, we aimed to analyze program websites' content and availability across parameters that have been used by evaluators of websites.
Methods: & Materials: A list of 181 fellowship programs were identified using The Fellowship and Residency Electronic and Interactive Database (FRIEDA). 160/181 were accessed via a hyperlink or Google search. Content of these websites was evaluated on a 40-point criteria system in 10 distinct domains. Websites without accessible links were excluded from the search.
Results: The 160 programs were divided based on the region with the North-East having the most programs (32.5%) and the West having the least programs (12.4%). Exactly 3/4th of the websites had been updated with the latest available information. "Program overview" (89%) was the most common domain present on the websites while "Alumni" was the least common, present on only (25%) of the websites.
Conclusion: When compared with previous similar research, there have been a few significant improvements across the programs' websites, however many still lack important information regarding certain domains. The content and availability of the program's website can encourage or deter an applicant, in their decision to apply to the program, hence making it necessary for programs to augment their websites.

Introduction

Fellowship training in a specific subspecialty of interest has always been considered crucial among the stages of post-graduate medical education to develop advanced clinical skills and subsequent career development [1]. Furthermore, applicants who pursue fellowships in various medical subspecialties are given higher priority for employment [2]. Hematology and medical oncology, comprising of 172 ACGME-accredited fellowship programs, is amongst the most competitive subspecialties with more than 700 applicants per year [[3], [4], [5]]. However, the growing demand for hematology and medical oncology services will soon outpace the addition of new graduates entering the workforce, according to a study by the American Society of Clinical Oncology (ASCO) [6]. This increasing demand is largely driven by the aging and subsequent retirement of the oncology workforce in rising numbers [7]. Moreover, the improvement in cancer survival rates further intensifies this situation [7].

The significance of websites for residency and fellowship applications in different fields has previously been highlighted by multiple studies [[8], [9], [10]]. Since the emergence of the COVID-19 pandemic, residencies and fellowship programs are adapting to the virtual model for the selection and hiring of prospective applicants through online web-based interviews. Furthermore, the preponderance of fellowship programs prefers processing applications through ERAS (Electronic Residency Application Service) and thus further emphasizing increased web usage to search for prospective fellowship opportunities [11]. Trainees assign great value to a residency or fellowship program's website as a primary source for information about application requirements, deadlines, clinical training, didactics, benefits, and research opportunities [12]. The content, quality, and quantity of information accessible on program websites play a pivotal role in potentially influencing applicants' decision [13].

Studies in several specialties have highlighted the difficulties in accessing the information on the internet about fellowship programs [14]. Published literature suggests many medical discipline websites provide insufficient online information on key aspects of the program [[15], [16], [17]]. A previous study was conducted by Ruddell et al., in 2019 for hematology and medical oncology, however, there has been an increase in the number of fellowship programs positions in hematology and medical oncology from 549 to 638 positions, so we decided to gauge whether the increase brought an improvement in content and accessibility of fellowship programs websites [29,30]. Also, we added programs from Canada into our study to reflect hematology and medical oncology programs in North America. We decided to conduct our study with the following aims:

Analyze and assess the accessibility, content, quality, and comprehensiveness of information presented on the hematology and medical oncology fellowship program websites based on a 40-point criteria.

To evaluate the different characteristics of hematology and medical oncology programs of the United States and Canada and study differences in it based on rankings and their geographical locations.

Methods

All information utilized for this study was extracted from publicly available online sources, hence the study design and methodology did not require institutional review board (IRB) approval. Keywords such as ‘hematology and medical oncology and ‘Fellowship’ were inserted onto The Fellowship and Residency Electronic Interactive Database Access (FRIEDA) and programs based in the US were extracted for assessment [18]. Canadian programs were obtained by a simple Google® search using the keywords ‘Hematology’, ‘Medical Oncology’, ‘Fellowship programs’, and ‘Canada’. All programs that did not have a working link to the website were removed. Program websites were accessed through the link provided by FRIEDA or through a simple Google® search using a combination of keywords such as ‘Program name’ ‘hematology and medical oncology, ‘fellowship’, etc. Only the first page of search results was viewed since potential applicants were not likely to look for results beyond the first page [28]. Cookies were disabled and all sponsored links were excluded from the search. The U.S. News and World Report evaluated 899 hospitals creating a cancer ranking list depending on their eligibility criteria, we selected the top 25 programs from this ranking that matched our obtained program list as “ranked” in our data collection [26].

The program websites were reviewed by the authors and evaluation was started from general information such as region and availability of overall information. US regions were divided into regions as designated by the US census bureau [27]. Programs were further assessed for specific content in 10 domains as stated below.

Domains for website evaluation

Program Overview

Fellowship Program Application

Fellowship Program Curriculum

Current Fellows

Alumni

Faculty

Research

Benefits and Incentives

Residential/Housing

Website Updates

The domains were further sub-divided into a 40-point criterion. These criteria were chosen based on previous similar searches, such as those established and used by Niesen et al. and Daniel et al., respectively [21,25]. All data collected was fed into a Microsoft Excel® sheet. Websites were also examined to determine if they had been updated by fulfilling two or greater variables on a 4-point update criterion. The variables were as follows: ‘2019–2020 fellows enlisted’, ‘2020 copyright enlisted’, ‘2020 application deadline’, and ‘2020 stipend information’. Each variable that was present on the website, or through a direct link from the website, was given a positive score (1) or a zero (0) if the variable was not present. The total score for each program was then calculated out of 40 based on 39 standard data points plus 1 from the update domain.

All the material collected was analyzed through Statistical Package for the Social Sciences (SPSS), a data analysis platform by IBM. Programs were also divided and analyzed based on regions Northeast, Midwest, South, West, and the mean score for each region of the US was calculated. The data were examined to see the mean percentage of fulfillment of each variable across all the programs. The percentage fulfillment for all variables for each domain was calculated and the mean percentage for each domain was assumed as the domain's score. Mann-Whitney U test was employed to statistically compare the mean score between ranked and unranked programs. The Kruskal-Wallis test for comparing the mean score between the different regions of the US. P-value of >0.05 was considered significant in all statistical analyses.

Results

Analysis of Fellowship Program Characteristics:

A total of 184 hematology and medical oncology programs were identified using the FRIEDA directory and a Google® search, out of which 3 programs were excluded due to being ‘military’ or ‘other’ programs as designated on FRIEDA. Out of the remaining 181 programs, 160 programs (88.4%) had accessible websites through hyperlinks provided in the FRIEDA directory or Google® search while the remaining 21 programs (11.6%) were inaccessible. 2 of these 21 inaccessible programs were due to the website being in French and unable to be translated. The North-East (32.5%) and South (32%) regions had the greatest number of programs in our study, while the West (12.4%) had the least in number. Table 1 details the total number of program websites, according to region and state, that were accessed for analysis.

Table 1

Breakdown of the Total Number of Hematology and Medical Oncology Programs in each Region and State.

Region	State	Total Number of Programs (n = 181)
North-East		55
	Connecticut	2
	Massachusetts	7
	New Hampshire	1
	New Jersey	7
	New York	25
	Pennsylvania	11
	Rhode Island	2
South		54
	Alabama	1
	Arkansas	1
	District of Columbia	3
	Florida	10
	Georgia	2
	Kansas	1
	Kentucky	2
	Louisiana	4
	Maryland	2
	Mississippi	1
	North Carolina	5
	Oklahoma	1
	Puerto Rico	2
	South Carolina	1
	Tennessee	2
	Texas	10
	Virginia	3
	Vermont	1
	West Virginia	2
Mid-West		39
	Ohio	7
	Iowa	2
	Illinois	9
	Michigan	10
	Minnesota	2
	Missouri	4
	Nebraska	1
	North Dakota	1
	Wisconsin	3
West		21
	Arizona	2
	California	14
	Colorado	1
	New Mexico	1
	Oregon	1
	Utah	1
	Washington	1
Canada		12

Analysis of Fellowship Website 40-Point Criteria:

The most common domain present in the program websites was “Program overview” at 89% and the least common domain present in the program websites was “Alumni” at 25%. 113 programs (75%) had their websites updated. Analysis of the ten reviewed domains is described in depth below with Table 2 detailing individual domains and variables. Fig. 1 highlights the overall satisfaction in separate domains.

Table 2

40-Point criteria breakdown denoting the number of fellowship programs fulfilling each criteria.

	Fellowship Website Criteria	Number of Programs Providing Information (n = 160)	Percentage of Programs Providing Information
1.	Program Overview
	Mission Statement/Introductory Message	158	99%
	Fellowship Program Director Profile	144	90%
	Fellowship Program Director Contact Information	124	78%
2.	Application
	Fellowship Application Requirements	128	80%
	USMLE Step Scores	73	46%
	Number of Fellows Recruited	100	63%
	ERAS Link Application	146	91%
	Application Deadline	89	56%
	Interview Dates Information	82	51%
3	Curriculum
	Didactics	139	87%
	Rotation Schedule	108	68%
	Medicine Grand Rounds	67	42%
	Clinical Rotation Sites	114	71%
	Associations with Societies/Organizations	29	18%
	Board Pass Rate	9	6%
4	Fellows
	Number of Fellows	110	69%
	Name of Fellows	104	65%
	Pictures of Fellows	82	51%
	Fellow's Background Information	86	54%
5	Alumni
	Alumni Names	48	30%
	Alumni Picture	18	11%
	Alumni Job Location	47	29%
6	Faculty
	Faculty Name	135	84%
	Faculty Picture	123	77%
	Faculty Background Information	118	74%
7	Research
	Research Opportunities/Facilities	150	94%
	Research Conference Information	76	48%
	Journal Club	83	52%
8	Benefits and Incentives
	Salary	71	44%
	Meal	43	27%
	Vacations	80	50%
	Educational Funds	66	41%
	Fellow Wellness Program	74	46%
	Insurance	81	51%
	Visa Information	102	64%
9	Residential Information
	Surrounding Area/Neighborhood/City Information	97	61%
	Diversity Inclusion	78	49%
	Social Media	134	84%
	Housing information	57	36%
10	Website Updated	116	73%

Fig. 1

Illustrating the mean percentage of fulfillment in each domain.

Program Overview:

Program overview was provided by 89% of all the program websites with a mission statement/introductory message being the most prevalent information provided by 158 programs (99%) followed by fellowship program director profile which was provided by 144 programs (90%) and lastly fellowship program director contact information provided by 124 programs (78%).

Fellowship Program Application Information:

Fellowship program application information was provided by 65% of all the program websites with ERAS link application being the most prevalent information being provided by 146 programs (91%) followed by fellowship application requirements 128 programs (80%), the number of fellows recruited 100 programs (63%), application deadline 89 programs (56%), interview dates information 82 programs (51%) and lastly, USMLE step scores 73 programs (46%).

Fellowship Program Curriculum:

Fellowship program curriculum was provided by 49% of all the programs with didactics being the most prevalent being provided by 139 programs (87%) followed by rotation schedule 108 programs (68%), medicine grand rounds 67 programs (42%), association with societies/organizations 29 programs (18%) and lastly board pass rate 9 programs (6%).

Current Fellow Information:

Current fellow information was provided by 60% of all the programs with the number of fellows being the most prevalent information provided by 110 programs (69%) followed by the name of the fellows 104 programs (65%), fellow's background information 86 programs (54%) and lastly pictures of fellows 82 programs (51%).

Alumni Information:

Alumni information was provided by 23% of all the programs with alumni names and alumni job location being the most prevalent being provided 48 (30) and 47 (29) programs, respectively, followed by alumni pictures 18 programs (11%).

Faculty Information:

Faculty information was provided by 78% of all the programs with faculty name being the most prevalent being provided by 135 programs (84%) followed by faculty picture 123 programs (77%) and lastly faculty background information 118 programs (74%).

Research Information:

Research information was provided by 65% of all the programs with research opportunities/facilities being the most prevalent being provided 150 programs (94%) followed by journal club 83 programs (52%) and lastly research conference information 76 programs (48%).

Benefits and Incentive Information:

Benefits and incentive information was provided by 46% of all programs with visa information being the most prevalent being provided by 102 programs (64%) followed by insurance 81 programs (51%), vacations 80 programs (50%), fellow wellness program 74 programs (46%), salary 71 programs (44%), educational fund 66 programs (41%) and lastly meal information 43 programs (27%).

Residential/Housing Information:

Residential/housing information was provided by 58% of the programs with the most prevalent information being social media is provided by 134 programs (84%) followed by surrounding area/neighborhood/city information 97 programs (61%), diversity inclusion information 78 programs (49%), and lastly housing information at 57 programs (36%).

Update Information:

The updated domain was analyzed as a whole and was fulfilled by 116 (73%) programs.

Analysis of 40-point criteria with study characteristics:

Comparing the mean score amongst the regions, West had the highest mean number of criteria fulfilled at 26.2, followed by the South at 24.5, Mid-West at 21.6, and lastly North-East at 21.4, also as shown in Fig. 2. A statistical difference was found when comparing the mean 40-point score between different regions (p > 0.05).

Fig. 2

Illustrating the mean number of criteria fulfilled within each region.

Analysis of Top-25 ranked programs:

Analysis of the top 25-ranked programs in hematology and medical oncology was done which revealed a mean score of 24.7 for ranked programs with a standard deviation of ±6.1. Comparing the mean score between the top US 25 ranked and the rest of the unranked programs was found to be significant (p > 0.05). Table 3 presents the findings for ranked programs along with their obtained score from our 40-point criteria.

Table 3

Represents the 40-point criteria score obtained by the top 25 hematology and medical oncology fellowship programs.

Name of Program	Ranking	40-point Criteria Score
University of Texas MD Anderson Cancer Center	1	18
Memorial Sloan Kettering Cancer Center	2	14
Mayo Clinic	3	34
John Hopkins Hospital	4	32
Cleveland Clinic	5	27
Dana-Farber/Brigham and Women's Cancer Center	6	12
Cedars-Sinai Medical Center	7	26
Northwestern Memorial Hospital	8	31
Seattle Cancer Care Alliance/University of Washington Medical Center	9	23
UCSF Medical Center	10	28
Siteman Cancer Center at Barnes-Jewish Hospital	11	28
UCLA Medical Center	12	24
City of Hope Comprehensive Cancer Center	13	21
UPMC Prebysterian Shadyside	13	24
Houston Methodist Hospital	15	22
Massachusetts General Hospital	16	12
New York-Presbyterian Hospital-Columbia and Cornell	17	19
Hospitals of the University of Pennsylvania-Penn Presbyterian	18	35
Mayo Clinic-Phoenix	19	31
University of Chicago Medical Center	20	21
The University of Alabama at Birmingham Hospital	21	17
Dan L Duncan Comprehensive Cancer Center at Baylor St. Luke's Medical Center	22	33
Thomas-Jefferson University Hospitals-Sidney Kimmel Cancer Center	23	20
University of Kentucky Albert B. Chandler Hospital	24	26
Ohio State University James Cancer Hospital	25	25

Discussion

Our study aims to characterize hematology and medical oncology fellowship websites and analyze the information presented on their websites. Foremost, we highlight the concerning fact that out of a total of 169 program websites attempted to access, 19 did not have an accessible website. In the age of the internet where virtually all information about prospective fellowship programs should be accessible online, this is alarming. The most common domain mentioned on websites was “Program Overview” mentioned by 89% of programs whereas alumni information was the least reported domain at a meager 23%. Less than half (46%) of the programs reported information on USMLE steps scores, an instrumental piece of information for applicants. However, this is a big jump from the 0% reported by Ruddell et al. in his 2019 study, signifying how programs have started to understand the importance of this key piece of information [29]. Application information as an entire domain was covered by 65% of the programs, another commendable increase from the 55.9% reported by Ruddell et al. [29] Less than half (49%) of the programs reported information on the program curriculum, one of the most crucial elements to consider for applicants with regards to their training and didactics.

The American Board of Internal Medicine (ABIM) is a physician-led board that certifies physicians in internal medicine and 21 subspecialties including hematology and medical oncology [18]. ABIM reports a board-pass rate of 94% and 93% for first-time exam takers for hematology and medical oncology respectively [19]. Data reported on board pass rate was alarmingly a low reported variable i.e., only 9 (6%) programs reported this data. Board certification is usually mandated for positions in hematology and medical oncology making this an important piece of information for applicants to assess the program's didactic capability and competency. The transparency of this data has barely risen from the 4.2% reported by Ruddell et al. which is a major cause of concern as this tells how fellowship programs have still failed to understand and act on the implication of this data [29].

Research as a domain was covered by numerous programs (65%), however, information regarding journal clubs and research conferences was only provided by half of the programs. Ruddell et al. reported “Research Publications and Activities” at 86%, which is a steep decline in the number of programs providing research information. This is worrying news in today's world, especially in this era of cutting-edge research where everyone is working to find the next big breakthrough in basic science, clinical, and translational research by setting up research conferences, journal clubs, and other forums [29]. Weaver et al. discussed how collaborative research projects have increased at her Family Medicine residency program, we expect a similar trend for all types of programs [28]. Alumni information as a whole domain was only covered by around one-quarter (23%) of the programs which means most applicants are in the dark about program graduates. This value is similar to the 21.7% reported by Ruddell et al. denoting how only a few programs have understood the significance of alumni career paths to prospective residents. Moreover, another domain that was inadequately covered (46%) was benefits and incentive information with very sparse data concerning salary, vacation benefits, and insurance. All this information is vital for applicants as they form their respective rank order lists based on the financial impetus offered by programs. Salary information alone was provided by only 44% of the programs. Most programs accept international medical graduates and many programs had visa information (64%) on their websites.

Faculty information was covered by roughly four-fifth (78%) of all programs which is indicative of programs realizing the importance of staff members to fellowship applicants, a thought proved by Nielsen et al. which reported orthopedic fellowship applicants considering faculty members as one of the most significant factors driving their interest in a program [20]. Moreover, program director contact information was provided by only 78% of the programs and the bar needs to be further raised in this regard as program directors are the face of the program and applicants often use their contact to gain further information about a program and solidify their interest in a program. However, some programs may be uncomfortable providing this information on a public platform due to the possibility of unsolicited emails.

About half of the websites talked about diversity and inclusion on their websites, a topic that has to be dealt with significantly to improve underrepresented groups in medicine [31]. Furthermore, a lot of information mentioned on websites changes yearly, and hence programs must update their website accordingly every year, ideally a few months before the application cycle, to ensure the most recent data is provided to applicants planning for hematology and medical oncology fellowship. In our study, most programs have their websites (73%) updated until 2020 which is a sign of programs being aware of the importance of keeping a website updated.

The top-ranked programs in hematology and medical oncology were found to be deficient in content pertinent to their fellowship training as evidenced by the fact that no single program, out of the top 25, fulfilled the 40-point criteria completely. The paucity of details and relevant information regarding a fellowship program can prove to be a detrimental factor in the recruitment process of an applicant, hence many steps can be taken to bridge this gap present between the hematology and oncology website information and applicant. Perhaps the most straightforward approach would be to survey prospective applicants to get their feedback on how to make fellowship sites more inclusive and applicant oriented [32,33] and this would provide a more holistic approach to gather input from all stakeholders pertinent to medical education within hematology and medical oncology.

Limitations for this study were few but present. The variables in this study were based on previous similar studies and the guidelines provided by the Accreditation Council for Graduate Medical Education (ACGME). Our study was conducted in March 2021 and therefore does not accommodate any changes made to the websites after this period. Programs with inaccessible website links were not contacted to further investigate this issue. Moreover, no department or individual from a program was directly contacted for information regarding their website. Some programs may choose to share a part of their information through other means such as electronic or postal mail; no survey was conducted to include the information provided through these methods.

Conclusion

We hope our study has highlighted some of the areas that need to be addressed on a priority basis such as program accessibility, program curriculum, alumni information, and research opportunities. An integrated and standardized checklist needs to be devised by the American Society of Hematology (ASH) and the American Society of Clinical Oncology for fellowship programs to use as a yardstick when developing their websites.

Lesson for practice

Hematology and Medical Oncology fellowship programs websites are increasingly being used by prospective applicants to search desired content regarding postgraduate training

Only 73% of training programs were found to be updated

A standardized criterion needs to be employed to further optimized fellowship program websites to enhance applicant's virtual experience

Sources of funding

No financial or material support to report.

Ethical approval

Not Needed.

Consent

Not needed.

Author contribution

Muhammad Zain Farooq: conduct, literature search, manuscript preparation, manuscript editing, and manuscript review, final approval, and agreeing to the accuracy of the work. Muhammad Daniyal: conduct, literature search, manuscript preparation, manuscript editing, and manuscript review, final approval, and agreeing to the accuracy of the work. Palvisha Qasim: conduct, data acquisition, data analysis, statistical analysis, manuscript preparation, manuscript editing and manuscript, final approval, and agreeing to the accuracy of the work. Houria Jamshed: planning, conduct, data acquisition, data analysis, statistical analysis, manuscript editing, and manuscript review, final approval, and agreeing to the accuracy of the work. Shehzeen Fatima Memon: planning, conduct, manuscript preparation, manuscript editing and manuscript review, final approval, and agreeing to the accuracy of the work. Yumna Salman: planning, conduct, manuscript preparation, manuscript editing and manuscript review, final approval, and agreeing to the accuracy of the work. Bushra Zafar Sayeed: planning, conduct, manuscript preparation, manuscript editing and manuscript review, final approval, and agreeing to the accuracy of the work.#. Maham Abbasi: conduct, literature search, manuscript preparation, manuscript editing, and manuscript review, final approval, and agreeing to the accuracy of the work. Michael Jaglal (Guarantor): planning, conduct, manuscript preparation, manuscript editing and manuscript review, final approval, and agreeing to the accuracy of the work.

Registration of research studies

Name of the registry: N/A

Unique Identifying number or registration ID: N/A

Hyperlink to your specific registration (must be publicly accessible and will be checked): N/A

Guarantor

Muhammad Zain Farooq; Muhammad Daniyal; Palvisha Qasim, Houria Jamshed; Shehzeen Fatima Memon; Yumna Salman; Bushra Zafar Sayeed; Maham Abbasi; Michael Jaglal.

Declaration of competing interest

None to Report.