A key performance indicators redefinition initiative at a school of pharmacy using a modified Delphi consensus technique.

OBJECTIVE: The Outcomes and Assessment Committee at the Virginia Commonwealth University School of Pharmacy was tasked with refining the school's key performance indicators (KPIs) to improve programmatic assessment by focusing on the most important measures.
METHODS: Initially, 56 KPIs were tracked, nine of which were university mandated, divided into 10 modules: admissions, community outreach, continuing education, diversity, faculty experience and success, fundraising, graduate program, research and scholarship, staff experience and success, and student experience and success. Using a three-round Delphi consensus technique, KPIs were reviewed by faculty and staff. Each participant responded whether they considered each KPI to be essential or not essential for school quality assessment and improvement. Consensus for the first, second, and third rounds was defined as ≥90%, ≥80%, and ≥75% agreement, respectively.
RESULTS: Of 109 faculty and staff invited, 49 participated in the first round, 51 in the second, and 42 in the third. At the end of the third round, accumulated consensus was achieved for 35 out of 88 (39.8%) KPIs that were considered essential and 3 out of 88 (3.4%) that were considered non-essential. Consensus percentage per module was: 15.4% (2/13) admissions, 28.6% (2/7) community outreach, 33.3% (3/9) continuing education, 27.3% (3/11) diversity, 62.5% (5/8) faculty experience and success, 55.6% (5/9) fundraising, 40% (4/10) graduate program, 33.3% (3/9) research and scholarship, 57.1% (4/7) staff experience and success, and 66.7% (4/6) student experience and success.
CONCLUSIONS: Ultimately, 35 KPIs achieved consensus as essential to measure achievement of benchmarks for the school, which totals 44 KPIs, including nine university mandated KPIs. The process facilitated faculty and staff involvement in KPI selection and achieved improved focus for programmatic assessment. Copyright: © Pharmacy Practice and the Authors.

INTRODUCTION

Key performance indicators (KPIs) are quantifiable measures of performance developed based on an organization’s strategic plan that serve as the basis for tracking progress and measuring success in achieving the organization’s goals.1 Continuous performance evaluation is key to understand past events and plan future quality improvement strategies.

In higher education, KPIs focus on key elements of the higher education enterprise including students, research, outreach, effectiveness and efficiency of administrative and educational processes, communications, and educational outcomes.2 Previous authors compiled comprehensive lists of performance measures for higher education in the United States.3-5 However, the use of KPIs is not mandated for colleges and universities in the United States unlike other countries such as the United Kingdom, New Zealand, and Canada.6-9 KPIs allow higher education institutions to not only track their own performance but also to measure their performance against peer institutions.2 KPIs can also provide insight into resource allocation and determine funding decisions by governments.2,10,11

Little research focusing on performance measurement in schools and colleges of pharmacy has been conducted to date.12 It is likely that most of these data are kept for internal use only and not published. Information pertaining to, for example, the North American Pharmacist Licensure Examination (NAPLEX) pass rate, residency matching rates or National Institutes of Health (NIH) funding rank is commonly disseminated on the schools and colleges’ websites. With the rise in the number of pharmacy schools in recent decades, and a decline in the number of applications and enrollment over the past years, it is crucial that pharmacy schools develop robust strategic plans. KPIs emanating from those strategic plans will allow assessment of the organization’s performance and inform necessary changes for continuous quality improvement.13-15

A previous report described the implementation of a continuous quality improvement program focusing on programmatic advancement at a college of pharmacy.12 The authors used the Studer Group’s Hardwiring Excellence approach to guide implementation of an excellence program that would enhance achievement of education, research and service goals, which included development of a strategic plan and performance measurement by means of KPIs.16 Accountability and alignment between the college’s goals and individual faculty, staff and administrators’ performance were ensured by including KPIs specific to each person’s job responsibilities in their annual evaluations.12 A similar management approach was adopted at the Virginia Commonwealth University (VCU) School of Pharmacy (SOP). KPIs were developed for key areas by teams of faculty, presented to the VCU-SOP Executive Committee and remaining faculty for discussion and, ultimately, approved by faculty in May 2015. Feedback from faculty was obtained over the years resulting in an undesirably high number of KPIs. In 2019, the Outcomes and Assessment Committee at the VCU-SOP was tasked with refining the KPIs for the school in conjunction with the Dean. The aim of this study was to identify essential KPIs to be adopted school-wide using a modified Delphi consensus process. Rather than aiming at the generalization of results to other schools and colleges, our goal was to illustrate how a Delphi technique can be used in committee service to promote faculty and staff involvement in KPI selection instead of following a top-down approach wherein school administrators make all decisions. Delphi techniques have been applied in the pharmacy education literature to achieve consensus regarding multiple topics, including: general practice pharmacists’ educational requirements, faculty members’ leadership characteristics, learning objectives of model courses within the pharmacy curriculum, attributes of an organizational culture measure in academic pharmacy, competencies for individuals designated as assessment leads, among others.17-24 Despite having been previously utilized to develop KPIs in clinical areas, the application of the Delphi technique to the refinement of KPIs in higher education is unique in the pharmacy literature.25-30

METHODS

A three-round modified Delphi technique was used to gather consensus about essential KPIs defined as the minimum indispensable KPIs needed to track the VCU-SOP performance. Despite not having provided a definition for what constitutes essential and non-essential KPIs to the panel members, the goal of the process and what we expected to accomplish was explained at multiple school-wide meetings prior to survey launch. The Delphi technique is a consensus method based on the agreement of a panel of experts about a given subject. In the first phase of a traditional Delphi technique, experts contribute information pertaining to the subject area being studied in response to open-ended questions.31,32 In this modified Delphi technique, the initial list of items (KPIs) was provided a priori because those were the existing KPIs adopted to assess the school’s performance. Additionally, participants targeted included all faculty and staff at the VCU-SOP because the goal was to promote all personnel engagement in the KPIs redefinition initiative, rather than to generalize the results beyond our institution. Thus, despite not being subject matter experts (i.e., performance management) as required by best practices in conducting a Delphi consensus technique, our approach ensured that all individuals were given an opportunity to participate.31,32 The final result of the Delphi process served as the basis for internal discussions at the Outcomes and Assessment and Executive Committees levels regarding adoption of KPIs achieving consensus as school wide KPIs. Ultimately, the Executive Committee, with input from the Outcomes and Assessment Committee, adjudicated the final list of KPIs.

The VCU Institutional Review Board determined that this study was not considered human subjects research, thus no review was necessary.

Key performance indicators

The initial list included 56 KPIs, of which nine were mandated by the university, divided into the following nine modules: admissions (n=8), community outreach (n=4), continuing education (n=6), diversity (n=3), faculty experience and success (n=8), fundraising (n=6), graduate program (n=7), research and scholarship (n=5), and student experience and success (n=9). Because no KPIs pertaining to staff experience and success were part of the initial list, a new module including three KPIs similar to current KPIs assessing faculty experience and success was created. KPIs mandated by the university were presented to panelists during the Delphi process so that they were aware of university requirements, but not ranked as essential or not essential given that these are university rather than school wide KPIs that cannot be changed. Thus, 50 KPIs were included in the first Delphi round.

Delphi panel and process

All faculty and staff members were invited to participate in the process via email and participation in all three rounds was encouraged during faculty and staff retreats and at individual department meetings. Qualtrics® (Provo, UT) was used to develop and manage the Delphi process and a link to the list of KPIs was provided in the recruitment email. Participants were asked to select whether they considered each of the KPIs to be ‘essential’ or ‘not essential’ to the school. An option ‘unable to respond’ was also available. During the first and second Delphi rounds, participants were allowed to propose new KPIs or changes to existing ones. A free text box was included at the end of each module to allow documentation of suggestions. New items proposed and comments from the panel members were analyzed by two of the authors (TMS, LMF) and incorporated in subsequent rounds, while ensuring anonymity. Prior to the second and third rounds, and per Delphi procedures, panelists received a report of the overall group ratings as well as their own individual ratings to consider when responding to a new round.31,32 All faculty and staff were invited to participate in the first and second rounds, but only those who completed either of these two rounds were allowed to participate in the third, to maximize the number of participants who responded to at least two rounds. Participation was voluntary and no incentives were provided. The first Delphi round occurred in May 2019, the second in June 2019, and the third in July 2019. Each round was open for two weeks and e-mail reminders were sent one week after deployment of each round and on the closing date to foster participation.

The process, including data analysis, was led by one of the Outcomes and Assessment Committee members who is a tenure-track Assistant Professor with experience with the Delphi technique, assisted by a third-year pharmacy student. Only these two individuals had access to individual responses while the process was ongoing, and the faculty member did not participate in the Delphi panel.

Consensus criteria and data analyses

Consensus was defined as a specified percentage of participants rating KPIs as either essential or not essential. Items achieving consensus in each round were removed from the next round with essential KPIs being retained for strategic planning use and not essential KPIs being eliminated. No pre-defined criteria to establish consensus in Delphi methods exist.32 In this study, consensus criteria were more stringent in the first round (≥90%), and progressively decreased to ≥80% and ≥75% in the second and third rounds, respectively. These cut-offs were defined a priori and selected to ensure that the first and second rounds identified KPIs that the overwhelming majority considered essential. Consensus criteria were more relaxed in the third round to generate more internal discussion within the Outcomes and Assessment and Executive Committees regarding adjudication of KPIs that achieved consensus in this round as school wide KPIs. Percent consensus was calculated for each KPI. The ‘unable to respond’ option was recoded as missing data and not included in the percent consensus calculations. The SPSS version 25 (IBM Corp. Armonk, NY) was used for all analyses.

RESULTS

Of the 109 faculty and staff invited, 62 unique individuals participated in the Delphi process (56.9% response rate). Of these, 49 completed the first round, 51 the second, 42 the third, and 32 completed all three rounds. Of the 49 individuals who completed round 1, 38 also completed round 2. Participants had been working at the VCU-SOP for an average of 12 years, half were female, and over 70% were faculty members (Table 1).

Table 1

Characteristics of the Delphi panel (n=62)

Characteristic	n (%)
Gender
Female	31 (50.0)
Male	26 (41.9)
Prefer not to disclose	5 (8.1)
Rank
Assistant Professor	13 (21.0)
Associate Professor	18 (29.0)
Full Professor	14 (22.6)
Dean/Assistant Dean	1 (1.6)
University and Academic Professional	6 (9.7)
Staff	8 (12.9)
Other	2 (3.2)
Track (among 45 faculty)
Tenure-Track (i.e., research faculty)	8 (17.8)
Tenured	17 (37.8)
Term Faculty (i.e., clinical faculty)	20 (44.4)

Delphi Round 1: Consensus (≥90% agreement) for KPIs considered essential was achieved for 7 of the initial 50 (14%) KPIs assessed (Table 2 and ). Consensus items in this first round pertained to faculty, staff and student satisfaction, scholarly output (i.e., number of peer reviewed journal publications), and funding (i.e., total amount of private funds received as cash gifts annually and NIH funding ranking). No KPIs achieved consensus as being non-essential. By the end of the first round, and after reviewing input from panelists, one KPI was reworded and 26 new KPIs were included. Ultimately, 69 (=50-7+26) items were included in the second consensus round, including the new proposed KPIs and excluding KPIs that achieved consensus during round 1.

Table 2

Consensus achieved for key performance indicators considered both essential and not essential in each module during the three Delphi rounds

Modules	1st Delphi Round (≥90% agreement)		2nd Delphi Round (≥80% agreement)		3rd Delphi Round (≥75% agreement)		No. of essential consensus itemsc
	# valid items	No. consensus items (%)	# valid items	No. consensus items (%)	# valid items	No. consensus items (%)
1. Admissions	7	0	13	2 (15.4)	11	1 (9.1)	2
2. Community outreach	3	0	7	0	7	3 (42.9)	2
3. Contiuning education	6	0	8	1 (12.5)	7	2 (28.6)	3
4. Diversity	2	0	9	2 (22.2)	9	1 (11.1)	3
5. Faculty experience and success	7	1 (14.3)	6	2 (33.3)	5	2 (40.0)	5
6. Staff experience and success	3	1 (33.3)	3	2 (66.7)	4	2 (50.0)	4
7. Fundraising	6	1 (16.7)	8	3 (37.5)	5	1 (20.0)	5
8. Graduate program	7	1 (14.3)	6	1 (16.7)	8	2 (25.0)	4
9. Research and Scholarship	3	2 (66.7)	4	0	7	1 (14.3)	3
10. Student Experience and Success	6	1 (16.7)	5	3 (60.0)	2	0	4
Totala	50	7 (14.0)	69	16 (23.2)	65	15 (23.1)	35
Total accumulated consensusb	14.0% (7/50)		30.3% [(7+16)/76)]		43.2% [(7+16+15)/88)]

Does not include 9 KPIs mandated by the university.

Accumulated consensus is calculated by dividing the sum of KPIs that achieved consensus both as essential and non-essential in each round as the numerator and the sum of the initial number of KPIs and new KPIs suggested at the end of rounds 1 and 2 as the denominator.

The last column presents consensus achieved for items considered essential.

Click here for additional data file.

Delphi Round 2: Consensus (≥80% agreement) was achieved for 16 of the 69 (23.2%) KPIs, 15 of which were considered essential and one non-essential (Table 2 and Online appendix), representing an accumulated consensus of 23 (=7+16) out of 76 (=50+26) KPIs, i.e., 30.3%. None of the KPIs in the community outreach and research and scholarship modules achieved consensus in this round. Twelve new KPIs were suggested at the end of round 2 to be included in round 3. After excluding items that achieved consensus and adding proposed KPIs during round 2, 65 (=69-16+12) KPIs were included in the third Delphi round.

Delphi Round 3: Consensus (≥75% agreement) in the last round was achieved for 15 (23.1%) of the 65 KPIs, 13 considered essential and 2 non-essential (Table 2 and Online appendix), representing an accumulated consensus of 38 (=7+16+15) out of 88 (=50+26+12) KPIs, i.e., 43.2%. Ultimately, accumulated consensus was achieved for 35 out of 88 (39.8%) KPIs that were considered essential and 3 out of 88 (3.4%) that were considered non-essential.

DISCUSSION

The purpose of this report is to describe the process used to reassess the KPIs tracked by the school in order to reduce their number and improve programmatic assessment by focusing on the most important indicators. Burke and Minassians recommend that colleges or universities adopt no fewer than 10 and no more than 25 KPIs.33 Their research showed that a greater number of KPIs made it more cumbersome for institutions to track performance.33 The Delphi process proved effective, allowing a reduction in the number of KPIs from 56 to 44 (including 9 mandatory KPIs), while engaging faculty and staff in the process, and efficient given that no additional meetings or discussions were required to attain consensus. The Outcomes and Assessment and Executive Committees reviewed the KPIs that achieved consensus as essential and further reduced the number from 44 to 36 (including 9 mandatory KPIs). Despite being above the recommended upper limit, our initiative was successful in reducing the number of KPIs.33 KPIs that did not achieve consensus as either essential or not essential may continue to be monitored by departments or units per their needs.

As expected, a majority of participants (slightly over 70%) were faculty members. We did not have an a priori staff involvement goal but consider that having almost 30% participants being a mix of staff, university and academic professionals, and others is indicative of success. Providing all personnel with an equitable opportunity to participate in the process was a priority to the VCU-SOP administration. The panel size in our study was larger than the recommended number of experts in a Delphi study, between 15 and 30.31 Research indicated that Delphi panels of more than 30 experts do not improve results.34,35

Three rounds, as opposed to the traditional four, were used given that previous evidence showed that three rounds were sufficient to achieve consensus, with most changes to items occurring between the first and second rounds.34,35 The three-round Delphi technique is a common approach in the pharmacy literature and elsewhere, but other approaches including two rounds have also been adopted.18,22,26,28,36-38 We deliberately used high consensus criteria (as high as ≥90% in the first round and ≥80% in the second) to identify essential KPIs. Previous studies adopted a consensus threshold of ≥80%, or ≥75% throughout all rounds.18,26,38 Our study used varying consensus thresholds throughout the three rounds similar to previous research, with the goal to differentiate between KPIs for which consensus was obvious, and those that would generate more internal discussion regarding their inclusion for performance measurement purposes.17,37,39 This approach allowed us to clearly identify student satisfaction, scholarly output, and funding as the most critical KPIs to evaluate the school’s performance in round 1. These areas, in addition to the graduate program, were those for which more KPIs achieved consensus as essential at the end of round 3. These areas are consistent with KPIs described by Arif and Smiley.40 No specific trends were noted in KPIs achieving consensus in the second and third rounds (Online appendix).

Different rating scales have been utilized to assess consensus in Delphi studies. Some used Likert scales with varying formats (4-, 5-, 6- or 9-point), while others used dichotomous scales.17-19,26,36,38,39 When using Likert-type scales, consensus is often assessed by collapsing responses to the two most extreme options of agreement, technically rendering the scale dichotomous (i.e., agree or strongly agree versus everything else). In our study, we opted for a dichotomous option to obtain unequivocal decisions from participants regarding the essentiality of each KPI. We included an ‘unable to respond’ option to allow participants who were not familiar with certain aspects of the organization to avoid weighing in on the decision to keep or eliminate a KPI. This decision was left to the discretion of each participant rather than imposed a priori based on their role within the school. Other schools following a similar approach may choose to target individuals who they consider experts in a certain area to rate specific KPIs, which would result in different sets of KPIs being sent out to different groups of individuals. In our study, the number of individuals selecting the ‘unable to respond’ option ranged between 2-19 in round 1, 2-18 in round 2, and 1-12 in round 3 and did not negatively affect the recommended number of panelists in a Delphi process (between 15 and 30) in either round.31

Once the Delphi process was completed, the Outcomes and Assessment Committee recommended removal of 4 of the 35 KPIs that achieved consensus (excluding 9 mandatory KPIs) because they were either unable to be accurately measured, were already captured by other KPIs, or were not considered essential as a school wide KPI. Of note, all the KPIs removed had achieved consensus in the third round. The Executive Committee accepted these recommendations and proposed further removing two KPIs that also achieved consensus in the third round, combining two KPIs pertaining to fundraising into one, and rewording another to improve clarity (Table 3). KPIs approved by the Executive Committee were presented to and reviewed by the school’s National Advisory Committee, which includes alumni and other stakeholders, as well as University administration.

Table 3

Final list of key performance indicators tracked by the VCU School of Pharmacy as well as data source and responsible party

KPI	Data Source	Responsible party
Module 1: Admissions
Total number of PharmD applications*	Admissions Data	Associate Dean for Student Services
Mean cumulative undergraduate GPA of entering class	Admissions Data	Associate Dean for Student Services
Percentage of entering class that identifies as an underrepresented minority	Admissions Data	Associate Dean for Student Services
Module 2: Community Outreach
Number of formal partnerships (local, national, international)*	University	Division of Community Engagement
Module 3: Continuing Education
Number of continuing education programs offered	Continuing Professional Education Director	Executive Associate Dean for Academic Affairs
Average satisfaction score of all continuing education programs	Continuing Professional Education Director	Executive Associate Dean for Academic Affairs
Module 4: Diversity
Percentage of underrepresented minority PharmD students, Master and PhD students, faculty members, and staff members (includes Black/African American, Hispanic/LatinX, American Indian)*	Office of Admissions and Student Services, Human Resources data, Graduate Program Office	Associate Dean for Student Services, Human Resources Administrator, Graduate Program Director
Percentage of overall faculty, students and staff who feel the School of Pharmacy is welcoming to those with diverse backgrounds	Students: AACP Graduating Student Survey Faculty & Staff: School of Pharmacy Diversity Climate Survey	Director of Education and Assessment
Percentage of underrepresented minority and women in School leadership positions (deans, chairs, vice chairs, directors)	Human Resources Data	Dean
Module 5: Faculty Experience and Success
Faculty attrition rate (except retirements)*	Human Resources Data	Human Resources Administrator
Percentage of faculty who are satisfied with the VCU School of Pharmacy	AACP Faculty Survey Questions 4, 10, 18, 23, 24, 25, 30, 35, 37	Director of Education and Assessment
Percentage of faculty who believe that they have adequate resources	AACP Faculty Survey Questions 26-29	Director of Education and Assessment
Percentage of faculty members serving as leaders in state/national organizations (elected officers or appointed chairs)	Department Annual Reports	Department Chairs
Percentage of faculty salaries equal to or exceeding the AACP 50th percentile	Human Resources Data	Dean
Percentage of faculty who feel valued at work	Faculty survey to be developed	Director of Education and Assessment
Module 6: Staff Experience and Success
Percentage of staff who are satisfied with the VCU School of Pharmacy	Staff survey to be developed	Director of Education and Assessment
Percentage of staff who believe that they have adequate professional development	Staff survey to be developed	Director of Education and Assessment
Percentage of staff who feel valued at work	Staff survey to be developed	Director of Education and Assessment
Staff attrition rate (except retirements)	Human Resources Data	Human Resources Administrator
Module 7: Fundraising
Percentage of alumni who participate in the annual giving	Development Office	Director of Development
Total amount of funds raised, including private funds received as cash gifts and new gifts pledged annually.	Development Office	Director of Development
Module 8: Graduate Program
Percentage of graduate students completing their degree who are satisfied with their training	Graduate directors	Associate Dean of Research and Graduate Studies
Percentage of PhD students presenting at national or international scientific conference by graduation	Graduate directors	Associate Dean of Research and Graduate Studies
Percentage of students with at least 1 first author original research peer-reviewed article by graduation (in press or published)	Graduate directors	Associate Dean of Research and Graduate Studies
Percentage of students employed as a post-doc or a position in government, industry, or related field within 6 months of PhD graduation	Advisors, students	Associate Dean of Research and Graduate Studies
Module 9: Research and Scholarship
Total amount of grant funds awarded in fiscal year (all sources) and average $ per full-time faculty*	Budget office	Assistant Dean of Finance
Number of grant applications awarded in fiscal year*	Budget office	Assistant Dean of Finance
Number of peer reviewed journal publications (average number per full-time faculty member)	Department Annual Reports	Department Chairs
NIH Funding Ranking (AACP and Blue Ridge Institute for Medical Research)	AACP and Blue Ridge rankings	Dean
Module 10: Student Experience and Success
NAPLEX® and Multistate Pharmacy Jurisprudence Examination pass rate for first-time exam takers exceeds US average*	National Association of Boards of Pharmacy	Director of Education and Assessment
Four-year attrition rate for PharmD students (total and by demographics of age, gender, and race)*	Office of Admissions and Student Services	Associate Dean for Student Services
Percentage of graduating class with pharmacy jobs within 6 months of graduation*	Office of Admissions and Student Services	Associate Dean for Student Services
Percentage of graduating students who are satisfied with the VCU School of Pharmacy PharmD Program	AACP Graduating Student Survey Questions 44-60	Director of Education and Assessment
Percentage of P1-P3 students who are satisfied with the VCU School of Pharmacy PharmD Program	Student Experience Survey	Director of Education and Assessment
Percentage of graduating students who would choose the VCU School of Pharmacy again for their pharmacy education	AACP Graduating Student Survey Question 71	AACP Graduating Student Survey/Director of Education and Assessment
Number and percentage of graduating PharmD students matching with a residency program	Office of Admissions and Student Services	Associate Dean for Student Services

Denotes university mandated KPIs

AACP, American Association of Colleges of Pharmacy; GPA, grade point average; KPI, key performance indicator; KPI, key performance indicator; NAPLEX, North American Pharmacist Licensure Examination; NIH, National Institutes of Health; PharmD, Doctor of Pharmacy; PhD, Doctor of Philosophy; US, United States; VCU, Virginia Commonwealth University.

Regarding lessons learned, the Delphi technique allowed transparency in the process of adding, refining and removing school wide KPIs. Without the use of this technique, we would likely have remained with an undesirably high number of KPIs, a lack of focus, and low buy-in from faculty and staff. Prior to this initiative, most faculty and staff were not aware of the school’s KPIs. Their inclusion in this systematic process ensured that new voices, namely staff, junior faculty, or faculty without administrative duties, were heard.

KPIs are the basis for both internal and external accountability at the VCU-SOP. One of the recommendations by Burke and Minassians is for institutions to implement internal performance reporting.41 At the VCU-SOP, members of the Executive Committee are charged with setting target goals and responsible units for each KPI at the beginning of the fiscal year. Both the Executive and Outcomes Assessment Committees are responsible for designing and implementing actionable quality improvement strategies. This continuous review of KPIs ensures that we are using valid measures to assess the school’s performance and that we are measuring critical outcomes for accreditation purposes and state priorities, per Burke and Minassians’s recommendations.41 Furthermore, select KPIs applicable to each employee’s role have been integrated into annual performance evaluations. As a result of this process, the VCU-SOP added new items to already existing surveys and created new surveys to collect newly proposed KPIs (e.g., KPIs related to staff experience) or enhance collection of existing ones. As for external accountability, achievement of the nine KPIs mandated by the university is reported to the Vice Provost of Health Sciences who, in turn, communicates how the VCU-SOP contributes to University goals to higher levels of administration, as well as to alumni and donors.

One limitation is that the panel was not comprised of a group of experts in performance management, per best practices in conducting a Delphi technique.31 Additionally, the results are not generalizable to other pharmacy schools. However, our goal was to illustrate a way to engage faculty and staff in the process of refining the KPIs that other institutions may find useful.

CONCLUSIONS

This study illustrates how a Delphi consensus technique can be used in committee service as a means to promote faculty and staff involvement in school’s decisions, in this case KPIs selection, instead of following a top-down approach. The process resulted in the successful refinement of the number of KPIs to be tracked by the VCU-SOP while engaging faculty and staff.

Pharmacy schools’ administrators may opt to evolve to a more integrated perspective of organizational success by using widely known strategic management tools such as the balanced scorecard. The Delphi technique could be a viable approach to elicit specific measures of performance for the balanced scorecard dimensions, i.e., financial, customer, learning and innovation, and internal process perspectives. The Delphi technique can also be utilized for multiple other purposes in committee or college work when seeking to obtain consensus about specific skills, competencies, programmatic content, organizational aspects of the school, among others.