Development of an Accreditation System for Evaluating Biomedical Scientific Associations Through a Participatory Process in Iran.

Background: Accreditation is used to monitor, guide, examine, and assure the quality of higher education. There is no formal evaluation system to ensure the quality and quantity of scientific associations' activities. So this study aimed to develop and implement a national accreditation system to be applied for biomedical scientific associations through a participatory process among stakeholders.
Methods: Consensus development techniques, i.e., focus group and Delphi methods, were used to design the accreditation system. An expert committee, set up at the recognized accreditation body, devised the accreditation structure, procedure, and permit rules using the focus group technique. Then, we prepared the standards draft which was further modified in an expert panel in focus group sessions and finalized among the stakeholders through the Delphi technique. Finally, the procedure was performed for 66 associations.
Results: The accreditation structure, procedure, and standards were determined and legitimized. Standards included 20 ones in four domains of educational activities (3 standards), research affairs (2 standards), cross-sector collaborations (2 standards), and organization and equipment (13 standards). Among 66 associations, 16 and 12 were approved and conditionally approved respectively.
Conclusion: In spite of associations' influential academic and social activities, no study was found in regards to their accreditation. Standards devised in this research can be employed by scientific associations for developing their plans and enhancing their performance. This experience can be adopted for accrediting not only scientific associations but also any other academic institutions, especially non-student training ones.

The quality of higher education institutes is an indicator of institutional performance and quality education; thus, universities implement various quality-assurance methods to evaluate the quality of the institute and programs.

This experience can be adopted for accrediting not only scientific associations, but also any other academic institutions, especially non-student training ones. The system compliance with external evaluation standards of European Standards and Guidelines assures its comprehensiveness and transparency.

Introduction

The quality of higher education institutes is an indicator of institutional performance and quality education; thus, universities implement various quality-assurance methods to evaluate the quality of the institute and programs (1) and keep pace with global developments in order to provide outputs that satisfy the demands of the labor market, at local and global levels, with high efficiency and excellence in various fields. The quality assurance system in higher education helps governments deal with problems by addressing inputs, processes, outputs, and feedback; and achieve the highest possible international level by comparing these operations with competitive local standards (2). Among the different quality-assurance and evaluation approaches, accreditation is an expertise-oriented one, which is defined as a process whereby a recognized body grants approval of organizations, e.g., schools, universities, and hospitals (3). Accreditation is a recommended approach for evaluating a wide range of institutes. Among professional organizations, scientific associations, also known as scientific societies, are the community of experts and advocates of a profession or scientific area who work together to promote the progress of that field and provide the chance of the interaction of interested people at regional, national, or international levels (4).

Accreditation plays a more or less dominant role in relation to different measures that aim to monitor, guide, examine, and assure the quality of higher education. In the domain of education, accreditation procedure is applied for either educational institutions or study programs and is considered as a factor of public trust and an indicator of the accuracy and efficiency of the social policies (5).

In spite of the influential educational and research activities of scientific associations, we couldn’t find any study investigating or evaluating their performance at a national level. So, in this study, we aimed to design an accreditation system for Iranian biomedical scientific associations through a participatory process among stakeholders and then perform the first round of the accreditation procedure.

As a result of such investigation, academic institutions develop and deploy quality improvement strategies and modify their governance policies, financial system, resource allocation, and learning process in order to achieve accreditation standards’ requirements (6).

Methods

Study design

The accreditation model was adopted as the evaluation approach. Consensus developing methods, i.e., focus group and Delphi technique, were deployed for designing the system. Ministry of Health and Medical Education (MOHME) provided the ethical approval of the study.

Setting

In Iran, 90 active scientific associations related to health sciences are legitimized by the Medical Scientific Associations Commission (MSAC) at MOHME. These associations are active in the scientific, research, and technical domains and have the following responsibilities: establishing scientific collaboration among researchers, specialists, and other experts who work in various disciplines; collaborating with universities and other academic centers in educational, research, health, and treatment planning; and motivating scientists, researchers, and students to conduct scientific, research, and educational activities and publish scientific documents. MOHME is in charge of associations’ evaluation and monitoring. Hence, as in some other higher education sectors that governmental bodies perform accreditation of formal or informal organizations (7,8), MSAC at MOHME ordered the present study.

Procedure

The steps of this study were as follow:

(1) Defining the structure and procedure of the accreditation system: A ten-member expert panel was set up at MSAC consisting of related authorities of MOHME, representatives of some active associations, and some experts of the field. This panel held six focus group sessions in order to define the structure and procedure of the accreditation system. In addition, the panel determined the framework for accrediting rules of permits issuing.

(2) Devising the accreditation standards: Accreditation standards were devised using the focus group and Delphi technique:

- We first devised a draft of standards in regards to the relevant literature and nationally approved documents, including the rules and regulations of scientific associations and the descriptions of their missions and responsibilities.

- Then, we presented the draft to an expert panel comprising of ten specialists in the field of accreditation and evaluation. The specialists worked on the draft in three focus group sessions and reached a consensus on the revised version of the draft.

- Subsequently, we adopted the Delphi technique to collect the opinions of relevant experts and associations’ authorities. The eligibility criteria for the experts participating in this Delphi were: 1. Medical educationists with prominent experience in the field of accreditation and program evaluation, 2. Faculty members of Medical Sciences Universities of the country with more than five years of activities in scientific associations, and 3. Head of medical sciences associations with more than 10 years of related activities.

We prepared a document of the standards’ draft, in which participants could express their opinions on each standard (agree, disagree, or modify the standard) and add new standards, if applicable. Then, we revised the document accordingly and resent it to the participants. The Delphi rounds continued until reaching a consensus on the standards.

- Ultimately, the final set of the standards was reviewed again in the expert panel to address any inconsistencies.

(3) Legitimizing and performing the accreditation system: The system was legitimized at MOHME. Then, we performed the procedure for accrediting volunteer biomedical associations.

Results

The accreditation structure and procedure

Specialized Committee of Accreditation (SCA) was established at MSAC. Its main responsibilities include the development and periodical revision of accreditation guidelines and standards and proposing them for approval, the development and revision of accreditation questionnaires and checklists, planning for the collection of data from associations, and monitoring their activities. Furthermore, it is responsible for reaching mutual understanding through providing guidance and information to the associations.

The volunteer association applies to MSAC for undergoing the accreditation procedure and starts the process with its internal evaluation based on the checklists and questionnaires that cover the accreditation standards. The association sends its internal evaluation report and the gathered data to SCA. Then, SCA’s evaluators perform the site visit to validate the provided data, gather further information and prepare a detailed report. The report is thoroughly investigated at SCA in order to reach a proposed decision on the association’s permit. The preliminary decision, along with explanations and deficiencies, is announced to the association in order to provide the chance of stating any complaints and appeals. Finally, the permit is legitimized at MSAC.

The rulings about the permits are as follows:

“Approved”: The association has acceptable compatibility with the accreditation standards. This permit is valid for 3 years.

“Conditionally approved”: A conditional permit is issued for an association that does not have sufficient compatibility with the standards, though it fulfills them to some extent. All the newly-established associations receive a conditional permit by default. If a conditional permit is issued, the next visit is scheduled within one year. The conditional status can be repeated only twice.

“Disapproved”: Associations that have major incompatibilities with the standards are prohibited from activity. For re-examination, these associations should undergo a process to make a decision on their fate.

Accreditation standards

64 out of 92 experts participated in the Delphi technique (response rate = 69.6%), and the consensus was reached after two rounds. 49 of the respondents (76.6%) were associations authorities and the rest were evaluation experts.

The set of a totally 20 standards was developed in four domains: educational activities (3 standards), research affairs (2 standards), cross-sectoral collaborations (2 standards), and organization and equipment (13 standards). For each standard, a statement describing the optimal state of the association in that standard and related indicators were determined. Besides, it was approved that the set of standards should be reviewed and amended every two years. Tables 1 to 4 present the set of standards and their describing statements for educational activities, research affairs, cross-sectoral collaborations, and organization and equipment respectively.

Table 1

Scientific associations’ accreditation standards for the domain of educational activities

No.	Standard	Description
1	Participation in Continuous Professional Development (CPD) of graduates	The scientific association should possess the mechanism for the professional development of the graduates and demonstrate that this education is effectively implemented.
2	Participation in university students’ education	The scientific association should possess the mechanism for cooperation in educating the university students and demonstrate its preparedness for this cooperation.
3	Participation in public education	The scientific association should possess the mechanism for educating the public and demonstrate that this education is effectively implemented.

Table 4

Scientific associations’ accreditation standards for the domain of organization and equipment

No.	Standard	Description
1	Physical location	The scientific association should possess a physical space sufficient for its mission.
2	Members	The scientific association should plan for recruiting members and involving them in its activities, and demonstrate that it has attracted a suitable percentage of the target population.
3	Staff	The scientific association should possess a sufficient number of skillful human resources.
4	Website	The scientific association should have an active and efficient website and demonstrate that it has plans to keep it updated.
5	Newsletter	The scientific association is better to publish a newsletter for informing the members of the most recent scientific findings in its domain of specialty.
6	Service provision to members	The scientific association should provide its members with a variety of facilities and incentives to increase their motivation.
7	Meetings of the board of directors and committees	The scientific association should regularly hold board of directors’ meetings and form specialized committees for improving the state of affairs.
8	General assembly	The scientific association should regularly hold its general assembly.
9	Provincial/state branches	The scientific association should establish branches for improving the state of affairs in different provinces/states of the country.
10	Action plan	The scientific association should have short- and long-term planning for its activities and annually present its action plan.
11	Databank of members	The scientific association should have an updated databank of its members.
12	Attraction of financial resources	The scientific association should have plans for funding its activities and demonstrate its success in this matter.
13	Transparent documentation of financial affairs	The scientific association should transparently register all the documents on revenues and expenditures for auditing and present them when necessary.

Determining associations’ level of fulfilling the standards

To determine and optimally exploit the potentials, scientific associations that have obtained approved or conditional permits receive one to five stars according to their compliance with the standards. Each standard is divided into three levels based on the realization of its indices. As there are 20 standards, the maximum score that an association can receive in all the standards is 60. Accordingly, all the approved or conditional associations receive one star if they score <25, two stars if they score 25-44, three stars if they score 45-54, and five stars if they score >54. The advantages associated with stars, namely grants, supports, and awards are approved and notified by the MSAC annually.

Performing the accreditation system

Among 90 active associations, 66 volunteered to participate in the procedure (73.3%). 16 and 12 ones were issued as “approved” and “conditionally approved,” respectively. Only three associations received 5 stars, i.e., their scores were more than 54. It was determined not to prohibit the activity of the “disapproved” ones for this first round of accreditation. They were provided a one-year opportunity to fulfill shortcomings and apply for re-examination.

Discussion

Regarding the vital role of scientific associations as academic institutions that work with both society and universities, it is high time to establish accreditation systems for assessing their accountability and quality of services. In this study, we developed the accreditation system of scientific associations, which was legitimized at MSAC as the recognized accreditation body.

Indeed, accreditation is a formal professional review system with (1) a structure or organization established to conduct periodic reviews; (2) published standards (and possibly instruments) to be employed in such reviews; (3) a pre-specified schedule based on which reviews will be performed; (4) opinions of several experts combined to reach the overall judgments of value; and (5) an impact on the status of the reviewed body, depending on the outcome (3). In fact, accreditation has the following characteristics: It confirm (or refute) that a certain standard is met in higher education programs or institutions; it always involves a benchmarking assessment; its verdicts are solely based on quality criteria (8), and the main decision is determined by the consensus of expert evaluators of the recognized accreditation body (9).

The established system includes the main components mentioned for accreditation, namely the accreditation body, standards, and certification procedures. The same is true for the accreditation procedure which contains the main steps for such systems, i.e., self-evaluation, site visits and decision making (7,10).

In order to discuss the results further, we use the Standards and Guidelines for Quality Assurance in the European Higher Education, also known as the European Standards and Guidelines (ESG). The ESG is adopted by institutions and quality assurance agencies as a reference document for internal and external quality assurance and accreditation systems in higher education (11,12). Due to the comprehensiveness of the ESG, its standards are applicable for any formal or informal higher education organizations (8). Hence, in the following, the external quality assurance standards are introduced one by one, and the related actions of the accreditation model of this study are described.

The first ECG standard is “Consideration of internal quality assurance” and states that external quality assurance should be performed in a way that assures the effectiveness of internal self-assessments (13). If under review institutions are aware of the standards, their self-assessment can lead to improvement (7). In our study, each scientific association initially self-assessed its status on the basis of the checklists and questionnaires designed according to the standards. This internal evaluation is reviewed by SCA and, if acceptable, subsequent external accreditation steps will be taken. Meanwhile, we encountered some challenges regarding associations’ internal evaluation. Although the process and importance of internal assessment were thoroughly explained, some associations had overestimated their status.

The second ESG standard is “Designing methodologies fit for purpose”. External quality assurance should fit and achieve the aims and objectives of the reviewing body and follow relevant regulations. Hence, stakeholders should be involved in the evaluation process design and improvement (13). We did the same and devised the set of accreditation standards not only in accordance with scientific associations’ tasks and existing regulations but also through obtaining stakeholders’ opinions using consensus development methods. Furthermore, the approved plan for revising the standards every two years assures the system’s ongoing improvement.

“Implementing processes” is another ESG standard.An external evaluation process should be “reliable, useful, pre-defined, implemented consistently and published”. The process includes an internal assessment followed by site visits for external assessment, an external evaluation report, and a consistent follow-up (13). In our experience, the approved standards were officially announced to all the associations. In addition, the initial self-assessment official site visits, along the written evaluation report, ensured that this standard was met. Moreover, we had devised appropriate checklists and questionnaires to help associations in self-assessment.

One of the ESG standards is “Peer-review experts,” which emphasizes on the variety and competency of the members of external experts including reviewers from institutions, academics, students and employers/professional practitioners (13). Comparing this standard with our experience shows some scantiness. There were typically no students included in the reviewers’ team. In addition, although we selected competent, experienced evaluators for site visits, it is better to clearly address appropriate skills for evaluators. Furthermore, engaging reviewers from a wide range of health sciences disciplines promote inter-professional activities (12).

One of the ESG standards is “Criteria for outcomes”. ESG states that the decisions or judgments of any external quality assurance should be made based on clear published criteria (13). This was taken into account in this study, i.e. the set of standards and their descriptions and checklists were nationally devised and formally announced to all associations.

“Reporting” is the other ESG standard. This standard indicates that experts’ reports, as well as the formal decision, should be published and become accessible to the related partners or interested individuals (13). In our system, the reports and related decisions were officially submitted to the SAC. Although the results of the first round of accreditation were sent to the association confidentially, it was decided that the results would be made public in the next rounds.

The last ESG standard is “Complaints and appeals,” which explains the processes of complaints and appeals so that there is a possibility for institutions to state their dissatisfaction about the process, evaluators and decisions (13). In our experience, after the initial analysis, the credentials were sent to the associations so that if there were any objections to the shortcomings of the documents, they could be identified and sent for re-examination. Meanwhile, it seems necessary to define a clear workflow in this regard.

Finally, one of the important factors for a successful accreditation system is the existence of mutual trust between the accreditation body and the under-review institutions, which is formed through a common understanding of the goals and the process (7). One strategy for this purpose may be the involvement of institutions in designing the accreditation system (14). We believe that performing the Delphi technique and involving associations authorities in the system design were the key elements in the comprehensive establishment of the system.

Study Limitation: There are some limitations in our research. We couldn’t find any case in developing an accreditation system for scientific associations and it made our research the first experience. This accreditation model has some limitations and shortcomings which should be addressed and resolved in the next turn. In designing each standard, there was a tendency to improve the desired level in the association; however, considering the actual level of the associations, it was necessary to lower the level of standards in order to secure minimum conditions. This limitation is stated in the first rounds of other accreditation systems as well. The other limitation was the lack of variety of evaluators’ expertise that is of focus of newly published articles.

Conclusion

This study clarifies the accreditation process and provides a documented basis for evaluating and monitoring the associations’ performance. We identified the important aspects of association activities as standards that can help them to improve their quality of services through developing strategies for fulfilling them. In addition, approved associations have the chance of expanding the range of their activities, outputs, partners, and audience.

This experience can be beneficial for applying this type of expertise-oriented evaluation model in any other learning/academic environment, especially non-student training ones, to assure their outcomes. The system’s compliance with ESG external evaluation standards assures its comprehensiveness and transparency.

Acknowledgment

This research was supported by Tehran University of Medical Sciences and Health Services (Grant No: 96-01-159-34084). The funders had no role in study design, data collection, and analysis, decision to publish, or preparation of the manuscript.

Conflict of interests

The authors declare that they have no competing interests. AM has experience in the accreditation of medical universities at the Ministry of Health and Medical Education. AM and RM are colleagues and at the same time, they are a married couple.

Table 2

Scientific associations’ accreditation standards for the domain of research affairs

No.	Standard	Description
1	Research projects	The scientific association should possess the mechanism for the workflow, funding, receiving/placing orders, and conducting research projects within its limits of responsibility and demonstrate that these are fulfilled.
2	Scientific publications	The scientific association should possess the mechanism for publishing scientific documents (e.g., articles, journals, books, and clinical guidelines) and demonstrate its implementation.

Table 3

Scientific associations’ accreditation standards for the domain of cross-sectoral collaborations

No.	Standard	Description
1	Collaboration with organizations, institutions, and ministries	The scientific association should possess the mechanism for cooperation with supervisory organizations, institutions, and ministries for developing, delineating, and propagating the professional principles of the relevant major and demonstrate that it is active in this domain.
2	Executed contracts and letters of understanding	The scientific association should possess the mechanism for drafting letters of understanding and contracts for realizing its goals and promoting its specialized domain and demonstrate its success in this regard.