Designing, implementation, and evaluation of internship comprehensive system for assessment and monitoring.

BACKGROUND: During the past decade, the benefits of using portfolios, especially electronic portfolios, were recognized. Due to the lack of using portfolios and especially electronic portfolio in the clinical evaluations of internship training in medical schools of Iran, this study has designed, implemented, and evaluated a comprehensive system for monitoring and evaluating the activities of interns.
MATERIALS AND METHODS: This study was a software development study in the three phases of design, implementation, and evaluation. All stages of the project were carried out in the Isfahan University of Medical Sciences during 2019-2020. The software design phase was performed using the Rapid Application Development Methodology (RAD model). In the implementation phase, it was used as a pilot study in the internal department of the medical school. System evaluation was performed using a combination of quantitative and qualitative methods. Data were analyzed using the content analysis for qualitative data and descriptive statistical analysis using the SPSS software for quantitative data.
RESULTS: In the educational standards survey, in 10 items, the percentage of choosing "perfectly fit" and "fit" was above 90% (high quality). In the technical standards questionnaire, out of 35 items related to software technical quality, eight items were of acceptable quality and 27 of them were partially acceptable. In the Student Satisfaction Questionnaire, in 9 items, interns' satisfaction with the system was high or very high, and in either case, there was little or no dissatisfaction or satisfaction.
CONCLUSIONS: Positive educational effects can be used in all clinical settings if modification and improvement of the software continues, with slight modifications. Copyright:

Introduction

Clinical training is the heart of professional education and ensuring clinical competency is one of the most important goals of the clinical education process and clinical evaluation is one of the most important challenges in this process. Internships and clinical internships can be considered as the activities that facilitate learning in the real environment. These courses provide an opportunity for students to turn theoretical knowledge into the required mental-psychological and motor skills. Therefore, its importance and special place in medical graduates’ professional role are very important. Identifying the existing problems during the internship period of medical students and taking action to eliminate and improve it will facilitate the achievement of educational goals and training skilled physicians and improving the quality of the health-care system.[1] The wide scope of internships, attendance at shifts, and high contact with specialized and highly specialized patients have created obstacles for some learners in achieving the educational goals of this course. As we know, self-education is one of the main methods of learning in the internship. The most important problem of self-education is the reduction of teachers’ supervision on the quantity and quality of learning as well as the inability of interns to self-regulate learning.[2] These challenges are particularly pronounced in the large departments such as internal medicine and surgery, which have more interns and professors, and have made it more difficult to obtain evidence of learners’ progress in these areas.

Portfolio is one of the clinical evaluation tools that helps supervisors in clinical departments and helps learners in self-regulated learning.[3] A portfolio is a collection of documents that shows a person's abilities and learning process over time.[4] Portfolios may also include anything that provides good evidence of learning and achieving success, such as video or audio, artwork, or photography.[5] Some portfolios are as simple as a logbook, while others offer a deep, long-term perspective over a long period of time.[67] One of the most important and common uses of the portfolio is to develop self-assessment skills, assessment (with the aim of formative and summative assessment), showing current successes and academic advancement of learners, improving thinking skills and learning strategies, and selecting and validating competencies.[5891011] There are two types of paper and electronic portfolios available. Electronic portfolios are the electronic versions of paper portfolios that have the capacity to store audio-visual content and are designed to support a variety of pedagogical and assessment processes.[1112]

A review of the evidence in 56 studies examining the use of portfolios in postgraduate studies shows the effectiveness of the portfolio, especially in terms of increasing responsibility for learning as well as developing professional skills.[13] Medical students who have more responsibility and self-regulation in the learning process are more successful in the scientific and clinical skills.[14] Moreover, the use of electronic portfolios has more advantages than paper ones. These include their flexibility in access and content and the potential for communication. Learners spend more time in the e-portfolio and faculty review the e-portfolio more easily and quickly. In the electronic type, feedback is more effective and encourages reflection. In these studies, limited evidence has been obtained from the effect of some factors on the performance of the portfolio. Among these factors are the continuous support of trainers or peers, the method of using the portfolio, the comprehensive attitude toward using the portfolio and the level of basic training for using the portfolio.[13]

Studies on the use of e-portfolios have been conducted in some medical schools around the world.[815] However, a review of studies in the field of medical education in Iran revealed that most of the work in the field of nursing and related internships have used portfolios to evaluate the students’ activities.[1617] In the field of medicine, only the assistants of the specialized course are evaluated by the portfolio. According to the program of the Ministry of Health in the past, a specific portfolio has been prepared for each specialized assistant, in which the professors record the results of their studies of the assistant's performance in this portfolio.[18] Due to the lack of use of portfolio and especially electronic portfolio in the clinical evaluations of internship training in medical schools of Iran, this study has designed, implemented, and evaluated a comprehensive system for monitoring and evaluating the activities of interns. In addition to including an electronic portfolio, this system is a more comprehensive and complete system than an electronic portfolio and allows monitoring and evaluation of all activities. In designing this comprehensive monitoring and evaluation system, the educational and technical necessities and requirements of the electronic portfolio have been considered to obtain the most coverage of educational and technical standards and to provide acceptable satisfaction with its use by interns and professors.

Materials and Methods

This study was a tool development study in three phases of design, implementation, and evaluation. All stages of the project were carried out during 2018–2019. The approval for this research was obtained from the Ethics Committee of IR.MUI.RESEARCH.REC.1398.282.

Design phase

This stage of the study was a software development process using the Rapid Application Development Methodology (RAD model). Based on the RAD model, the software was designed during the following steps.

Research (feasibility study, determining the necessities, and requirements of the e-portfolio system and software)

The list of portfolio requirements in two educational and technical sections was done by reviewing the available texts, reviewing the existing internal and external e-portfolios that were prepared for different target groups, and also holding focus group meetings with members of the portfolio committee.

Planning (planning the components of the system and software of the electronic portfolio)

The initial framework and general components of the software that can be read by software planners were prepared. At this stage, five two-hour focus groups were formed with the presence of the main researchers and the head of the software engineering team and the initial framework was prepared. Then, in a 2-h focus group with the presence of the main researchers and three faculty members of the internal group, the prepared framework was finalized.

Design

The layout and interface of the software and system were designed in accordance with the prepared plan. In this phase, the previously designed roadmaps were given to the programmer, and the application was designed.

Module development

At this stage, the modules were designed, in each of which there was a development process, then testing, and finally feedback. Finally, the main software of the portfolio was designed to be web-based and compatible with the Android system. Simultaneous with the design by the engineering team, at intervals determined by the committee, the ease of working with the system and its graphics were considered and approved by the researchers.

Integration

At this stage, the entire software was installed by the technical team as an output. Furthermore, an attempt was made to increase the level of software integration with other university application systems.

Setup (development and installation of electronic folder system and software)

Web server, database, APIs, and storage solutions were determined by the engineering team, and the necessary negotiations were conducted by the researchers. Back end activities specify how the site works, updates and changes. Because all information such as user profiles, photos, and anything users upload are stored in a database, maximum security predictions and precautions were taken. Approval of bylaws and obtaining relevant permits in the council of the internal group to use the system Pilot, training, and empowerment of professors and interns of the group to use the portfolio system, planning and preparing the software pilot application platform, membership of all interns in the software among the activities of this stage was.

Maintenance

The activities of the software maintenance stage that were performed by the software support software company in four categories: corrective maintenance, adaptive maintenance, supplementary maintenance, and preventive maintenance.

Implementation phase

At this stage, the system was put into the operation and used as a pilot in the internal department of the medical school. Quarterly activities of attending the internal department, including attending rounds, number of patients visited, diagnostic procedures defined for the intern based on educational curriculum, presenting an article in the club journal, morning report and attending the clinic and other items in the system, was recorded in this software daily by interns. Through their panel, professors could monitor recorded daily activities and send feedback to their intern. At the end of each 2 weeks, the system is closed for the intern and the relevant instructor reviews and evaluates the intern's documentation for 7 days. At the end of the course, the overall score is summarized by the intern and presented as the intern's overall score.

Evaluation phase

Participants for this stage included internal faculty members and interns. The study sample consisted of professors who used the comprehensive monitoring system during the 3 months of intervention and interns who spent their internal part during the intervention for 3 months, were chosen through the purposive sampling (for professors) and census (for interns). System evaluation was performed using a combination of quantitative and qualitative methods. In the first stage, using semi-structured interviews with system user professors, the strengths, weaknesses, and strategies for improving the system were identified. These interviews were conducted by purposeful sampling of faculty members who used the system during the pilot period. Interviews were analyzed using the content analysis method. In this way, the interviews were fully implemented and typed. Then, the meaningful sentences were coded, and the codes that contained a common concept were placed in a category. Finally, using the results of content analysis, the necessary corrections were made in the maintenance phase of the system.

In the second stage, software evaluation tools for software training standards were developed by the researchers. The tool was prepared using a five-point scale (from perfectly suitable to not at all suitable). The validity of the instrument was assessed using the opinions of three medical education specialists and three internal medicine professors. Reliability was obtained using the Cronbach's alpha coefficient of 0.81. At the end of a period of software use in the internal department of Isfahan Medical School, a questionnaire was E-mailed to all internal faculty members who had worked with the system. Descriptive statistical analysis was performed. The score standard obtained in this section was considered to be of low quality below 90% and high quality above 90%.

In the third stage of evaluation, according to the software testability criteria and international software quality standards,[19] a checklist for evaluating software technical standards based on different levels of testing (unit testing, integration testing, system testing, and acceptance testing) was developed by the researchers. The checklist was prepared using a three-point scale (yes, somewhat, and no) and its validity was confirmed using the opinion of software design experts at the Isfahan University of Medical Sciences. The checklist was completed by three software engineers approved by the virtual education department of the university. Two similar comments out of three comments were considered as the final comments. The final “yes” comment was used as an acceptable quality and the “somewhat” and “no” comments were used as a guide for correction in the supplementary maintenance phase.

Finally, to assess the acceptability of the software, a satisfaction questionnaire was designed based on the QUIS standard questionnaire, which is a valid tool for measuring user satisfaction by interacting with the information systems.[20] In this questionnaire, first the general reaction of users to interaction with the software was measured, then questions were asked about the information screen, software words and messages, ease of learning, guidance and capabilities and capabilities of the software. The checklist was designed with a five-point scale (strongly agree, agree, have no opinion, disagree, and strongly disagree) and obtained the average score of each item. The validity of this checklist was confirmed using the opinion of medical education specialists and professors of the internal group. The reliability of the questionnaire was calculated using the Cronbach's alpha coefficient of 0.89.

These questionnaires were completed by all interns who were in the internal group for 3 months and used the software. Statistical indicators (mean and standard deviation) of each item were calculated and finally the general satisfaction of the participants from interacting with the software was determined. Data were analyzed using content analysis for qualitative data and descriptive statistical analysis using SPSS version 21.0 [IBM SPSS Statistics for Windows, Version 21.0. Armonk, NY: IBM Corp.] for quantitative data.

Results

Strengths, weaknesses, and system improvement strategies

Content analysis of semi-structured interviews with nine female faculty members and one male faculty member of the internal group from three Al-Zahra, Khorshid and Amin hospitals was performed and using the above results, the necessary corrections were applied before the second round of using the system [Table 1].

Table 1

Main categories and extracted codes from the content of interviews with professors by content analysis method

Advantages of the system	Disadvantages of system	Strategies for system improvement
Considering important points in training and assessment of interns	Editing score is impossible after entering them	Activate score editing
Evaluating interns with no need for professor’s presence		Possibility to compare interns before scoring
Professor’s assistance for reminding interns through their profile picture	Occasional problems in system login	Considering a section for commenting and scoring of assistants to interns
Interns are able to send private and direct feedback to their professors	Working difficulty with electronic systems for aged and experienced professors	Considering a section for importing the strengths and weaknesses of the intern
Suitable data security
Suitable support from the manager and software engineer	The possibility of duplicating case histories	Considering a section for intern’s comments about ward and shifts (professors are able to find out which intern has given this opinion)
System troubleshooting according to users’ feedback	Extra works for interns
Increment of intern’s precision and discipline in recognition of case and his/her medical history and gaining case history accurately	Time-consuming for general professors who have many students	Necessary modification should be made for evaluating interns and assistants
Increasing learning rate of interns
More accurate assessment of interns	Difficulty of reading electronic texts and eye fatigue	Considering a place for uploading patient’s medical record
Saving time
On-time scoring	Resumption of the login process in case the phone rings	Importing patient’s history in brief
Available every places and other benefits of an electronic system		Professors can score together for their common interns
Easier use for professor	Wasting time due to system security processes
Reduction of deception		Easing the login process
Increasing justice in student evaluation	Impossibility to save username and password	Upgrading system over time
Archiving information safely	Impossibility to use in MAC OS	Considering a solution for duplicating case histories
Elimination of paperwork and saving more papers	Impossibility of evaluating the work in the clinic	Considering a section for clinic assessment

Evaluation of software training standards

The results of a survey of 21 professors using the system (12 [57.14%] women and 9 [42.85%] men) on the various sections in the faculty panel showed that in 10 cases, the selection percentage was quite appropriate and above 90 (high quality). Only the journal club lists were considered suitable by <90% by the professors. Moreover, it was mentioned in the description section that the journal club was usually presented by the assistants, and therefore, in the internship portfolio the intern should be able to determine whether he was only a participant or presenter [Table 2].

Table 2

The average of professors’ opinions about the quality of each part of the internship portfolio system

Parts of system	Works appropriate and perfectly appropriate (%)	Average±SD
1. Desk: Recording the end of course assessment (knowledge, skill, and interpersonal relationship)	95.2	4.61±0.74
2. Desk: Recording the score of professional behavior	90.5	4.71±0.78
3. Desk: Providing explanations and necessary points to the intern	100	4.80±0.40
4. Desk: List of recorded items in educational topics	95.2	4.71±0.56
5. Desk: List of submitted journal clubs	85.7	4.42±0.87
6. Desk: List of submitted conferences	90.04	4.61±0.66
7. Desk: Registration of the intern’s absence	100	4.85±0.35
8. Desk: Registration of intern leave	100	4.90±0.30
9. Opinion poll about classes	100	4.90±0.30
10. Opinion poll about the sections	100	4.8±0.40
11. Professional ethic (only for professors who study ethic course)	100	4.38±0.92

SD=Standard deviation

Software testability and software quality standards

Statistical analysis of the checklist related to the technical standards of the system shows that out of 35 items related to the technical quality of the software, eight items are of acceptable quality and 27 items are somewhat acceptable which were manipulated and developed in the final stage. As shown in Table 3, there was no technically unacceptable item in the software.

Table 3

Frequency of software technicians’ opinions about the quality criteria of the internship portfolio system

Row	Characteristic	Technical quality criteria of system	Frequency of selection
			Yes	Fairly	No
1	Operational	Suitability: Suitability of operations in the software	3
2		Precision: Power and accuracy of operations	3
3		Interoperability: Sharing Information	1	2
4		Security: Prevent unauthorized access	2	1
5	Reliability	Ability to test errors: Ability to manage errors - Minimal bugs in the program		3
6		Recoverability: Return to safe mode when a problem occurs	1	2
7		Maturity: System fixes bug over time	1	2
8	Maintainability	Testability: Ability to test the program	1	1	1
9		Stability: Versatility and support for different versions		2	1
10		Ability to change: Ability to update the program		3
11		Ability to analyse: Identify possible errors and provide appropriate information to the user		3
12	Efficiency	Time behaviour: Ability to respond quickly	2	1
13		Resource usage: Optimal use of the entire user desk environment	1	2
14		Resource usage: The amount of user interface configuration from outside the application to the source	1	2
15		Resource usage: Prevent execution from running due to special processing	1	2
16		Resource usage: Correct use of buffering	1	2
17	Using	Usability: Proper support of the program in Persian language	3
18		Usability: Ability to move between pages and specify the user’s location	3
19		Usability: No dependence on other applications	2	1
20		Usability: Customization of program performance	2	1
21		Usability: Maintain the latest status of the program for the next load	3
22		Usability: Appropriate number of clicks required to reach the goal	3
23		Ability to absorb: The amount of use of internal symbols	1	2
24		Ability to absorb: Fit and beauty of items on the page	1	2
25		Ability to absorb: Coordinate the user interface throughout the application	1	2
26		Comprehensibility: Delivering messages about the appropriate behaviour of the system based on its functionality	1	2
27		Comprehensibility: Informing the user about performing a specific process	1	2
28		Comprehensibility: The degree to which the program adheres to the principles of other programs	1	2
29		Learning ability: Having a thematic guide	3
30		Learning ability: Existence of online guide	3
31	Content	Accuracy: The degree of accuracy and quality in data acquisition and information preparation	1	2
32		Precision: The accuracy of the information provided in the application	1	2
33		Enrichment: The amount of content enrichment	1	2
34		Interoperability: Communication with peripheral systems	1	2
35		Appropriateness: The volume and appropriateness of the information provided in the program	1	2

Software satisfaction and acceptance

The satisfaction assessment of 51 students using the system (25 women and 26 men) was about different aspects of the system. Participants were 13 (25.5%) in the first 6 months of the internship, 24 (47.1%) in the second 6 months of the internship, and 14 (27.5%) in the third 6 months of the internship. In the first 3 items of the questionnaire, which was related to the trainees’ overall satisfaction with the use of the e-portfolio, the frequency of choosing to agree and strongly agree was more than 50%. Furthermore, the average score in each of these three items was higher than three and about average (2.5–3.5). Of the next 14 items, in 9 items (items 17, 16, 15, 13, 12, 9, 6, and 5) the interns’ satisfaction with the system was high or very high, and there were no cases of dissatisfaction or low satisfaction [Table 4].

Table 4

Frequency and average of interns’ satisfaction of the internship portfolio system

Items	Frequency of selection					Average±standard deviation
	Quite agree (score 5)	Agree (score 4)	Have no opinion (score 3)	Disagree (score 2)	Completely disagree (score 1)
1. The system helps interns to achieve the goals of internal ward	4 (7.8)	22 (43.1)	6 (11.8)	12 (23.5)	7 (13.7)	3.07±1.24
2. The system helps professors for a better supervision on interns learning	5 (9.8)	22 (43.1)	11 (21.6)	9 (17.6)	4 (7.8)	3.29±1.11
3. The system provides an appropriate feedback for both professors and interns	5 (9.8)	21 (41.2)	5 (9.8)	15 (29.4)	5 (9.8)	3.11±1.22
4. The font and size of words make them easy to read on the screens of the system	21 (41.2)	27 (52.9)	2 (3.9)	0	1 (2)	4.31±0.73
5. Necessary information can be easily found on the screens of the system	15 (29.4)	26 (51)	6 (11.8)	3 (5.9)	1 (2)	4.00±0.91
6. The arrangement of information on each screen of the system is logical	17 (33.3)	25 (49)	6 (11.8)	2 (3.9)	1 (2)	4.07±0.89
7. The overall design of the system has a good visual appeal	9 (17.6)	11 (21.6)	7 (13.7)	19 (37.3)	5 (9.8)	3.00±1.31
8. It is possible to go back and forth between different pages of the system	10 (19.6)	19 (37.3)	2 (3.9)	17 (33.3)	3 (5.9)	3.31±1.28
9. The words used in the system are free of any ambiguity	11 (21.6)	18 (35.3)	19 (37.3)	2 (3.9)	1 (2)	3.70±0.92
10. If I have problem using the system, the system will guide me through sending messages	8 (15.7)	6 (11.8)	9 (17.6)	25 (49)	2 (3.9)	2.80±1.23
11. System messages are short and vague	8 (15.7)	13 (25.5)	13 (25.5)	16 (31.4)	1 (2)	3.21±1.11
12. It is easy to learn how to work with the system and enter information in it	14 (27.5)	30 (58.8)	4 (7.8)	2 (3.9)	1 (2)	4.05±0.83
13. The briefing session with the system was enough for my training	8 (15.7)	31 (60.8)	10 (19.6)	1 (2)	1 (2)	3.86±0.77
14. The number of steps required for successful data entry is too much	13 (25.5)	20 (39.2)	9 (17.6)	8 (15.7)	1 (2)	3.70±1.08
15. If the network speed is good, the system works with a good speed	14 (27.5)	29 (56.9)	3 (5.9)	4 (7.8)	1 (2)	4.00±0.91
16. Correcting my mistakes easily with no need to repeat the steps from the beginning	11 (21.6)	23 (45.1)	4 (7.8)	10 (19.6)	3 (5.9)	3.56±1.20
17. The system works in a variety of devices such as mobile phones, tablets and computers with an acceptable quality	4 (7.8)	32 (62.7)	13 (25.5)	1 (2)	1 (2)	3.72±0.72

Discussion

The results of evaluation of educational and technical standards show that the software was acceptable in terms of educational standards. With respect to the standards, although there was no technically unacceptable item in the software, most of its items need to be technically upgraded and should be considered in the software maintenance stage. Also, with respect to the satisfaction of software users, there was no low satisfaction or dissatisfaction in any case. There was a general belief that the portfolio affects both the process and the outcome of learning, and overall the portfolio improves feedback and self-awareness, enhances knowledge and understanding, and improves student-coach communication, and improves the ability to deal with problematic emotional situations. However, achieving the mentioned benefits will be possible if the portfolio is accepted by users.[821] Considering that there is resistance from users at the beginning of using any new system, the results of the satisfaction survey in this study were satisfactory. This good satisfaction of software users reinforces the prospect that the use of this portfolio in the future will also affect learners’ learning. The following are the most important factors mentioned in the sources[8] on the acceptability and feasibility of the portfolio, about the internship portfolio is discussed.

User attitude

The average attitude of internship portfolio users in the first period of its use is shown in the results. Existing studies on the attitude of portfolio users are contradictory. In some studies, most participants believed that the portfolio should be used to promote rethinking, but in some others, users were unsure that the items in the portfolio should not be used against them.[8] In the study of the Maastricht Medical School, their experience in using the portfolio showed that the conditions for the success of the portfolio with the aim of rethinking are: proper mentoring, portfolio structure and instructions, sufficient experiences and new resources for rethinking and cumulative evaluation.[15]

Different levels of organizational support in portfolio implementation

The portfolio implementation process is more important than the portfolio structure.[8] In the portfolio used for surgeons in the United States, the initial participation rate was low (<50%), but after improvements in the implementation process, including monthly feedback, organized discussions, users’ ability to contact portfolio supervisors and regular information feeding, the participation rate reached 100%.[22] In our study, although the cumulative role of the portfolio was emphasized and the interns had to complete the portfolio to receive their score, in order to be able to use the cumulative role of the portfolio more and the interns participate more actively, regular feedback and discussion sessions for rethinking are necessary. However, during this process, users were completely satisfied with the organizational support of the portfolio during its implementation.

Initial and continuous support and mentoring

Mentoring is one of the important factors in the success of the portfolio. Studies have shown that monthly feedback from mentors assigned to each individual has increased participation by about 50%–100%.[8] During the present process, there was support and mentoring by the relevant professors for all interns. Of course, professors need to receive more training over time to provide ongoing feedback to students.

Challenges related to time and cost

What was considered in this process as technical standards and also the use of the basic process of software design has made the process of using the portfolio as easy as possible and reduce its time-consuming nature. The results of user satisfaction (both professors and students) indicate the ease of use and speed of use. In terms of cost, there was no problem for users and its use is possible through mobile phones, tablets and ordinary personal computers, and users have been satisfied in this regard.

Purpose of the portfolio

Most portfolios have the purpose of formative evaluation and half of them are used for the purpose of summative evaluation, which may or may not be with formative evaluation. Studies have shown that when the portfolio is used for the purpose of compression assessment, the level of user cooperation is higher and when it is not used for this purpose, it is possible to stop using it.[8] According to this matter, in the present process, the portfolio is designed with both compression and developmental goals, and its emphasis on the role of compression is emphasized until the completion of the trainings.

Conclusions

In this study, in addition to software design, the strengths and weaknesses, coverage of technical and educational standards of the software, as well as the level of user satisfaction with its various parts were determined. These results are used to continue the process of establishing and using the internship portfolio in clinical departments. The following are also suggested:

Continue the process of maintenance and technical upgrade of the system

Planning for continuing education of students on the importance of rethinking

Educate professors about the importance and how to provide feedback to students

Upgrading the educational structure and evaluation of the portfolio according to user feedback.

Financial support and sponsorship

This work was supported by medical education Research Center, Isfahan University of Medical Sciences, Isfahan, Iran (No. 298061).

Conflicts of interest

There are no conflicts of interest.